LS110 Linux Kernel Debugging Training and Courseware Course Outline

Unit 1: Course Overview

  • Introduction
  • Course Prerequisites
  • System Environment
  • Conventions Used In The Student Materials
  • References
 

Unit 2: Kernel Source Tree

  • Unit Overview
  • View of Kernel
  • Installing Kernel Source
    • SUSE 10.x Source Install
    • Practice Exercise: SUSE Source Installation
    • Fedora Core 5 Source Install
    • Practice Exercise: Installing FC5 Source
    • kernel.org Source Install
    • Practice Exercise: Installing the Vanilla Kernel Source
  • Patching the Kernel
    • Practice Exercise: Applying a Kernel Patch
  • Finding References
    • Kernel Symbol Tables
    • Searching with Linux Commands
    • The cscope Command
    • Practice Exercise: Using cscope
    • The kscope Command
    • Practice Exercise: Using Kscope
  • Unit Review
 

Unit 3: Building the Linux Kernel

  • Unit Overview
  • The Build Environment
    • Practice Exercise: Getting Ready
  • Steps for Building Kernel
    • Creating a Configuration File
    • Practice Exercise: Creating a Configuration File
    • Editing the Configuration File
    • Practice Exercise: Editing Configuration File
    • Make the Kernel
    • Practice Exercise: Build the Kernel
    • Install the Modules
    • Practice Exercise: Install the Modules
    • Install the Kernel
    • Practice Exercise: Install the Kernel
    • Modify Boot Loader Files
    • Make Additional Kernel Modules (SUSE Only)
  • Unit Review
 

Unit 4: Linux Kernel Modules

  • Unit Overview
  • Module Overview
  • Module Example
  • Compiling the Module
  • Managing Modules
    • The insmod Command
    • The rmmod Command
    • The lsmod Command
    • The modprobe Command
    • The modinfo Command
  • Unit Review
  • Unit Exercise: Building a Module
 

Unit 5: Kernel Space to User Space
  • Unit Overview
  • Kernel Space versus User Space
    • System Call Interface
    • The strace Command
    • Practice Exercise: The strace Command
  • The /proc Directory
    • Process Structure of /proc
    • Practice Exercises: Processes in /proc
    • Kernel Data in /proc
    • Practice Exercise 1: /proc Kernel Data
    • Practice Exercise 2: /proc Kernel Data
    • Kernel Parameters in /proc/sys
    • The sysctl Command
    • The sysctl.conf File
    • Practice Exercise: The sysctl Command
  • The /sys Directory
    • Overview of kobjects
    • kobject Reflection in /sys
    • Practice Exercise: The /sys Directory
  • The /configsys Directory
  • Unit Review
 

Unit 6: Debugging with printk
  • Unit Overview
  • The printk Statement
    • printk Log Levels
    • printk Kernel Parameters
    • printk Message Console
    • The netconsole Module
    • The dmesg Command
    • Practice Exercise: Using printk
  • The klogd Daemon
  • The Logging Daemons
    • The syslog.conf File
    • Practice Exercise: syslog Daemon
    • The syslogng. conf File
    • Practice Exercise: sysnlogng Daemon
  • The logger Command
    • Practice Exercise: The logger Command
  • Unit Review
 

Unit 7: Kernel Probes
  • Unit Overview
  • What are Kernel Probes
  • Configuring Kernel Probes
  • Types of Kernel Probes
    • How kprobes Work
    • Practice Exercise: kprobe with Offset
    • How jprobes Work
    • Practice Exercise: Using jprobes
    • How kretprobes Work
    • Practice Exercise: kretprobes
  • Unit Review
 

Unit 8: Using Systemtap
  • Unit Overview
  • Why Systemtap
  • Systemtap Architecture
  • Systemtap Requirements
  • The stap Command
    • A Simple Example
    • stab Options
    • Practice Exercise: The stap Command
  • The Scripting Language
    • Basic Syntax
    • Defining Probe Points
    • Practice Exercise: Basic Probes
    • More on Targeted Variables
    • Using Functions
    • Probe Point Aliases
    • Practice Exercise: Functions and Aliases
  • More on the Scripting Language
    • Control Statements
    • Working with Arrays
    • Collecting Data with Aggregates
    • Embedded C
    • Checking Kernel Version and Architecture
    • Practice Exercise: Arrays and Aggregates
  • Developing Tapsets
    • Writing Functions
    • Tabset Libraries
    • Naming Conventions
  • Unit Review
 

Unit 9: Oops and Dumps
  • Unit Overview
  • The Oops Message
    • Oops Analysis Hints
    • An Oops Message
    • Practice Exercise: Oops Message
  • Magic SysRq Key
    • SysRq Command Keys
    • Apparent Kernel Hangs
    • Practice Exercise: Magic SysRq Key
  • kexec and kdump
    • The kexec Kernel
    • The kdump Kernel
    • Linking the Two Kernels
    • Saving the Dump Images
    • Practice Exercise: kexec and kdump
  • Linux Kernel Crash Dump
    • LKCD Process Flow
    • LKCD Configuration
    • Saving the LKCD Dump
    • Testing the LKCD Configuration
    • Practice Exercise: LKCD
  • Using Diskdump (Fedora Only)
  • The netdump Utilities
  • Unit Review
 

Unit 10: Dump Analysis
  • Unit Overview
  • Using gdb to Debug the Kernel
    • Useful gdb Commands
    • Notes on Using gdb
    • Using gdb with the Running Kernel
    • Practice Exercise: Using gdb with the Running Kernel
  • Using the crash Utility
    • The Crash Commands
    • Options to crash Utility
    • Analyzing a Running Kernel
    • Practice Exercise: Analyzing a Running Kernel
    • Dump Analysis with crash
    • Practice Exercise: Dump Analysis with crash
  • Using the lcrash Utility
    • The lcrash Commands
    • Options to lcrash Utility
    • Using lcrash on the Running Kernel
    • Practice Exercise: Using lcrash on the Running Kernel
    • Dump Analysis with lcrash
    • Practice Exercise: Dump Analysis with lcrash
  • Unit Review
 

Unit 11: Kernel Mode Debuggers
  • Unit Overview
  • Limitations to Kernel Debuggers
  • Using the KDB Debugger
    • Applying the KDB Kernel Patches
    • Configuring KDB
    • Invoking the Kernel Debugger
    • Overview of KDB Commands
    • More on KDB Commands
    • Customizing the KDB Environment
    • Practice Exercise: Using the KDB Debugger (Optional)
  • Using the KGDB Debugger
    • Kernel Patches for KGDB
    • Configuring and Installing KGDB
    • Practice Exercise: Patching, Configuring, and Installing KGDB (Optional)
    • Establishing a Serial Connection
    • Establishing an Ethernet Connection
    • Controlling Kernel Execution
    • Performing Stack Traces
    • Analyzing Threads with GDB
    • Debugging Modules
    • Practice Exercise: Using KGDB (Optional)
  • Unit Review
 

Unit 12: Profiling the Kernel
  • Unit Overview
  • System Wide Profiling with Oprofile
  • Oprofile Configuration
  • Oprofile Commands
    • The opcontrol Command
    • The oprof_start Command
    • The ophelp Command
    • Practice Exercise: Collecting Profile Data
    • The opreport Command
    • Practice Exercise: Using opreport
    • The opannotate Command
    • Practice Exercise: Using opannotate
    • The oparchive Command
    • The opgprof Command
  • Unit Review