Fotis' Blog

Recently I received some comments from a friend about a previous article on linux kernel debugging using kgdb. What he asked me was how could he load symbols from a kernel or a kernel module. So I wrote a quick guide to help you start with kernel debugging. After each step I will show you the gdb output.

First of all you should start gdb!

$ gdb
GNU gdb (GDB) 6.8.50.20090628-cvs-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb)

Then you should load all kernel symbols from the vmlinux file. This can be found at the directory where you compiled the kernel, most probably /usr/src/linux. Remember to compile the kernel using debug information by setting the appropriate option, it will help you a lot!

(gdb) file vmlinux
Reading symbols from /home/fotisl/programs/kgdb/vmlinux...done.
(gdb)

You’re ready to start debugging! Set the target and use the Alt-SysRq-G sequence as it was described at the previous post. You can now set breakpoints, watch anything you want in memory, step or continue running the kernel!

(gdb) target remote /dev/pts/12
Remote debugging using /dev/pts/12
kgdb_breakpoint (key=103, tty=0x0) at kernel/kgdb.c:1721
1721            wmb(); /* Sync point after breakpoint */
(gdb)

Now let’s see how we can debug kernel modules. I will test the l2cap bluetooth kernel module.

You first need to find the object file which contains the module. For l2cap this is net/bluetooth/l2cap.o in the kernel source tree. Transfer this to the host (or the machine running gdb if you’re not using a virtual machine). Then load the module in the virtual machine. This creates a new directory in /sys/module named after the module name, i.e. l2cap. Inside this directory, there is another one named sections which contains the addresses where all sections are loaded. We are interested in the .text section so we read the file /sys/module/l2cap/sections/.text.

$ cat /sys/module/l2cap/sections/.text
0xe0c77000

We know where the .text section is loaded so we can now load the symbols from l2cap.o using the add-symbol-file gdb command.

(gdb) add-symbol-file l2cap.o 0xe0c77000
add symbol table from file "l2cap.o" at
        .text_addr = 0xe0c77000
(y or n) y
Reading symbols from /home/fotisl/programs/kgdb/l2cap.o...done.
(gdb)

If you need to load other sections too, in case they are not contiguous with the text in memory, you need to read their addresses. For example we’ll load both the .text and the .data sections (you should do .bss too but it’s omitted since I wanted to write a quick and dirty guide and it’s already very big!)

Find where both .text and .data are loaded.

$ cat /sys/module/l2cap/sections/.text
0xe0c77000
$ cat /sys/module/l2cap/sections/.data
0xe0c7b438

Then you load apart from the .text section the .data too.

(gdb) add-symbol-file l2cap.o 0xe0c77000 -s .data 0xe0c7b438
add symbol table from file "l2cap.o" at
        .text_addr = 0xe0c77000
        .data_addr = 0xe0c7b438
(y or n) y
Reading symbols from /home/fotisl/programs/kgdb/l2cap.o...done.
(gdb)

You’re now ready to start debugging your kernel module!

Kgdb is a source level debugger for the linux kernel. It requires two machines, one running a kernel compiled with kgdb enabled and the second one running gdb. It can be found at sourceforge and a light version has been merged into the 2.6.26 kernel. There is an article at kerneltrap which contains all the appropriate information about this light version and it’s differences from the full one. I am going to describe how you can debug a linux kernel running under VirtualBox using the kgdb-light debugger.

First of all you must define a serial port. Go to the settings of your virtual machine, then at the “Serial Ports” and enable “Port 1″. Use port number COM1, port mode ‘Host Pipe’, check ‘Create Pipe’ and enter a path, e.g. /home/fotisl/virtualbox/myvm/serial1. You can use another port number, e.g. COM2, but then you’ll have to change the device below to ttyS1, ttyS2 for COM3 etc. Furthermore, you can create the pipe yourself and not automatically using:

$ mkfifo /home/fotisl/virtualbox/myvm/serial1

At your virtual machine you must have a kernel compiled with the option CONFIG_KGDB. You can find this under the “Kernel debugging” menu. I also advise you to enable the CONFIG_DEBUG_INFO to insert debug symbols.

At the host machine you only need to install socat and of course gdb. Socat is a multipurpose relay which can be found here. You should also transfer the uncompressed image of the kernel running at the vm. It can be found at the directory where you compiled the kernel and it’s name will be vmlinux.

You are now ready to start. At the host machine run:

$ socat -d -d /home/fotisl/virtualbox/myvm/serial1 pty:
2009/01/01 00:00:00 socat[12345] N opening connection to AF=1 "/home/fotisl/virtualbox/myvm/serial1"
2009/01/01 00:00:00 socat[12345] N successfully connected from local address AF=1 "\x04\b\xAB"
2009/01/01 00:00:00 socat[12345] N successfully connected via \xD0\xA7\x10
2009/01/01 00:00:00 socat[12345] N PTY is /dev/pts/4
2009/01/01 00:00:00 socat[12345] N starting data transfer loop with FDs [3,3] and [4,4]

You must note the PTY, in this case /dev/pts/4. Now fire gdb and load vmlinux. Then set the remote baud to 115200 and attach to the serial port.

$ gdb ~/vmlinux
GNU gdb (GDB) 6.8.50.20090628-cvs-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
(gdb) set remotebaud 115200
(gdb) target remote /dev/pts/4
Remote debugging using /dev/pts/4

Now switch to the virtual machine. You must first set the serial port that kgdb will use.

# echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc

You’re ready to start debugging! When you want to break use the Alt-SysRq-G key combination or use

# echo g > /proc/sysrq-trigger

If you want to start the debugging when the kernel starts loading, append

kgdboc=ttyS0,115200 kgdbwait

to the command line parameters of the kernel. You must use this order! First you must register the I/O driver and then kgdb will be able to wait.

You can now explore the linux kernel! Warning, messing with various structures and executing code that you shouldn’t can cause kernel panics and mess up your virtual machine! But you already know that, that’s why you use virtualbox!