How to Debug the Linux Kernel with QEMU and Libvirt

How can you verify that your Linux kernel image will boot on real hardware? Booting your image with virtualization technologies can’t provide you with an absolute guarantee that your software will run on metal. However, QEMU and Libvirt give you enough capabilities to conduct a quick sanity check and get rid of most bugs at the development stage.

In this article, we explain how you can debug your Linux kernel and its modules during runtime. This article will be useful for Linux kernel developers who want to speed up the time to market of their software.

Environment
Debugging without source code
Debugging with source code
Debugging with third-party modules
Conclusion

We’ll provide you with step-by-step instructions on how you can find errors in your Linux kernel module or kernel image at the development stage. Our method includes three levels:

Debugging without source code
Debugging with source code
Debugging with third-party modules

You can either use one of these levels or dig deep and go through the whole process of kernel debugging. If you need to look for errors in Windows application, check out our guide to using WinDbg for kernel debugging.

Environment

To conduct kernel debugging, you should prepare the necessary environment. First of all, you need to set up the guest target.

The guest is a Libvirt or QEMU-style virtual machine that we’ll debug. The host is an operating system where debugging is taking place.

To create an environment, we use Ubuntu 16.04 as the host operating system and Ubuntu 16.04 as the guest operating system.

However, you can also use other guest Linux distributions. Just make sure to install the corresponding packages beforehand.

Here’s what you need to install for debugging:

QEMU as a hypervisor and emulator for hardware virtualization
Libvirt as a daemon and a toolkit for managing platform virtualization
GDB as a project debugger
A QEMU virtual machine (in our case, it’s Ubuntu 16.04 under the name 04)

Need custom drivers to bring out the full potential of your product?

Reach out to our kernel and driver developers to improve your system’s stability, performance, and efficiency!

Debugging without source code

Debugging without the source code is the simplest way to quickly debug the kernel if you need to look at the call stack and variable values. You can verify the kernel image in this way:

Configure Libvirt on the host operating system with the following command:

ShellScript

virsh edit ubuntu16.04

In your editor of choice, you can open the file with virtual machine settings. Find the domain tag and add the following option:

XML

<domain type='kvm'
        xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' >
 	<qemu:commandline>
      	<qemu:arg value='-s'/>
 	</qemu:commandline>

Save the file and exit the editor. Now you need to restart the virtual machine (if it’s already running).

The -s parameter means that QEMU will open port 1234 for debugging by default. If you need to open other ports, you can set them in the domain XML file:

XML

<qemu:commandline>
    	<qemu:arg value='-gdb'/>
    	<qemu:arg value='tcp::1235'/>
  </qemu:commandline>

The specified port should open after the virtual machine starts.

Disable Kernel Address Space Layout Randomization (KASLR) on the guest. To do this, you need to apply changes directly to the GRand Unified Bootloader (GRUB) configuration. Open /etc/default/grub and add nokaslr to the GRUB_CMDLINE_LINUX_DEFAULT variable:

ShellScript

GRUB_CMDLINE_LINUX_DEFAULT="splash quiet nokaslr"

Update GRUB with this command:

ShellScript

sudo update-grub

and reboot the guest operating system.

The Linux kernel packages do not include debug symbols, as they’re stripped from binary files at build time. However, you need to install debug symbols to let GDB assist you during debugging. To install debug symbols for the existing kernel on the guest, see the detailed description here.

Here how we’ll install debug symbols on Ubuntu:

ShellScript

#GPG key import
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
#Add repository config
codename=$(lsb_release -c | awk  '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}  	main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF
 
sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

You can manually download and install the ddeb package from here as an example. However, be careful to install a package with the compatible kernel and architecture.

Here’s how to install debug symbols on Fedora Linux:

ShellScript

sudo dnf debuginfo-instal kernel-$(uname -r)

You can find a complete description of how to install debug symbols on Fedora Linux here.

Copy the /usr/lib/debug/boot/vmlinux-$(uname -r) file from the guest to any folder on the host so GDB can use it. In our case, we have the following file: /usr/lib/debug/boot/vmlinux-4.13.0-36-generic
Run the QEMU virtual machine on the host and connect it to the debugger:

ShellScript

gdb ./vmlinux-4.13.0-36-generic
(gdb) target remote :1234

Here’s an example of Linux kernel debugging for Ubuntu 16.04 (4.13.0-36):

ShellScript

kernel_new# gdb ./vmlinux-4.13.0-36-generic
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./vmlinux-4.13.0-36-generic...done.
(gdb) target remote :1234
Remote debugging using :1234
native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54
54  /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h: No such file or directory.
(gdb) bt
#0  native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54
#1  0xffffffff8191e20e in arch_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/paravirt.h:92
#2  default_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/process.c:354
#3  0xffffffff81036b45 in arch_cpu_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/process.c:345
#4  0xffffffff8191e743 in default_idle_call () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:98
#5  0xffffffff810cd5f2 in cpuidle_idle_call () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:156
#6  do_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:246
#7  0xffffffff810cd853 in cpu_startup_entry (state=CPUHP_ONLINE) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:351
#8  0xffffffff81910e78 in rest_init () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/init/main.c:437
#9  0xffffffff824a514d in start_kernel () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/init/main.c:702
#10 0xffffffff824a42d5 in x86_64_start_reservations (real_mode_data= <optimized out>) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head64.c:318
#11 0xffffffff824a4417 in x86_64_start_kernel (real_mode_data=0x8a000 <error: Cannot access memory at address 0x8a000>) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head64.c:299
#12 0xffffffff810000cf in secondary_startup_64 () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head_64.S:220
#13 0x0000000000000000 in ?? ()
(gdb) break do_dentry_open
Breakpoint 1 at 0xffffffff812528e0: file /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c, line 704.
(gdb) c
Continuing.
 
Thread 1 hit Breakpoint 1, do_dentry_open (f=0xffff88007b1a4a00, inode=0xffff88007a0a1008, open=0x0 <irq_stack_union>, cred=0xffff8800775eea80) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c:704
704 /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c: No such file or directory.
(gdb) print inode->i_ino
$1 = 4026532024
(gdb)

Anti Debugging Protection Techniques with Examples

Protect your software from malicious reversing. Leverage our expertise in reverse engineering and implement a set of techniques that will keep your code safe from hackers.

Learn more

Debugging with source code

Source code allows the debugger to associate events and circumstances in program execution with their corresponding location. So using the source code, you can run the (gdb) list command without errors like “No such file or directory.” If you have a source code file, make it available to GDB for kernel debugging in the following way:

Load the source code to the host (see detailed description here). You can read how to download the source code from the git repository here.

ShellScript

git clone git://kernel.ubuntu.com/ubuntu/xenial-maverick.git
# find for kernel 4.13.0-36
git tag -l | grep 4.13.0-36
# Ubuntu-hwe-4.13.0-36.40_16.04.1
git checkout -b ubuntu-hwe-4.13.0-36 Ubuntu-hwe-4.13.0-36.40_16.04.1

Connect the source code to GDB with the (gdb)set substitute-path command:

ShellScript

gdb ./vmlinux-4.13.0-36-generic
(gdb) set substitute-path /build/linux-hwe-4GXcua/linux-hwe-4.13.0 /media/veracrypt2/linux_kernel_new/ubuntu-xenial

If you look at the previous logs, you can see the path to the source code in the debug symbols that is /build/linux-hwe-4GXcua/linux-hwe-4.13.0

ShellScript

/build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h: No such file or directory.

Substitute the original source code path with the path to our source code using the set substitute-path command. In our case, the new path is /media/veracrypt2/linux_kernel_new/ubuntu-xenial.

Related project

Device Firmware Reversing to Obtain the Screen Mirroring Protocol

Discover how Apriorit’s reverse engineers efficiently analyzed the original firmware, reconstructing the secure connection process. This saved time and money for the client, enabling quick cross-platform compatibility for video mirroring without developing new functionality.

Project details

Debugging with third-party modules

This method is for those who want to debug a module that’s not included in the official Linux kernel version. For instance, we’ll try to debug our own module.

Build the module with debug symbols on the guest and install it. To build the module with debug symbols, add the -g flag (EXTRA_CFLAGS += -g) to the compiler options. Copy the built module to the host folder together with the debug symbols (or to the folder with the source code). In our case, we get this:

ShellScript

ls /media/veracrypt2/linux_kernel_new
myspecificmodule.ko ubuntu-xenial vmlinux

Build the GDB scripts on the host so we can debug modules that are loaded. Unfortunately, scripts are provided only with the whole kernel, so we need to stop the building process after we get the debian/build/build-generic/scripts/gdb/linux/constants.py file (you can find more information about kernel building here).

ShellScript

sudo apt build-dep --yes linux-image-4.13.0-36-generic   	
sudo apt install --yes fakeroot libncurses5-dev
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic binary-perarch

If you want to continue building after interruption, make an additional link to the script:

ShellScript

ln -s /media/veracrypt2/linux_kernel_new/ubuntu-xenial/scripts/gdb/vmlinux-gdb.py /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/build/build-generic/vmlinux-gdb.py

Permit the GDB to load. Create the ~/.gdbinit file as a user on the host:

ShellScript

$ cat ~/.gdbinit
add-auto-load-safe-path /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/build/build-generic/vmlinux-gdb.py

The file with debug symbols should be named vmlinux to make our scripts runnable, so rename this file on the host:

ShellScript

mv vmlinux-4.13.0-36-generic vmlinux

Run GDB and make sure that everything is okay with the scripts. We can also check if it’s possible to set breakpoints in our module and debug it (see comments in the script). The breakpoint makes the debugger stop execution after a specified function or line number in the source file.

ShellScript

gdb ./vmlinux
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./vmlinux...done.
(gdb) set substitute-path /build/linux-hwe-4GXcua/linux-hwe-4.13.0 /media/veracrypt2/linux_kernel_new/ubuntu-xenial
(gdb) source /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/build/build-generic/vmlinux-gdb.py # load and check if scripts work correctly
(gdb) target remote :1234
Remote debugging using :1234
native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54
warning: Source file is more recent than executable.
54  }
(gdb) lx-symbols # load all available modules
loading vmlinux
scanning for modules in /media/veracrypt2/linux_kernel_new
no module object found for 'snd_hda_codec_generic'
no module object found for 'snd_hda_intel'
no module object found for 'snd_hda_codec'
no module object found for 'snd_hda_core'
no module object found for 'snd_hwdep'
no module object found for 'snd_pcm'
no module object found for 'crct10dif_pclmul'
no module object found for 'crc32_pclmul'
no module object found for 'ghash_clmulni_intel'
no module object found for 'pcbc'
no module object found for 'snd_seq_midi'
no module object found for 'snd_seq_midi_event'
no module object found for 'aesni_intel'
no module object found for 'aes_x86_64'
no module object found for 'snd_rawmidi'
no module object found for 'crypto_simd'
no module object found for 'glue_helper'
no module object found for 'cryptd'
no module object found for 'snd_seq'
no module object found for 'snd_seq_device'
no module object found for 'snd_timer'
no module object found for 'input_leds'
no module object found for 'snd'
no module object found for 'joydev'
no module object found for 'serio_raw'
no module object found for 'i2c_piix4'
no module object found for 'soundcore'
no module object found for 'mac_hid'
no module object found for 'parport_pc'
no module object found for 'ppdev'
loading @0xffffffffc0151000: /media/veracrypt2/linux_kernel_new/myspecificmodule.ko # our module is loaded 
no module object found for 'lp'
no module object found for 'parport'
no module object found for 'autofs4'
no module object found for 'qxl'
no module object found for 'ttm'
no module object found for 'drm_kms_helper'
no module object found for 'syscopyarea'
no module object found for 'sysfillrect'
no module object found for 'sysimgblt'
no module object found for 'fb_sys_fops'
no module object found for 'psmouse'
no module object found for 'virtio_blk'
no module object found for 'virtio_net'
no module object found for 'floppy'
no module object found for 'drm'
no module object found for 'pata_acpi'        
(gdb) break do_dentry_open_operation_handler # breakpoint on the function of our module 
Breakpoint 1 at 0xffffffffc0154c50: file /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c, line 118.
(gdb) c
Continuing.
[Switching to Thread 2]
 
Thread 2 hit Breakpoint 1, do_dentry_open_operation_handler (file=0xffff880073d2c300, inode=0xffff88007af766e8, open=0x0 <irq_stack_union>, cred=0xffff8800737ff900)
    at /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c:118
118 /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c: No such file or directory.
 
(gdb)

The line “no module object found for” means that there are no other modules that can be loaded. If the debugger can’t find source code files, you can manually add links to the source code for the modules. If any modules are available on the machine, you’ll see the following:

ShellScript

(gdb) lx-symbols # load all available modules
loading vmlinux
scanning for modules in /media/veracrypt2/linux_kernel_new
loading @0xffffffffc02bb000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/hda/snd-hda-codec-generic.ko
loading @0xffffffffc0240000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/hda/snd-hda-intel.ko
loading @0xffffffffc0288000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/hda/snd-hda-codec.ko
loading @0xffffffffc0273000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/hda/snd-hda-core.ko
loading @0xffffffffc023a000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-hwdep.ko
loading @0xffffffffc0250000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-pcm.ko
loading @0xffffffffc0235000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/crct10dif-pclmul.ko
loading @0xffffffffc022b000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/crc32-pclmul.ko
loading @0xffffffffc026e000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/ghash-clmulni-intel.ko
loading @0xffffffffc01f6000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/pcbc.ko
loading @0xffffffffc01f1000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/seq/snd-seq-midi.ko
loading @0xffffffffc01bc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/seq/snd-seq-midi-event.ko
loading @0xffffffffc01fc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/aesni-intel.ko
loading @0xffffffffc01c2000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/aes-x86_64.ko
loading @0xffffffffc01e8000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-rawmidi.ko
loading @0xffffffffc01e3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/crypto_simd.ko
loading @0xffffffffc01b7000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/glue_helper.ko
loading @0xffffffffc01dc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/cryptd.ko
loading @0xffffffffc01cb000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/seq/snd-seq.ko
loading @0xffffffffc01a6000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-seq-device.ko
loading @0xffffffffc01ae000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-timer.ko
loading @0xffffffffc0182000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/input-leds.ko
loading @0xffffffffc0191000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd.ko
loading @0xffffffffc0187000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/joydev.ko
loading @0xffffffffc017d000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/serio/serio_raw.ko
loading @0xffffffffc00d3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/i2c/busses/i2c-piix4.ko
loading @0xffffffffc009f000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/soundcore.ko
loading @0xffffffffc0071000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/macintosh/mac_hid.ko
loading @0xffffffffc0174000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/parport/parport_pc.ko
loading @0xffffffffc016a000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/char/ppdev.ko
loading @0xffffffffc0151000: /media/veracrypt2/linux_kernel_new/myspecificmodule.ko
loading @0xffffffffc008d000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/char/lp.ko
loading @0xffffffffc0144000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/parport/parport.ko
loading @0xffffffffc0066000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/fs/autofs4/autofs4.ko
loading @0xffffffffc0133000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/qxl/qxl.ko
loading @0xffffffffc011b000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/ttm/ttm.ko
loading @0xffffffffc00f1000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/drm_kms_helper.ko
loading @0xffffffffc0061000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/syscopyarea.ko
loading @0xffffffffc00ea000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/sysfillrect.ko
loading @0xffffffffc00e3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/sysimgblt.ko
loading @0xffffffffc00de000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/fb_sys_fops.ko
loading @0xffffffffc00ae000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/mouse/psmouse.ko
loading @0xffffffffc00a4000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/block/virtio_blk.ko
loading @0xffffffffc0094000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/net/virtio_net.ko
loading @0xffffffffc0079000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/block/floppy.ko
loading @0xffffffffc0008000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/drm.ko
loading @0xffffffffc0000000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/ata/pata_acpi.ko

If you want to check other modules, you can add links to the source code for them in the same way as we did in the first step:

ShellScript

(gdb) set substitute-path /home/quest/Desktop/mymodule /media/veracrypt2/linux_kernel_new/mymodule

Conclusion

Debugging a Linux kernel image or module is a complex task that many developers face. Using such virtualization technologies as QEMU and Libvirt allow them to detect and fix the majority of defects at the development stage.

In this article, we showed you three levels of debugging: debugging with and without source code and debugging with the help of third-party modules. Depending on the task at hand, you can either debug your kernel image or module on one of these levels or go through the whole process.

Here at Apriorit, we have a team of passionate kernel and driver develo pers with experience in Linux kernel debugging, bootloader programming, and more. We will be glad to assist you in creating your own software solution for the Linux platform.

Looking for a dedicated kernel development team?

Benefit from our extensive 20-year track record in specialized kernel development to boost your software quality and reliability!

Have a question?

Ask our expert!