Kernel Module Debugging Techniques

There are several Debugging Techniques, few efficient Debugging techniques are listed Below. For kernel Module debugging techniques are classified in below categories.

1. Debugging by Printing
2. Debugging by watching
3. Query/Request based Debugging(On request Based)

*Debugging by printing (by using prink and adjusting Log levels)

Printing an essential data on standard output or log file helps to debug module. By looking appropriate information on console we will be able to know driver/device is at which state and what are the values of important variables. Printing is almost common debugging technique used in a common ‘C/C++’ programs. it’s easy to use and no external things are required. In User space program to print data on standard output device programmers use “printf” API where depending on format specifies different data can be printed. Same scenario follows in kernel space too but API is “printk” and it comes with some additional functionality.

-> printk: printk allows to classify devices messages according to the severity of messages by associating different log levels or priority. Loglevel or priority can be decided by giving Macro in printk as shown below. KERN_INFO is information log-level macro. There are eight different log levels available. Macro is internally string so printing log string and macro string get concatenate that’s why ‘,’ (comma) is not there

printk(KERN_INFO“dummy Log”);

Depending upon Log level, the kernel may print the messages to the current console.If no log level messages then kernel it self select a configured default log level.

Note: If driver/device is used frequently by system then to many prints make device slow down in performance.

Industry Scenario: At the time of development multiple prints in device are added and once driver is working fine as expected. Several time tested and verified then developers disable the logs so performance issue or so many prints will not occur. But if system is on field and you have bare minimum access to system at that time if issue occurs then again need to enable log and reproduce the issue then fix the issue again disable log a and load device which will make debugging process complex.

Advantages:

1. Easy to use, no external Interface required.

Disadvantages:

1. Makes Field Debugging Difficult, Too many prints in production code make your system/device slow down.

2. printing Large Number of data will not be easy.(ex- 4Kbyte of data due to buffer limit)

Debugging by watching

* strace method (Watching trace of every API of driver)

strace is a diagnostic, debugging and instructional user space utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, In the simplest case strace run s the specified command until it exits. Strace simply intercepts all function call and system calls which are called by a process or a signal received by process. It prints data on standard error of file specified with option -o. Strace usually print function or system call name, passing argument and return values

programmers will find that since system calls and signals are events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

We can run “strace” command on driver files like “/dev” Node and “/proc” files so open, read,write,release function call backs will be printed with there argument and return type. Which helps for debugging purpose

In Practical scenario you may see in strace output too many calls happened that is because you might be using command or any program to read from file, which will read so it will also make few system calls to read from the file. You will get combined output we need to separate driver / kernel module made function call.

*Dump_stack Method

Using the dump_stack() function is as easy as calling dump_stack() wherever you wish to print out the stack. This will cause a stack trace to be printed at that point. In “dmesg” or “/var/log/syslog” we can see the stack trace of point where dump_stack is called.

“dump_stack” is a function call which will dump the current stack on syslog file which can be seen in dmesg. This will help to understand at perticular point what stack is holding.

Below are supporting logs for dump_stack() call. Here dump_stack() is called in module __init function.

Supporting Logs:

Query/Request based Debugging(On request Based)

*Procfs (Proc file system to get data from driver)

*seq_file interface (To Handle large debug Data)

procfs is a Virtual file system it is also known as pseudo file system, it’s file size appear as zero. Files under /proc directory are all proc files registered by kernel modules or kernel it self. Accessing Those files indirectly invoke internal proc VFS registered function call back of kernel modules.

When amount of output Data enlarges and it is not feasible to get data by proc read call back then sequential file interface is used. This interface is bit complicate to understand. But it is worth to learn and use.

To implement seq file interface at first we need to create “proc file” which mean we need to register for proc file system. Registration to proc file system can be done by calling API “proc_create()”. This will create a proc file under parent directory given by argument or by default it take “/proc” as a parent directory. proc_create "Create" File and make entry in proc VFS (pseudo File system)

When ever created proc file will be read/write registered call back will be invoked.

Now we are registered to procfs, As mentioned file_operation we will not define read, close function call back because that is not needed in this case, those default definition will be taken care by header file, At this time we will only define “open” file operation for Procfs. In open call back we will register for another set of function call which is for seq_file interface. To be clear this seq_file operations are different set of call backs of type “struct seq_operations”. By seq_open API we resister to seq_file layer of kernel.

The call backs are set of iterations.

static struct seq_operations scull_seq_ops = {

.start = mydrv_seq_start,

.next = mydrv_seq_next,

.stop = mydrv_seq_stop,

.show = mydrv_seq_show,

};

when ever created proc file will be accessed, whose proc open call back is replaced with seq_file interface layer function calls. So iteration function calls will be take place. Operations will be in below Manners.

start and next function call are to do iterative operation for example if linked list is there, Head pointer will be returned by start function call then show function call will be invoked which transfer the information to user space. To transfer the information to user space below API calls are used at seq_file interface layer.

using "seq_printf()" we can pass info. to user space, via sequence file layer's kernel buffer

seq_printf(struct seq_file *sfile,const char *, ...);

ex: seq_printf(sfile, “my Driver No-=%d”,1);

here sfile is seq_file layer object which holds the kernel buffer. we are copying data from a current object of navigation into kernel buffer of *sfile.

Similar API

seq_putc(struct seq_file *sfile, char c, ...);

seq_puts(struct seq_file *sfile,const char *s);

General Notes:

1. Few More debugging option we can find under kernel hacking option. Kernel Hacking option is present under kernel config File. Once Configurations are done need to build and load kernel once again

2. Like GNU debugger gdp kernel debugger is also present Known as kdb(Kernel debugger). This is not an official kdp form linux community it’s a patch from “oss.sgi.com”. To use kdb we must obtain the patch, apply it, rebuild and reinstall the kernel. Once it is done successfully we can enter into kdb and start debugging is like gdb only but commands will be different (As this is not an official patch kdb details are not added).

3. procfs interface is changed after kernel 3.10 release, if you are referring to any any other reference please cross check your kernel version other wise you will end up with compiler error.