Chapter 4. Debugging Techniques

⚠️ syslogd is considered deprecated and largely replaced by rsyslogd.

4.1 Debugging support in the Kernel

A lot of debugging related kernel configurations are introduced in the book. And followings are my kernel configurations.

CONFIGURATIONS MENTIONED IN LDD	MY SELECTION
`CONFIG_DEBUG_KERNEL`	Y
`CONFIG_DEBUG_SLAB`	SLAB is deprecated and replaced by SLUB. Hence, CONFIG_SLUB_DEBUG and CONFIG_SLUB_DEBUG_ON is enabled
`CONFIG_DEBUG_PAGEALLOC`	Y
`CONFIG_DEBUG_SPINLOCK`	Y
`CONFIG_DEBUG_SPINLOCK_SLEEP`	not found
`CONFIG_INIT_DEBUG`	not found
`CONFIG_DEBUG_INFO`	Y
`CONFIG_MAGIC_SYSRQ`	Y
`CONFIG_DEBUG_STACKOVERFLOW`	N
`CONFIG_DEBUG_STACK_USAGE`	Y
`CONFIG_KALLSYMS`	Y (by default)
`CONFIG_ACPI_DEBUG`	Y (by default)
`CONFIG_DEBUG_DRIVER`	Y
`CONFIG_SCSI_CONSTANTS`	Y (by default)
`CONFIG_INPUT_EVBUG`	m (by default)
`CONFIG_PROFILING`	Y (by default)

Please feel free to use .config.debug as your linux .config.

4.2 Debugging by Printing

💡 Full code in Debugging by Printing

The way I enable the debugging options for scull device is a bit different from the one shown in the book. As mentioned in Chapter2, it relies on the kbuild system.

`scull/Kconfig`

Boolean symbol of CONFIG_DEBUG_SCULL is defined in the Kconfig of scull directory as below.

Note that both CONFIG_DEBUG_KERNEL and CONFIG_SCULL should be properly set to enable the configuration of CONFIG_DEBUG_SCULL

config SCULL
    tristate "Scull driver by Gaby, Kim"
    depends on LDD
    help
        This is scull device driver based on Linux Device Drvier, 3rd edition,
        compatible with linux 6.7.0, Authored by Gaby, Kim.

config DEBUG_SCULL
    bool "Enable debugging features of scull driver by Gaby, Kim"
    depends on DEBUG_KERNEL && SCULL
    help
        To enable debugging features of this driver, choose y here.

We can see that make menuconfig shows the required dependencies to set CONFIG_DEBUG_SCULL.

menuconfig

`Makefile`

When CONFIG_DEBUG_SCULL=y, we have to define the SCULL_DEBUG so that debugging macros in scull.h are activated. It can be done by a simple line of ccflags-$(CONFIG_DEBUG_SCULL) in Makefile.

# If KERNELRELEASE is defined, we've been invoked from the
# kernel build system and can use its language.
ifneq ($(KERNELRELEASE),)
	obj-$(CONFIG_SCULL) += scull.o
	scull-y := main.o
	ccflags-$(CONFIG_DEBUG_SCULL) := -O -g -DSCULL_DEBUG

...

You can find the documents regarding ccflags in https://github.com/torvalds/linux/blob/master/Documentation/kbuild/makefiles.rst#compilation-flags

`scull.h`

With the SCULL_DEBUG defined, debugging macros in scull.h will be defined upon pre-processing. Note, the PDEBUG is modified so that it also prints out the name of the function through __func__ for verbose debugging messages.

/*
 * Macros to help debugging
 */
#undef PDEBUG             /* undef it, just in case */
#ifdef SCULL_DEBUG
#  ifdef __KERNEL__
     /* This one if debugging is on, and kernel space */
#    define PDEBUG(fmt, args...) printk( KERN_DEBUG "scull: [%s] " fmt, __func__, ## args)
#  else
     /* This one for user space */
#    define PDEBUG(fmt, args...) fprintf(stderr, "[%s] " fmt, __func__, ## args)
#  endif
#else
#  define PDEBUG(fmt, args...) /* not debugging: nothing */
#endif

4.3 Debugging by Querying

💡 Full code in Debugging by Querying

`proc filesystem` without `seq_file` interface

A lot of changes have been made into proc filesystem API in modern Linux kernel.

create_proc_read_entry() is replaced with proc_create().
remove_proc_entry() is replaced with proc_remove().
struct proc_ops is used instead of struct file_operations upon /proc file creation.
Function signature of new proc_ops.proc_read() differs a lot from the one of original file_operations.read().

Other than the changes due to Linux version update, I also slightly refactored some logics of scull_read_procmem().

use copy_to_user() instead sprintf() for safe data transfer
scull_read_procmem returns at most one quantum of data at a time, just like the scull_read.
cautiously monitor the number of bytes copying to user to prevent larger return data than the requested one.

Since the way scull_read_procmem delivers the data to user is much of the same with the one of scull_read, code in original scull_read has been refactored into scull_read_util to minimize the code duplication.

ssize_t scull_read_util(struct scull_dev* dev, char __user *buf, size_t count, loff_t *f_pos)
{
	PDEBUG("called with count: %lu, f_pos: %lld\n", count, *f_pos);
	// case 1. offset is alreay beyond EOF
	if (*f_pos >= dev->size)
		return 0;
	
	// case 2. offset is within EOF, while offset + count is beyond EOF
	if (*f_pos + count > dev->size)
		count = dev->size - *f_pos;

	// case 3. offset + count is within EOF, look for the target quantum to copy to user space
	int list_node_size = dev->qset * dev->quantum;
	// locate the q_set list node
	int qset_pointer_offset = *f_pos / list_node_size;
	struct scull_qset *target = scull_follow(dev, qset_pointer_offset);

	// then the quantum pointer, finally pinpoint the f_pos within the quantum
	int quantum_pointer_offset = (*f_pos % list_node_size) / dev->quantum;
	int quantum_offset = (*f_pos % list_node_size) % dev->quantum;
	
	if (!target || !target->data || !target->data[quantum_pointer_offset])
		return 0;

	// read only up to the end of this quantum
	count = min(count, (size_t)dev->quantum - quantum_offset);
	if (copy_to_user(buf, target->data[quantum_pointer_offset] + quantum_offset, count))
		return -EFAULT;

	*f_pos += count;
	PDEBUG("read %lu bytes\n", count);

	return count;
}

Our scull_read_procmem simply loops through all the devices and transmits all internal contents until the EOF of last scull device.

/*
 * The proc filesystem: function to read and entry
 */
static ssize_t scull_read_procmem(struct file *filp, char __user *buf, size_t count,  loff_t *offset)
{
	PDEBUG("called with count: %lu, offset: %lld\n", count, *offset);
	loff_t device_offset = *offset;
	ssize_t len = 0;
	for (int i = 0 ; i < scull_nr_devs && len <= count; ++i) {
		struct scull_dev *device = &scull_devices[i];
		if (down_interruptible(&device->sem))
			return -ERESTARTSYS;

		// skip to next device if needed
		if (device_offset >= device->size) {
			device_offset -= device->size;
			up(&device->sem);
			continue;
		}

		len = scull_read_util(device, buf, count, &device_offset);
		// scull_read_util might return -EFAULT
		if (len >= 0) {
			*offset += len;
			PDEBUG("read %ld bytes in total", len);
		}
		up(&device->sem);
		break;
	}

	return len;
}

static const struct proc_ops scull_proc_fops = {
	.proc_read = scull_read_procmem,
};

FYI, the refactored version of scull_read is as follows.

ssize_t scull_read(struct file *filp, char __user *buf, size_t count, loff_t *f_pos)
{
	ssize_t retval = 0;
	struct scull_dev *dev = filp->private_data;

	if (down_interruptible(&dev->sem))
		return -ERESTARTSYS;

	retval = scull_read_util(dev, buf, count, f_pos);

	up(&dev->sem);
	PDEBUG( "read %ld bytes from scull device", retval);

	return retval;
}

`proc filesystem` with `seq_file` interface

Fortunately, seq_file interface didn’t change that much. Our implementation is almost the same with the one in the book.

static void *scull_seq_start(struct seq_file *s, loff_t *pos)
{
	if (*pos >= scull_nr_devs)
		return NULL;
	seq_printf(s, "iterate starts at /device/scull%lld\n", *pos);
	return scull_devices + *pos;
}

static void *scull_seq_next(struct seq_file *s, void *v, loff_t *pos)
{
	if (*++pos >= scull_nr_devs)
		return NULL;
	seq_printf(s, "iterate to /device/scull%lld\n", *pos);
	return scull_devices + *pos;
}

static int scull_seq_show(struct seq_file *s, void *v)
{
	struct scull_dev *device = v;
	if (down_interruptible(&device->sem))
		return -ERESTARTSYS;

	struct scull_qset *qset = device->data;
	for (; qset; qset = qset->next) {
		if (!qset->data)
			continue;
		for (int j = 0; j < device->qset; ++j) {
			if (!qset->data[j])
				continue;
			seq_printf(s, (const char *)qset->data[j]);
		}
	}

	up(&device->sem);
	return 0;
}

static void scull_seq_stop(struct seq_file *s, void *v)
{
	/* Actually, there's nothing to do here */
}

static const struct seq_operations scull_seq_ops = {
	.start = scull_seq_start,
	.stop = scull_seq_stop,
	.next = scull_seq_next,
	.show = scull_seq_show,
};

static int scull_proc_open(struct inode *inode, struct file *filp)
{
	return seq_open(filp, &scull_seq_ops);
}

static const struct proc_ops scull_seq_proc_ops = {
	.proc_read = seq_read,
	.proc_open = scull_proc_open,
	.proc_release = seq_release,
	.proc_lseek = seq_lseek,
};

With proc_ops ready, we can now create or remove the /proc files with following function calls.

static void scull_create_proc(void)
{
	scull_proc_entry = proc_create(SCULL_PROC, 0644, NULL, &scull_proc_fops);
	if (!scull_proc_entry)
		PDEBUG("/proc/%s not created", SCULL_PROC);

	scull_seq_proc_entry = proc_create(SCULL_SEQ_PROC, 0644, NULL, &scull_seq_proc_ops);
	if (!scull_seq_proc_entry)
		PDEBUG("/proc/%s not created", SCULL_SEQ_PROC);
}

static void scull_remove_proc(void)
{
	proc_remove(scull_proc_entry);
	proc_remove(scull_seq_proc_entry);
}

Don’t forget to call these functions upon module loading and unloading!

void scull_cleanup_module(void)
{
...
#ifdef SCULL_DEBUG
	scull_remove_proc();
#endif
...
}

static int __init scull_init_module(void)
{
...
#ifdef SCULL_DEBUG
	scull_create_proc();
#endif
...
}

Test `/proc` files

write_and_query is a simple shell script to write data into all /dev/scull[0-3] and read both /proc/scullproc and /proc/scullseq

#!/bin/sh
device="/dev/scull"
proc_file="/proc/scullproc"
proc_seq="/proc/scullseq"

for i in 0 1 2 3; do
    echo "Hello from scull${i}" | sudo dd of="${device}${i}"
done

read_file() {
    file=$1
    if [ -e "${file}" ]; then
        echo "read from" ${file}
        cat ${file}
    else
        echo "${file} does not exit"
    fi
}


read_file "${proc_file}"
read_file "${proc_seq}"

write_and-query result

⚠️Remaining sections regarding other debugging techniques are not fully experimented, hence, left undocumented.

4.1 Debugging support in the Kernel

4.2 Debugging by Printing

scull/Kconfig

Makefile

scull.h