Class 4: kernel compilation

Date: 18.03.2025

Small Taks #4

Tip

The Linux version fixed for the labs is 6.12.6. Some useful links:

Kernel Development

You can read in detail about the Kernel Development Process here: https://www.kernel.org/doc/html/latest/process/development-process.html

You can check the kernel version you are running with:

uname -r

Linux uses a highly distributed approach to development facilitated by git trees. In short, you may encounter:

Linus's mainline tree

The tree maintained by Linus Torvalds is the top destination of changes. It uses an X.Y-rcZ format to indicate consecutive release candidates for upcoming versions. (These are not Semantic Versioning numbers -- it is inadequate to kernels.) Once most of most new bugs are resolved, the tree becomes stable and is maintained by another team.

Stable trees

Stable trees use three-part revisions: X.Y.ZZ and contain critical fixes after the release. Some of them are considered for long-term support by the stable team.

Subsystem trees

The development effort of Linux is distributed along subsystems. Maintainers of various subsystems do the day-to-day management of changes from thousands of developers, collecting them in their trees to present them for inclusion in bulk. These version often contain a suffix like -mm or -bpf.

The -next tree

This is a special tree used for integration testing of the multitude of the aforementioned forks.

Distributions of the kernel

If you are not compiling the Kernel from the main sources, your version will likely have a lot of suffixes which are vendor specific. Depending on the platform, finding the exact source used to build your kernel may be a tedious process.

Downloading kernel sources

Hands-on

You will need sources of the Linux Kernel for today's labs. Get them by either:

  • On the QEMU image, just go to /home/zso/linux-6.12.6.

  • Alternatively, clone the git repository from git.kernel.org (repository linux/kernel/git/linux-stable) or download and unpack tarball of version 6.12.6 from https://www.kernel.org/. If using git, check out tag v6.12.6.

Important

Compiling the sources from /hshare may not work.

The easiest way to get kernel sources is to download a compressed .tar file from https://kernel.org/, which is the official kernel release archive. In addition to complete packages with a source, .patch files are also published, allowing you to update the previously downloaded package to a newer version.

A bit more complicated, but a much more flexible way of getting the sources is using git. This allows you to work on the latest code (not yet included in any official release), and to move almost immediately between all previously released versions included in history (from 2.6.12). Using git is also required if you send your own changes to be included in the official kernel version.

The mainline repository, maintained by Linus Torvalds and used as a basis for new kernel releases, is located at the following address:

git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

All main releases and the release candidates are available in this repository as tags (e.g., v6.12, etc.).

Stable trees are developed in a separate repository, available at the following address:

git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

If we want to work only on external kernel modules without modifying existing code, it is not necessary to have complete sources. Sufficient kernel headers, installed mostly in /usr/src/linux and symlinked to /lib/modules/<version>/build. However, they must be true kernel headers, not those intended for use by libc. The kernel version and configuration from which these headers come must also exactly match the kernel version under which the compiled modules will be used. Some distributions ship such headers in a separate package, named, for example, "linux-headers".

Distributions also often ship kernel sources as packages in the standard repository -- such sources usually contain patches applied by the distribution and are not identical to the official release.

Applying patches to the kernel source

Instead of downloading an entire new source tarball, you can apply patches (files named e.g. patch-xx.xx.xx.gz) on the old source through the patch command, e.g.

cd /usr/src/linux
gzip -cd patch.xx.xx.xx.gz | patch -p1

You can also use the patch-kernel script in the scripts directory (it automatically applies patches found in the directory from which it was launched).

The 'official' patches with the name patch-6.12.19.gz will work with the previous release (relative to the release named in the patch name), i.e., the mentioned patch will work with the 6.12 kernel. Unofficial patches (for example patch-2.6.11-ac4.gz) usually refer to the same release as in the name.

Before installing a new kernel, read the "changes" document that contains compiler version requirements, installed packages, library versions, etc. and make sure that the relevant versions are installed.

The structure of the kernel sources

Upon browsing the main directory, you will find the following entries:

Tip

LXR (Linux Cross Reference) is a very useful site to browse the sources in a browser: https://elixir.bootlin.com/linux/v6.12.6/source

The contents of the main directory

Documentation

a directory containing the documentation -- in particular, please read the process/coding-style.rst (rendered) file

LICENSES

various legal stuff

arch

source code dependent on the processor architecture

block

kernel block layer functions

certs

signing of the kernel (modules)

crypto

cryptographic functions (as well as compression and decompression)

drivers

device drivers (most of the weigh!)

fs

file systems

include

header files

init

platform independent part of the system initialization

io_uring

efficient asynchronous I/O interface

ipc

IPC (System V inter-process communication)

kernel

kernel core -- process management, interrupts, DMA, time

lib

auxiliary procedures (e.g., writing to the screen, unpacking compressed kernels)

mm

memory management

net

network protocols

rust

Rust in the Kernel <https://www.kernel.org/doc/html/v6.12/rust/index.html>_

samples

examples of the use of some internal kernel interfaces

scripts

scripts (e.g., for configuration)

security

security code (LSM -- Linux Security Modules)

sound

sound card drivers and sound system code (ALSA)

tools

various userspace tools (e.g., perf)

usr

support programs; currently gen_init_cpio used to create a ramdisk loaded with the system kernel.

virt

virtualization-related code (KVM)

arch -- code dependent on the hardware platform

The code for Intel processors is in the arch/x86 directory. It is possible to compile the kernel for a different processor than the one we work on. However, it requires (besides Linux sources) a compiler for the given platform, operating on our system (a so-called cross-compiler).

Kernel headers

The kernel contains two sets of headers: internal headers and headers for user space. The internal headers are intended for use only by kernel code and modules, and are installed in /usr/src/linux (if at all). The user space headers are in subdirectories named uapi and are intended for use by both kernel code and user programs. These headers are installed in /usr/include. Thanks to compatibility guarantees of kernel interfaces, the installed headers version may be different from the version of the kernel used.

Inside the kernel sources, headers are scattered across multiple directories:

  • include: main kernel headers

  • include/generated: main kernel headers, generated at compile time

  • arch/<processor>/include: kernel headers specific to the architecture

  • arch/<processor>/include/generated

The most important header subdirectories are asm, asm-generic and linux.

Kernel configuration

Before compilation, the kernel has to be configured with one of the following commands:

  • make config (text version, will ask one question for every option -- not recommended),

  • make menuconfig (ncurses -- may not work over virtio console of QEMU, ssh is recommended),

  • make xconfig (X11 interface),

  • make oldconfig (like make config, but updates a configuration from an older kernel version -- only asks about new options).

The simplest to use version is menuconfig or xconfig, though in the latter there are more errors.

The following points describe the essential elements of the configuration (the titles correspond to elements of the main menu in menuconfig).

Warning

Kernel options have quite complex dependencies and configuration programs do not show options that are excluded by other choices (e.g. if support for virtio devices is not selected in virtualization options, we will not see the virtual network card virtio-net option in the network drivers options at all). If we are unable to find an option where we expect it, it is worth using the search function (just press / in make menuconfig) -- the results will show us where the option is in the selection tree and what options it depends on.

General setup

This part of the configuration controls the key components of the Linux kernel. The most important options are:

  • 'System V IPC' -- handling inter-process communication

  • 'Initial RAM filesystem and RAM disk (initramfs/initrd) support' -- allows booting Linux from ramdisk loaded before running (e.g.m by GRUB), which allows you to load drivers for disks or file systems available only as modules or start the system from software-RAID devices.

  • 'Initramfs source file(s)' -- list of files to be included in the ramdisk

64-bit kernel

Selects whether the compiled kernel will run in 64-bit mode and support running 64-bit programs (running 32-bit programs is always possible, unless we explicitly disable this option in later configuration).

Processor type and features

Allows you to configure support and optimization of the kernel for a given processor (Processor family). Note -- if you select an incorrect value, the kernel may not work at all, or work erroneously. Other important options available in this menu are:

  • 'Symetric multi-processing support' -- support for multiple processors (cores).

  • 'MTRR (Memory Type Range Register) support' -- support for memory access control registers, allowing the PCI/AGP bus to be set to "write-combining" mode, which can significantly speed up graphics applications.

  • 'Randomize the address of the kernel image (KASLR)' -- support for address space randomization.

  • 'Linux guest support' -- detection and optimizations for running as a guest (e.g., under KVM).

Mitigations for CPU vulnerabilities

Allows to setup mitigations for known hardware vulnerabilities like SPECTRE.

Power management and ACPI options

Selection of supported energy saving methods -- including ACPI support and changes in processor speed during system operation.

Bus options (PCI etc.)

Selection of supported system buses and their parameters. For modern computers, it's a good idea to enable PCI bus support.

Binary Emulation

In the case of 64-bit kernels, we can enable or disable support for 32-bit programs here.

Enable loadable module support

Note

Module -- a part of the kernel code that can be loaded or removed from the kernel on demand. All parts of the kernel that are not needed at startup system and are not constantly used while the system is running should be compiled as modules. Even most parts needed at boot time can be modules, as long as an initial ramdisk is used.

Configures support for kernel modules. Depending on your needs, you can enable or disable module loading (Enable loadable module support), enable removing modules (Module unloading, Forced module unloading), enable automatic loading of modules by the kernel (Automatic kernel module loading) and enable loading modules compiled for other kernel versions by adding additional information about required functions (Module versioning support). We can also put a checksum in all modules (Source checksum for all modules).

Enable the block layer

Unless you are building the kernel for an embedded system, you want to have this option enabled. Otherwise, you will not be able to enable most of disk drivers and filesystems.

Executable file formats

Support for executable file formats. Without ELF format support, there's not much that can be done with traditional Linux distributions.

Memory Management options

Various options related to memory management, for instance:

  • 'Support for paging of anonymous memory (swap)' -- support for virtual memory on the disk

Networking support

Typically, it is not possible to disable network support (Networking support), because not much would work without it. In addition to the 'Networking options' menu described below, various communication methods can be configured here -- infrared, Bluetooth, Wi-Fi.

Networking options

This menu contains the configuration of network components and protocols. The most important of them are:

'Packet socket'

direct access to network devices.

'Unix domain sockets'

Unix sockets, enabling inter-process communication in a similar way to network communication. Such sockets are used for instance by X-Windows or PostgreSQL.

'TCP/IP Networking'

TCP/IP protocol support -- very important. But difficult to not choose.

'The IPv6 protocol'

support for the new version of TCP/IP. Currently not yet necessary, but in a while you probably will not be able to do without it.

'Network packet filtering framework (Netfilter)'

filtering and modifying packets (firewall, NAT).

Device Drivers

Various driver settings grouped into multiple menus.

If you are building a kernel for classes, it is best to choose only necessary drivers -- this will significantly speed up the process of building the kernel.

Block devices

The most important options are:

'Loopback device support'

pseudo-device to create a block 'device', the contents of which are stored in a regular file.

'RAM block device support'

support for RAM disks.

'Packet writing on CD / DVD media'

allows you to write CDs / DVDs

'Virtio block driver'

virtualized block device with a low overhead

NVME Support

Support for SSD drives mounted directly on the motherboard (M.2 connector).

SCSI device support

It mainly enables SCSI bus support, but also allows support for many other kinds of block devices using SCSI emulation (including SATA, ATA and USB). To use these devices, enable 'SCSI disk support', 'SCSI CD-ROM support', 'SCSI generic support'. Additionally, in the 'SCSI low-level drivers' menu you can enable support for a hardware SCSI controller (if you have one).

Serial ATA and Parallel ATA drivers (libata)

Support for ATA and SATA disk devices. To use a disk, you should also enable support for your ATA or SATA controller here. 'AHCI SATA Support' and 'Generic ATA support' options support most devices, but may have less functionality than a specialized driver. This driver is made on the basis of the SCSI layer -- to use the disk or optical drive, you should also enable support for the right type of device in the SCSI menu.

Multiple devices driver support (RAID and LVM)

'RAID support'

enables software RAID support, enabling using multiple disks as one, which can improve performance and safety of disk operations.

'Device mapper support'

support for low-level volume manager, which is used by programs that allow to define colume groups and logical disks (volumes) on them to simplify disk management in large systems.

The 'RAID support' option is also useful in home systems, assuming they have at least two hard drives -- in this situation RAID-0 mode allows doubling the performance of disk operations.

Network device support

Allows you to compile the drivers for wired network adapters ('Ethernet (10 or 100Mbit)' , 'Ethernet (1000 Mbit)'), wireless network cards ('Wireless LAN') and PPP support ('PPP (point-to-point protocol) support'), as well as many other types of network cards and protocols. These four options, however, will be used most often.

Input device support

Support (general) for input devices. If we want to use mouse, keyboard, joystick or similar devices, turn on this option (luckily it is difficult to disable it) and the appropriate module. Support for USB input devices is enabled in the 'HID Devices' menu (see below).

Virtio drivers

Enables support for virtio devices -- virtual devices with low overhead, provided by QEMU. It is worth to enable this option when compiling the kernel for a virtual machine. Some of the virtio drivers can be found in other places (e.g., virtio network device is among other network cards).

Other sections with drivers

In the remaining menus, you can configure various devices located in the system. Usually, they can be easily compiled as modules, because they are not needed to boot the system.

File systems

Allows you to enable support for various file systems. The most important thing is the system that is used on the system boot partition (usually ext4 or btrfs) -- it must be compiled into the kernel or stored on the initial ramdisk. Other file systems can be compiled as modules. It is also important to select support for 'Tmpfs virtual memory file system support', '/proc' and sysfs (all from 'Pseudo filesystems'). 'Filesystem in Userspace support' is FUSE, which allows using file system drivers running in the user space. Other file systems can be compiled depending on your needs.

Compiling the kernel -- Kbuild

Kbuild is the Linux build system. It is built on top of specially prepared Makefiles.

As always with Makefiles, it is used as follows:

make <options> <target> <optional variables for Kbuild>

A useful option to make is -j <N> -- it will invoke parallel compilation with up to N processes at the same time (please do not overuse it on students).

make clean

Removes compiled files.

make bzImage

Compiles the kernel and places it in the arch/<architecture>/boot directory under as bzImage. This kernel is compressed (it unpacks itself at system startup).

make modules

Compiles parts of the kernel that have been configured as modules. They should then be installed with the command make modules_install.

make all

Works like make bzImage with make modules.

make modules_install

Installs modules into the /lib/modules/<version>/ directory and calls depmod to create dependency information. If the variable INSTALL_MOD_PATH is given, it installs in $INSTALL_MOD_PATH/lib/modules/<version>/.

make help

Shows available make commands.

make mrproper

Cleans the source directory exactly (including the configuration!), Removes dependencies, modules, etc.

make prepare

Prepares the target build directory. Especially useful when building in a directory other than the source.

make install

Installs the kernel and adds it to the bootloader configuration. Does not always work as expected.

make htmldocs

Compiles documentation in the DocBook format to HTML format.

Variables

Some kernel compilation parameters can be specified by the make program variables (adding VARIABLE=value at the end of the make command). The most important:

V=1

Verbose, Kbuild will write exactly what it does.

ARCH=<arch>

Forces <arch> architecture, i386 for example.

EXTRA_CFLAGS=<flags>

When compiling, <flags> will be added to gcc calls (it may be useful to provide -g to have symbols).

INSTALL_MOD_PATH=<path>

Installs modules in the specified location.

O=<path>

Places the resulting files in a separate directory; after the first make call with this option, you can use make in the specified directory without additional options (an appropriate Makefile is placed there).

Hands-on

Configure and compile the kernel and modules.

  • Change the 'General setup -> Local version' option to your identify you somehow.

Remember to include all the options necessary to start the system, including:

  • selecting 64-bit kernel

  • PCI bus support (Bus options -> PCI support)

  • ELF format support (Exectable file formats -> Kernel support for ELF binaries)

  • virtio support (Device drivers -> virtio -> PCI driver for virtio devices)

  • virtio balloon support (Device drivers -> virtio -> Virtio balloon driver)

  • UNIX sockets (Network support -> Networking options -> Unix domain sockets)

  • IPv4 protocol (Network support -> Networking options -> TCP/IP networking)

  • virtio block driver (Device drivers -> Block devices -> Virtio block driver)

  • virtio net driver (Device drivers -> Network device support -> Virtio network driver)

  • virtconsole driver (Device drivers -> Character devices -> Virtio console)

  • ext4 file system (File systems -> The Extended 4 (ext4) filesystem)

  • proc file system (File systems -> Pseudo filesystems -> /proc file system support)

  • FUSE file system (File systems -> FUSE (Filesystem in Userspace) support)

Bootloaders

A bootloader is a program that loads the operating system. On older systems, when the computer started, the BIOS would load the tiny 512B master boot record (MBR) from disk and give it control. Nowadays, we have EFI which enable support for complex bootloaders from the very beginning.

There are many bootloaders available, but GRUB (2) is the most popular.

GRUB

GRUB can be used through two interfaces: a more powerful command line interface or a simple menu. Usually the latter is enough. Once started, GRUB searches for a configuration file. If it's found, GRUB displays entries from this file (corresponding to Linux images or other systems) in the form of menu items. The menu items can be edited (but the changes only affect the current boot, they are not remembered) -- switching to the editing mode is done by pressing 'e' (and in this way it can be possible to fix errors from the configuration file). From the menu, you can also go to the command line by pressing 'c' (return via ESC). If your GRUB is protected by a password, it is only possible to enter the editing mode or the command line after pressing 'p' and entering the password.

The GRUB configuration file is /etc/grub.conf. The file consists of entries for kernel images or other operating systems. The entries are numbered starting from 0.

The most important commands included in an entry are:

  • TITLE image_name -- the name that will be visible in the menu. Starts the entry; the entry ends at the end of the file or with the next title command (starting next entry).

  • ROOT root_device -- where we will look for the image.

  • ROOTNOVERIFY root_device -- as above, but the device will not be mounted, and the command is usually used to specify the location of another boot loader, which allows the so-called chain-loading and loading eg Windows.

  • KERNEL image_file [kernel_options]

The important commands that are also part of grub.conf are:

  • DEFAULT=entry_index -- which entry will be selected by default (if not specified, entry 0 will be selected).

  • TIMEOUT=time_in_seconds -- GRUB will wait time_in_seconds for system selection. If this does not happen, it will load the default.

Line comments are started with the # sign.

The grub.conf file can look eg. like this:

# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/hda2

default=1
timeout=3

title dos
    rootnoverify (hd0,0) # sets the GRUB root device, without mounting
    makeactive
    chainloader +1       # loads another boot loader
title linux
        root (hd0,4)
        kernel /vmlinuz ro root=/dev/hda5 hdc=ide-scsi
title new
    root (hd0,4)
    kernel /vmlinuz-2.2.17 ro root=/dev/hda5 hdc=ide-scsi

GRUB 2

GRUB 2 is the successor to GRUB, with greater modularity and functionality. The menu interface is quite similar to GRUB, but the configuration has changed significantly.

The GRUB 2 configuration file is /boot/grub/grub.cfg, but it should not be modified directly -- it is generated by grub-mkconfig program based on scripts in the /etc/grub.d directory and configuration in the file /etc/default/grub.

Thanks to the configuration scripts, you do not have to manually create the configuration entries for each kernel - just put the kernel as /boot/vmlinuz-<version>, and possibly initramfs as /boot/initrd.img-<version>. Options passed to the installed kernel can be set in the /etc/default/grub file.

After changing the configuration, issue the following command:

grub-mkconfig -o /boot/grub/grub.cfg

or just:

update-grub2

To install GRUB 2 for the first time, you must issue the following command:

grub-install /dev/<disk>

Exercise

Hand-on

Copy the compiled kernel image to /boot and update grub. On the QEMU image, you may just run:

sudo make modules_install
sudo make install

Restart the virtual machine, hope for the best and boot with the new kernel! (You may need to select it in the GUI -- don't run QEMU with it disabled.)

If it doesn't boot, try selecting another kernel version in GRUB... or recreate your qcow2 disk image and learn that replacing your last working kernel image in /boot is not the best idea.

Small Taks #4

Modify the kernel source, so that it prints "Hello, ZSO!" to the system log while booting. Compile and install the modified kernel, then boot the system with it. As a proof, submit:

  • a short description what have you changed,

  • and a screenshot of either:

    • the boot console

    • relevant output of dmesg

Hint

Look at the lecture slides for the boot sequence or visit arch/x86/boot in the sources. Another tricky solution may involve using grep...

Extra Homework

Set up a workflow based on the git repository -- you will need it to easily generate patches of your changes as required for the next small task and large assignment #2. (Hint: use --depth 1 option for git clone). Play around to see if you prefer compiling Linux inside the image or outside of it: check the --kernel flag in QEMU.

Readings

The Linux documentation contains a lot of useful pages, but a lot of them are outdated by decades. Consider reading: