Assignment 1: BPF OOM Killer

Announcement date: 24.02.2026

Due date: 24.03.2026 (final due date 08.04.2026)

Additional materials

Introduction

The Out of Memory Killer (OOM Killer) is a mechanism that prevents the operating system from becoming completely unusable due to a lack of memory resources. Providing effective protection against running out of memory is a challenging problem — it is difficult to decide which processes should be killed first and when the killing should begin. Moreover, a solution that works well in one environment may not be optimal in another. Therefore, in addition to the kernel OOM Killer (see mm/oom_kill.c), there are other mechanisms such as system-oomd.

Recently, there has been ongoing work to use BPF to extend the functionality of the OOM Killer (see https://lwn.net/Articles/1019230/). However, in kernel version 6.18.5, which we use, BPF support for this feature has not yet been included (see https://lore.kernel.org/lkml/20260127024421.494929-1-roman.gushchin@linux.dev/).

This does not mean that existing BPF functionalities cannot help administrators protect the operating system from running out of memory. BPF is a powerful tool, and the goal of this assignment is to implement an OOM Killer using BPF.

Assignment

The assignment is to implement a BPF program that acts like an OOM Killer. If a significant portion of system memory is used, selected programs should be killed according to the OOM Killer policies.

The OOM Killer policy is as follows:

  • Do not kill any program unless at least 1 GB of RAM is being used.

  • Kill only programs with "oomp" in their process name.

  • Programs that use 100 or more file descriptors should be killed.

  • Programs that write to files 100 times or more should be killed.

  • Programs that read 10 MB or more should be killed.

  • Programs that spawn 100 or more threads should be killed.

  • Programs that call the rand() function 100 or more times should be killed.

  • Programs that send 1000 or more TCP packets should be killed.

The program should be implemented in C using the libbpf library. In addition to the *.bpf.c program, a userspace (*.c) program should be implemented to load and attach the BPF program. However, the userspace-side implementation should be minimal and should not implement the OOM Killer logic.

Technical details

  1. The OOM Killer requires information about processes to decide which ones should be killed. BPF_MAP_TYPE_HASH should be used to maintain this data.

  2. Information about system-wide memory usage should be obtained from si_meminfo. It is a good idea to utilize kprobe/si_meminfo and/or kretprobe/si_meminfo. To make sure that the si_meminfo is frequently accessed, you can access it periodically (e.g. once a second) from the userspace program that loads bpf.

  3. Deciding how to collect data to implement the OOM Killer policy is the major part of this assignment. Sometimes, the solution can be implemented in more than one way. The scope of monitoring depends on your implementation. For example, implementing the policy to kill programs overusing writing requires monitoring not only the write syscall but also pwrite64. The goal is to select reasonable monitoring methods that pass the tests as explained below.

  4. Up-to-date vmlinux.h is necessary to compile the solution. It can be generated with bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h when bpftool is installed in the system.

  5. You can install additional dependencies using apt. However, make sure that the installation of such packages is included in the Makefile.

Testing

The attached tests.zip contains tests that can be used to verify the OOM Killer's behavior. In the proper environment (more than 1 GB RAM used), the following programs should be killed automatically:

  • oomp_test_fds

  • oomp_test_threads

  • oomp_test_reads

  • oomp_test_readv

  • oomp_test_writes

  • oomp_test_pwrite64

  • oomp_test_packets

  • oomp_test_rand

In the same environment, the following programs should not be killed:

  • oomp_test_rw_below

  • oomp_test_packets_udp

Solution

The solution should be prepared as a zip file named in the format xz123456.zip, where xz are your initials and 123456 is your student number. The archive can contain any number of files, but running make in the directory with the contents of the archive should produce an ./oomkiller binary. Executing this binary as a superuser should run the OOM Killer.

The solution should compile without errors in the virtual machine environment provided for the laboratories. The kernel has already been compiled with numerous flags that allow deep BPF usage (such as CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT, and CONFIG_DEBUG_INFO_BTF), and no additional modifications to the kernel configuration are allowed.