Cross compiling a kernel module

Sometimes we want to compile a kernel module for an embedded device without replacing the kernel. This article provides a summary of the steps which have to be taken to compile the module.

For this article I use Ubuntu 20.10 on the build machine but it should work with other distributions too. Just use the right package manager for your distro. As an embedded device I use a Raspberry Pi 4 but the same concept works for TI AM335x, NXP iMX6/iMX8, etc. On the Raspberry Pi I enabled ssh so that I can transfer files with scp.

Setup

First we need to setup the build machine and download the kernel.

Build machine

On the build machine we need to install a cross compilation toolchain. If we use a recent version of the kernel we can normally just use the latest toolchain provided by the distribution. We need to know for which architecture we want to compile. If we have a Raspberry Pi it might run on a 64 bit or 32 bit kernel. We can figure out the architecture by doing uname -m on the target:

# 32 Bit kernel
pi@raspberrypi:~ $ uname -m
armv7l

# 64 Bit kernel
pi@raspberrypi:~ $ uname -m
aarch64

For the armv7l kernel we need to install the armhf version of gcc and for aarch64 the aarch64 version. On the build machine we do:

# 32 bit
build@machine:~/linux$ sudo apt-get install gcc-arm-linux-gnueabihf

# 64 bit
build@machine:~/linux$ sudo apt-get install gcc-aarch64-linux-gnu

Now we are prepared to cross compile.

Download the kernel

Downloading the right kernel can be a little bit tricky. It is often not clear from where the kernel comes from. The SoC manufacturer as well as the SoM/CoM provider do some downstream modifications and provide their version in their own repository. Here a list of some kernel repositories:

We want to clone the Raspberry Pi repository:

build@machine:~$ git clone https://github.com/raspberrypi/linux.git
build@machine:~$ cd linux

We don't know the exact version running on the Raspberry Pi because they don't include a git hash or unique tag name in the kernel version.

In comparison, Toradex would for example include the short git hash in their kernel build name:

root@colibri-imx7-emmc:~$ uname -r
4.14.159-00041-g1f43bce17a57

The number after -g is the short git hash. If we download the kernel from the Toradex repo we can just do a simple checkout of this hash:

# Don't do this if you work with a Raspberry Pi
build@machine:~/linux$ git clone git://git.toradex.com/linux-toradex.git
build@machine:~/linux$ git checkout 1f43bce17a57

Unfortunately this information is missing for the Raspberry Pi. When we do an uname -r we see:

pi@raspberrypi:~$ uname -r
5.4.83-v7l+

Therefore we just try to get as close as possible. To find the right branch we can list the kernel branches with git branch -a and we choose the branch which comes closest to the revision we see with uname -r. In our case it should be a 5.4 kernel with Raspberry Pi modifications. So rpi-5.4.y seems reasonable.

build@machine:~/linux$ git branch -a |grep 5.4
  remotes/rpi/rpi-5.4.y
build@machine:~/linux$ git checkout rpi-5.4.y

If we check the git log we see:

build@machine:~/linux$ git log --oneline
93349cdffc3f (HEAD, rpi/rpi-5.4.y) Hifiberry-DAC+:Avoids loading of headphone controls if not defined in DT-overlay
113831b7f514 Hifiberry DAC+ADC Pro fix for the PLL when changing sample rates
61e5a224f7ae ARM: dts: Declare Pi400 and CM4 have no audio pins
cf14b2710bf6 Enhances the Hifiberry DAC+ driver for Hifiberry AMP100 support
044360b0affa Adds the DT-overlays to support Hifiberry AMP100
e3fbd0199b01 drm/vc4: hvs: Fix buffer overflow with the dlist handling
7a146dcb583d kbuild: Silence unavoidable dtc overlay warnings
08ae2dd9e7dc staging: vc04_services: ISP: Add colour denoise control
68878170d8a9 uapi: bcm2835-isp: Add colour denoise configuration
bd2041302526 overlays: seeed-can-fd-hat: clarify how to identify HAT version
617a1c1722ae overlays: add spi override to merus-amp overlay
1f085dec8d41 overlays: add wm8960-soundcard overlay
abdf918004f2 overlays: Add overlay for Seeed Studio CAN BUS FD HAT v1 (based on mcp2517fd)
35120dfeb619 overlays: give Seeed Studio CAN BUS FD HAT a -v2 postfix
e1ad077f0406 overlays: Rebuild "upstream" with latest ovmerge
774f426315b2 Add overlay for Seeed Studio CAN BUS FD HAT (##4034)
40cc64a98a55 media: i2c: ov5647: Selection compliance fixes
6da087d0c70c overlays: Add missing addresses to ads1015/ads1115
5a598ac1bf9d overlays: mpu6050: Add 'addr' parameter
44c11fa2063f net: lan78xx: Ack pending PHY ints when resetting
76c49e60e742 (tag: raspberrypi-kernel_1.20210108-1, tag: raspberrypi-kernel_1.20210104-1) Merge remote-tracking branch 'stable/linux-5.4.y' into rpi-5.4.y
2bff021f53b2 Linux 5.4.83
66a08d1d3bd8 Revert "geneve: pull IP header before ECN decapsulation"

We see that this revision has some commits on top of 5.4.83 so we are probably using the right version or at least a version which is close enough to what is used by Raspberry Pi OS. To compile a kernel module which is compatible with the running kernel we luckily don't need to use the exact same version. However, it needs to be close, else we won't be able to load the module.

Get the kernel configuration

Now we compile the kernel with a configuration which is close to what was used to build the kernel of the distribution. There is a kernel module configs available under Linux. This module exports the configuration of the running kernel to /proc/config.gz. Sometimes this module is compiled into the kernel as built-in kernel module, sometimes it's a loadable module and sometimes it's not available at all. If it's not available it will be hard to restore the configuration. It might be that there is a defconfig available in the kernel sources which matches or the config is stored under /boot/config. Luckily, for the Raspberry Pi the module is available. We just need to do:

pi@raspberrypi:~$ sudo modprobe configs

Now we can fetch the configuration and extract it:

build@machine:~/linux$ ssh pi@<IP>:/proc/config.gz .
build@machine:~/linux$ zcat config.gz > .config

Compiling

Now that we have everything available on the build machine we are ready to compile the kernel and the kernel module.

Build the kernel

After we downloaded the kernel and fetched the configuration we can rebuild the kernel. We first need to set the ARCH and CROSS_COMPILE variable.

build@machine:~/linux$ # export LOCALVERSION="" # See notes
# 32 bit
build@machine:~/linux$ export ARCH=arm
build@machine:~/linux$ export CROSS_COMPILE=arm-linux-gnueabihf-

# 64 bit
build@machine:~/linux$ export ARCH=arm64
build@machine:~/linux$ export CROSS_COMPILE=aarch64-linux-gnu-

And finally we are ready to compile:

build@machine:~/linux$ make -j

We just need the kernel so that we have all build headers configured and available on the build machine. In theory we could also get them from Raspberry Pi OS. I won't go into details on how we can do that though.

Compile the kernel module

To compile the kernel module we first need to have a module available. As an example we use the following:

build@machine:~$ git clone git@github.com:embear-engineering/sample-kernel-modules.git
build@machine:~$ cd sample-kernel-modules/skeleton

Now we need to set the environment variables again if not already done before:

build@machine:~/sample-kernel-modules/skeleton$ export KDIR=~/linux

# 32 bit
build@machine:~/sample-kernel-modules/skeleton$ export ARCH=arm
build@machine:~/sample-kernel-modules/skeleton$ export CROSS_COMPILE=arm-linux-gnueabihf-

# 64 bit
build@machine:~/sample-kernel-modules/skeleton$ export ARCH=arm64
build@machine:~/sample-kernel-modules/skeleton$ export CROSS_COMPILE=aarch64-linux-gnu-

KDIR is a variable passed to make when compiling the module and should point to the kernel sources. It might be that other Makefiles use a different variable name:

KDIR ?= "/lib/modules/$(shell uname -r)/build"

all:
    make -C $(KDIR) M=$(PWD) modules

We can now compile the module with:

build@machine:~/sample-kernel-modules/skeleton$ make

And transfer it to the target:

build@machine:~/sample-kernel-modules/skeleton$ scp skeleton.ko pi@<IP>:~

Finally we can insmod the module on the Raspberry Pi:

pi@raspberrypi:~$ sudo insmod skeleton.ko
pi@raspberrypi:~$ lsmod
Module                  Size  Used by
skeleton               16384  0

We have successfully cross compiled our kernel module and are able to load it on the Raspberry Pi.

Notes

Some additional notes to this article.

LOCALVERSION

There is the variable LOCALVERSION which might be set when building the kernel. This depends on how the kernel was compiled. If you do uname -r and you see a + at the end:

pi@raspberrypi:~$ uname -r
5.4.83-v7l+

This means LOCALVERSION is not set to anything. If it would be:

pi@raspberrypi:~$ uname -r
5.4.83-v7l

Then you have to set LOCALVERSION to "":

build@machine:~$ export LOCALVERSION=""

Sometimes a string is appended. In this case you can set the LOCALVERSION to this string:

pi@raspberrypi:~$ uname -r
5.4.83-v7l+my-additonal-string
build@machine:~$ export LOCALVERSION="+my-additonal-string"

Kernel Hack

It is possible to use the cross compiled kernel as a drop in replacement for the running kernel by replacing the kernel on the file system (e.g. /boot/kernel7l.bin) and reboot. Because it used the same configuration file and the same version string the available modules should still load. This only works as long as the configuration doesn't differ too much. It can be a nice "hack" if we need to debug a built-in kernel module and don't want to recompile all modules or because the modules are under control of something like OSTree. However, if we e.g. add printks for debugging we need to make sure to set LOCALVERSION="+" (see previous section) because as soon as we have local modifications the build would add "-dirty" and then the modules won't load anymore.