Cross compiling a kernel module
Sometimes we want to compile a kernel module for an embedded device without replacing the kernel. This article provides a summary of the steps which have to be taken to compile the module.
For this article I use Ubuntu 20.10 on the build machine but it should work with other distributions too. Just use the right package manager for your distro. As an embedded device I use a Raspberry Pi 4 but the same concept works for TI AM335x, NXP iMX6/iMX8, etc. On the Raspberry Pi I enabled ssh so that I can transfer files with scp.
First we need to setup the build machine and download the kernel.
On the build machine we need to install a cross compilation toolchain. If we use a recent version of the kernel we can normally just use the latest toolchain provided by the distribution. We need to know for which architecture we want to compile. If we have a Raspberry Pi it might run on a 64 bit or 32 bit kernel. We can figure out the architecture by doing uname -m on the target:
# 32 Bit kernel pi@raspberrypi:~ $ uname -m armv7l # 64 Bit kernel pi@raspberrypi:~ $ uname -m aarch64
For the armv7l kernel we need to install the armhf version of gcc and for aarch64 the aarch64 version. On the build machine we do:
# 32 bit build@machine:~/linux$ sudo apt-get install gcc-arm-linux-gnueabihf # 64 bit build@machine:~/linux$ sudo apt-get install gcc-aarch64-linux-gnu
Now we are prepared to cross compile.
Download the kernel
Downloading the right kernel can be a little bit tricky. It is often not clear from where the kernel comes from. The SoC manufacturer as well as the SoM/CoM provider do some downstream modifications and provide their version in their own repository. Here a list of some kernel repositories:
- TI Sitara: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/
- NXP iMX: https://source.codeaurora.org/external/imx/linux-imx/
- Toradex: http://git.toradex.com/cgit/linux-toradex.git/
- NXP S32G: https://source.codeaurora.org/external/autobsps32/linux/
- Raspberry Pi: https://github.com/raspberrypi/linux
We want to clone the Raspberry Pi repository:
build@machine:~$ git clone https://github.com/raspberrypi/linux.git build@machine:~$ cd linux
We don't know the exact version running on the Raspberry Pi because they don't include a git hash or unique tag name in the kernel version.
In comparison, Toradex would for example include the short git hash in their kernel build name:
root@colibri-imx7-emmc:~$ uname -r 4.14.159-00041-g1f43bce17a57
The number after -g is the short git hash. If we download the kernel from the Toradex repo we can just do a simple checkout of this hash:
# Don't do this if you work with a Raspberry Pi build@machine:~/linux$ git clone git://git.toradex.com/linux-toradex.git build@machine:~/linux$ git checkout 1f43bce17a57
Unfortunately this information is missing for the Raspberry Pi. When we do an uname -r we see:
pi@raspberrypi:~$ uname -r 5.4.83-v7l+
Therefore we just try to get as close as possible. To find the right branch we can list the kernel branches with
git branch -a and we choose the branch which comes closest to the revision we see with uname -r. In our case it should be a 5.4 kernel with Raspberry Pi modifications. So rpi-5.4.y seems reasonable.
build@machine:~/linux$ git branch -a |grep 5.4 remotes/rpi/rpi-5.4.y build@machine:~/linux$ git checkout rpi-5.4.y
If we check the git log we see:
build@machine:~/linux$ git log --oneline 93349cdffc3f (HEAD, rpi/rpi-5.4.y) Hifiberry-DAC+:Avoids loading of headphone controls if not defined in DT-overlay 113831b7f514 Hifiberry DAC+ADC Pro fix for the PLL when changing sample rates 61e5a224f7ae ARM: dts: Declare Pi400 and CM4 have no audio pins cf14b2710bf6 Enhances the Hifiberry DAC+ driver for Hifiberry AMP100 support 044360b0affa Adds the DT-overlays to support Hifiberry AMP100 e3fbd0199b01 drm/vc4: hvs: Fix buffer overflow with the dlist handling 7a146dcb583d kbuild: Silence unavoidable dtc overlay warnings 08ae2dd9e7dc staging: vc04_services: ISP: Add colour denoise control 68878170d8a9 uapi: bcm2835-isp: Add colour denoise configuration bd2041302526 overlays: seeed-can-fd-hat: clarify how to identify HAT version 617a1c1722ae overlays: add spi override to merus-amp overlay 1f085dec8d41 overlays: add wm8960-soundcard overlay abdf918004f2 overlays: Add overlay for Seeed Studio CAN BUS FD HAT v1 (based on mcp2517fd) 35120dfeb619 overlays: give Seeed Studio CAN BUS FD HAT a -v2 postfix e1ad077f0406 overlays: Rebuild "upstream" with latest ovmerge 774f426315b2 Add overlay for Seeed Studio CAN BUS FD HAT (##4034) 40cc64a98a55 media: i2c: ov5647: Selection compliance fixes 6da087d0c70c overlays: Add missing addresses to ads1015/ads1115 5a598ac1bf9d overlays: mpu6050: Add 'addr' parameter 44c11fa2063f net: lan78xx: Ack pending PHY ints when resetting 76c49e60e742 (tag: raspberrypi-kernel_1.20210108-1, tag: raspberrypi-kernel_1.20210104-1) Merge remote-tracking branch 'stable/linux-5.4.y' into rpi-5.4.y 2bff021f53b2 Linux 5.4.83 66a08d1d3bd8 Revert "geneve: pull IP header before ECN decapsulation"
We see that this revision has some commits on top of 5.4.83 so we are probably using the right version or at least a version which is close enough to what is used by Raspberry Pi OS. To compile a kernel module which is compatible with the running kernel we luckily don't need to use the exact same version. However, it needs to be close, else we won't be able to load the module.
Get the kernel configuration
Now we compile the kernel with a configuration which is close to what was used to build the kernel of the distribution. There is a kernel module configs available under Linux. This module exports the configuration of the running kernel to /proc/config.gz. Sometimes this module is compiled into the kernel as built-in kernel module, sometimes it's a loadable module and sometimes it's not available at all. If it's not available it will be hard to restore the configuration. It might be that there is a defconfig available in the kernel sources which matches or the config is stored under /boot/config
pi@raspberrypi:~$ sudo modprobe configs
Now we can fetch the configuration and extract it:
build@machine:~/linux$ ssh pi@<IP>:/proc/config.gz . build@machine:~/linux$ zcat config.gz > .config
Now that we have everything available on the build machine we are ready to compile the kernel and the kernel module.
Build the kernel
After we downloaded the kernel and fetched the configuration we can rebuild the kernel. We first need to set the ARCH and CROSS_COMPILE variable.
build@machine:~/linux$ # export LOCALVERSION="" # See notes # 32 bit build@machine:~/linux$ export ARCH=arm build@machine:~/linux$ export CROSS_COMPILE=arm-linux-gnueabihf- # 64 bit build@machine:~/linux$ export ARCH=arm64 build@machine:~/linux$ export CROSS_COMPILE=aarch64-linux-gnu-
And finally we are ready to compile:
build@machine:~/linux$ make -j
We just need the kernel so that we have all build headers configured and available on the build machine. In theory we could also get them from Raspberry Pi OS. I won't go into details on how we can do that though.
Compile the kernel module
To compile the kernel module we first need to have a module available. As an example we use the following:
build@machine:~$ git clone email@example.com:embear-engineering/sample-kernel-modules.git build@machine:~$ cd sample-kernel-modules/skeleton
Now we need to set the environment variables again if not already done before:
build@machine:~/sample-kernel-modules/skeleton$ export KDIR=~/linux # 32 bit build@machine:~/sample-kernel-modules/skeleton$ export ARCH=arm build@machine:~/sample-kernel-modules/skeleton$ export CROSS_COMPILE=arm-linux-gnueabihf- # 64 bit build@machine:~/sample-kernel-modules/skeleton$ export ARCH=arm64 build@machine:~/sample-kernel-modules/skeleton$ export CROSS_COMPILE=aarch64-linux-gnu-
KDIR is a variable passed to make when compiling the module and should point to the kernel sources. It might be that other Makefiles use a different variable name:
KDIR ?= "/lib/modules/$(shell uname -r)/build" all: make -C $(KDIR) M=$(PWD) modules
We can now compile the module with:
And transfer it to the target:
build@machine:~/sample-kernel-modules/skeleton$ scp skeleton.ko pi@<IP>:~
Finally we can insmod the module on the Raspberry Pi:
pi@raspberrypi:~$ sudo insmod skeleton.ko pi@raspberrypi:~$ lsmod Module Size Used by skeleton 16384 0
We have successfully cross compiled our kernel module and are able to load it on the Raspberry Pi.
Some additional notes to this article.
There is the variable LOCALVERSION which might be set when building the kernel. This depends on how the kernel was compiled. If you do uname -r and you see a + at the end:
pi@raspberrypi:~$ uname -r 5.4.83-v7l+
This means LOCALVERSION is not set to anything. If it would be:
pi@raspberrypi:~$ uname -r 5.4.83-v7l
Then you have to set LOCALVERSION to "":
build@machine:~$ export LOCALVERSION=""
Sometimes a string is appended. In this case you can set the LOCALVERSION to this string:
pi@raspberrypi:~$ uname -r 5.4.83-v7l+my-additonal-string
build@machine:~$ export LOCALVERSION="+my-additonal-string"
It is possible to use the cross compiled kernel as a drop in replacement for the running kernel by replacing the kernel on the file system (e.g. /boot/kernel7l.bin) and reboot. Because it used the same configuration file and the same version string the available modules should still load. This only works as long as the configuration doesn't differ too much. It can be a nice "hack" if we need to debug a built-in kernel module and don't want to recompile all modules or because the modules are under control of something like OSTree. However, if we e.g. add printks for debugging we need to make sure to set LOCALVERSION="+" (see previous section) because as soon as we have local modifications the build would add "-dirty" and then the modules won't load anymore.