Booting RAM disk images over Ethernet

Using RAM disk images is useful in different stages of a project/product. When doing bring-ups we might want to use a RAM disks to initially boot Linux and then use Linux to partition the storage (eMMC, NAD-Flash). The advantage of using Linux instead of U-Boot is that we have more drivers available and often better read/write support for the media where we want to install our final image.

But what is a RAM disk? A RAM disk is a file which contains a partition. To boot Linux, the file contains an entire root file system. On x86/amd64 RAM disks (or in that case initramfs) are also used to do some initial booting, show a splash screen and then load drivers before switching to the actual root file system but for embedded devices that's not always necessary. In this article we use a initrd and not initramfs. Check this article to find out more about initramfs and to understand the differences.

Our setup for this article looks as follows: Board Setup The host system runs Linux with dnsmasq and the target runs U-Boot with a DHCP and TFTP client. This options are often already enabled in U-Boot or can be selected in the configuration. We need to make sure the following configurations are enabled in our U-Boot configuration:

CONFIG_CMD_TFTPBOOT=y
CONFIG_CMD_DHCP=y
CONFIG_CMD_PXE=y

The IP address of the host system where the embedded system is connected to is 192.168.1.254. If a firewall is used we need to make sure that the target device can access port 69 (UDP) for TFTP and port 67 (UDP) for DHCP.

In this example a Colibri iMX6 from Toradex is used but it works with most embedded systems which support U-Boot and have an Ethernet interface.

dnsmasq

To load images over Ethernet we use TFTP. With TFTP we transfer the kernel, device tree and the RAM disk image from the host system into the memory of our device. In this article we use dnsmasq on the host system to act as a TFTP server. The nice thing about dnsmasq is that it can serve as DNS, DHCP and TFTP server. We need to make sure that dnsmasq is installed on the host:

host@machine:~$ sudo apt-get install dnsmasq

Now we just create a relatively simple configuration file /etc/dnsmasq.conf:

listen-address=192.168.1.254
dhcp-range=192.168.1.50,192.168.1.200,24h
enable-tftp
tftp-root=/var/tftpd

This will instruct dnsmasq to listen on the Ethernet interface with the IP address 192.168.1.254 for DHCP requests and will enable TFTP with /var/tftpd as root directory. It will also start a DNS server which will resolve the same domain names as the host can resovle. However, for this article we don't need DNS.

We now start dnsmaq with the following command:

host@machine:~$ systemct start dnsmasq

TFTP from U-Boot

Now that we have configured dnsmasq to act as a TFTP server we test if we can transfer files and boot Linux. We do this by using the tftpboot command. In this example we load a Linux kernel, a device tree and a RAM disk image manually and boot it.

We have to compile a kernel, a device tree and a RAM disk image. To do this we can use Yocto and compile the Reference-Minimal-Image for the Colibri iMX6 following this article. To get a loadable RAM disk image we need to set the following in the local.conf file:

IMAGE_FSTYPES_append = " cpio.gz.u-boot"

This will generate a cpio archive which is compressed and has a U-Boot header. The cpio archive will then be unpacked by Linux and mounted as RAM disk root partition. We also need to copy the zImage and the imx6dl-colibri-eval-v3.dtb device tree file.

In this article we can use the following files if we don't want to compile the images: Kernel: zImage
Devicetree: imx6dl-colibri-eval-v3.dtb
Ramdisk: Reference-Minimal-Image-colibri-imx6.cpio.gz.u-boot

We put these files into the TFTP folder /var/tftpd/colibri-imx6.

Now we load the binaries and boot Linux:

Colibri iMX6 $ setenv autoload no
Colibri iMX6 $ dhcp
Colibri iMX6 $ tftpboot $kernel_addr_r colibri-imx6/zImage
Colibri iMX6 $ tftpboot $fdt_addr_r colibri-imx6/imx6dl-colibri-eval-v3.dtb
Colibri iMX6 $ tftpboot $ramdisk_addr_r colibri-imx6/Reference-Minimal-Image-colibri-imx6.cpio.gz.u-boot
Colibri iMX6 $ setenv bootargs rw earlyprintk rootwait root=/dev/ram0 rdinit=/init console=ttymxc0,115200n8 video=mxcfb0:dev=lcd,640x480M@60,if=RGB666 fbmem=8M
Colibri iMX6 $ bootz $kernel_addr_r $ramdisk_addr_r $fdt_addr_r

This will boot our RAM disk image from memory.

Distroboot

Instead of typing the commands manually we can use a boot script. This script will be loaded and then sourced, which means the commands are executed. Distroboot searches for boot.scr in the TFTP root directory. If it finds the file it will load it and then execute it.

The distroboot file look as follows, we put that into a file ~/boot.script

tftpboot $kernel_addr_r colibri-imx6/zImage
tftpboot $fdt_addr_r colibri-imx6/imx6dl-colibri-eval-v3.dtb
tftpboot $ramdisk_addr_r colibri-imx6/Reference-Minimal-Image-colibri-imx6.cpio.gz.u-boot
setenv bootargs rw earlyprintk rootwait root=/dev/ram0 console=ttymxc0,115200n8 rdinit=/init video=mxcfb0:dev=lcd,640x480M@60,if=RGB666 fbmem=8M
bootz $kernel_addr_r $ramdisk_addr_r $fdt_addr_r

We use mkimage to add the headers for U-Boot. For that we install u-boot-tools:

host@machine:~$ sudo apt-get install u-boot-tools

Now we run mkimage and create the U-Boot loadable boot script:

host@machine:~$ mkimage -A arm -C none -T script -d boot.script boot.scr
Image Name:
Created:      Sat May 22 10:30:36 2021
Image Type:   ARM Linux Script (uncompressed)
Data Size:    170 Bytes = 0.17 KiB = 0.00 MiB
Load Address: 00000000
Entry Point:  00000000
Contents:
   Image 0: 162 Bytes = 0.16 KiB = 0.00 MiB

We copy the file boot.scr to the TFTP root directory:

host@machine:~$ cp boot.scr /var/tftpd

Now we run the DHCP boot command which is part of distroboot:

Colibri iMX6 $ run bootcmd_dhcp

Normally when U-Boot is compiled with distroboot support, the following variables should already be set:

var descriptiton Colibri iMX6
fdt_addr_r address to where the device tree is loaded 0x12100000
ramdisk_addr_r address to where the RAM disk image is loaded 0x12200000
kernel_addr_r address to where the kernel is loaded 0x11000000
scriptaddr address to where the boot script is loaded 0x17000000

More information about distro boot can be found in the U-Boot distro README.

PXE

Another possibility to boot a RAM disk over TFTP is PXE. This is a protocol that originates from the x86 world. However, U-Boot also supports it to a certain degree. The advantage of PXE is that it is also supported by other bootloaders. Further, it has a search mechanism for the configuration file. This means we can have different boot configurations for different products on the same TFTP server. Of course we could achieve that as well with distroboot but we would have to script a mechanism.

We again use DHCP to get the module IP address and the module TFTP server IP. The TFTP and DHCP server have to have the same address. This is a limitation of the U-Boot PXE implementation. A list of PXE features supported by U-Boot can be found in the README.

By default U-Boot searches in the directory pxelinux.cfg for several pxe config files:

what description example file name
pxeuuid A custom uuid which can be set in the environment 0123456
MAC The MAC address of the Ethernet interface 01-00-14-2d-bc-61-4e
IP The IP address in hexadecimal form. Also partial IP addresses are supprted C0A801B0
C0A801B
C0A801
C0A80
C0A8
C0A
C0
C
default Default file if everything else fails. It also checks for some machine specific files in the form
default-CONFIG_SYS_ARCH-CONFIG_SYS_SOC-CONFIG_SYS_BOARD		 default-arm-imx6-colibri-imx6
default-arm-imx6
default-arm
default

We use the default file in this example. For that we create /var/tftpd/pxelinux.cfg/default with the following content:

# The menu title that will be shown by pxe boot
menu title Linux selections

# The default configuration that is used if nothing else will be selected
default colibri-imx6
# The timeout to wait for an alternative configuration in seconds
timeout 4

# A menu item, several menu items are possible
label colibri-imx6
   # The name shown for selection   
   menu label Colibri iMX6
   # The kernel file to load
   kernel colibri-imx6/zImage
   # The RAM disk image to load
   initrd colibri-imx6/Reference-Minimal-Image-colibri-imx6.cpio.gz.u-boot
   # The device tree file to load
   fdt colibri-imx6/imx6dl-colibri-eval-v3.dtb
   # bootargs for Linux which will be appended to bootargs already set
   append rw earlyprintk rootwait root=/dev/ram0 rdinit=/init console=ttymxc0,115200n8 video=mxcfb0:dev=lcd,640x480M@60,if=RGB666 fbmem=8M

# The same as above but with a different name, we can also remove that
label colibri-imx6
   # The name shown for selection   
   menu label Colibri iMX6 Test
   # The kernel file to load
   kernel colibri-imx6/zImage
   # The RAM disk image to load
   initrd colibri-imx6/Reference-Minimal-Image-colibri-imx6.cpio.gz.u-boot
   # The device tree file to load
   fdt colibri-imx6/imx6dl-colibri-eval-v3.dtb
   # bootargs for Linux which will be appended to bootargs already set
   append rw earlyprintk rootwait root=/dev/ram0 rdinit=/init console=ttymxc0,115200n8 video=mxcfb0:dev=lcd,640x480M@60,if=RGB666 fbmem=8M

The following environment variables need to be set in U-Boot to make PXE work (if not already be done by the board/SoC supplier):

var descriptiton Colibri iMX6
fdt_addr_r address to where the device tree is loaded 0x12100000
ramdisk_addr_r address to where the RAM disk image is loaded 0x12200000
kernel_addr_r address to where the kernel is loaded 0x11000000
pxefile_addr_r address to where the pxe script is loaded to 0x17100000

If we created the configuration file and made sure that all environment variables are set. We can use the following commands to boot Linux:

Colibri iMX6 $ setenv autoload no
Colibri iMX6 $ dhcp
Colibri iMX6 $ pxe get
Colibri iMX6 $ pxe boot

By setting autoload to "no" we disable the automatic load of files with the DHCP command. We can have several different menu items, so that we could e.g. select between a recovery image, a test image and a development image. Before a timeout occurs U-Boot asks which entry we want to boot. Here an example:

Colibri iMX6 $ pxe boot
Linux selections
1:      Colibri iMX6
2:      Colibri iMX6 Test
Enter choice:

If we don't choose anything U-Boot will select the default configuration after timeout seconds.

Using a FIT image

Instead of loading a kernel, device tree and ramdisk separately we can also boot a FIT image. Here an example configuration for a Toradex Easy Installer Image. We just need to copy the tezi.itb to /var/tftpd/colibri-imx6 and Toradex Easy Installer will load when we do the PXE boot above.

# The menu title that will be shown by pxe boot
menu title Linux selections

# The default configuration that is used if nothing else will be selected
default colibri-imx6-tezi
# The timeout to wait for an alternative configuration in seconds
timeout 4

# A Menu item, several menu items are possible
label colibri-imx6-tezi
   # The name shown for selection   
   menu label Colibri iMX6 Toradex Easy Installer
   # The fit image to load
   kernel colibri-imx6/tezi.itb
   # bootargs for Linux which will be appended to bootargs already set
   append console=ttymxc0,115200 quiet video=DPI-1:640x480-16@60D video=HDMI-A-1:640x480-16@60D rootfstype=squashfs root=/dev/ram autoinstall

FIT images are great for images in production tests or for recovery images. However, for bring-up and development they are less useful because to change the kernel or device tree we always need to repack the image. Using separate files is more flexible in this case.

Where can we use RAM disk boot?

There is a wide range of where we can use RAM disk images. Booting them over Ethernet is interesting during the bring-up phase as well as for production.

During bring-up we often want to boot into Linux as soon as we have a working Ethernet connection. We can then proceed with formatting flash, stress the memory and CPU, etc.

In production we use RAM disk images to do an initial setup and run production tests. The advantage of using a RAM disk image is that we have a full Linux environment available, can create partitions, format them and can run power-full tools like Python. After we installed the productive image we just reboot and the RAM disk image is gone.

Troubleshooting

There are some things which can go wrong when booting a RAM disk. Here are some points to look for.

Addresses

Often the load addresses are too close and the kernel will e.g. corrupt the RAM disk image. We need to make sure that the addresses kernel_addr_r, fdt_addr_r and ramdisk_addr_r leave enough space for the data and don't overlap.

Relocation

U-Boot relocates the RAM disk image and device tree before booting the kernel. Sometimes this will destroy e.g. the RAM disk image. We can see that U-Boot relocates them in the last two lines of the following output:

## Loading init Ramdisk from Legacy Image at 12200000 ...
   Image Name:   Colibri-iMX6_Reference-Minimal-I
   Image Type:   ARM Linux RAMDisk Image (uncompressed)
   Data Size:    40657952 Bytes = 38.8 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 12100000
   Booting using the fdt blob at 0x12100000
   Loading Ramdisk to 1d939000, end 1ffff420 ... OK
   Loading Device Tree to 1d924000, end 1d938a92 ... OK

If we want to prevent U-Boot from doing this copy operation we can set fdt_high and initrd_high to an all ones address (0xFFFFFFFF for 32bit and 0xFFFFFFFFFFFFFFFF for 64bit).

Colibri iMX6 $ setenv fdt_high 0xFFFFFFFF
Colibri iMX6 $ setenv initrd_high 0xFFFFFFFF

Now we will see that U-boot will not relocate the files:

## Loading init Ramdisk from Legacy Image at 12200000 ...
   Image Name:   Colibri-iMX6_Reference-Minimal-I
   Image Type:   ARM Linux RAMDisk Image (uncompressed)
   Data Size:    40657952 Bytes = 38.8 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 12100000
   Booting using the fdt blob at 0x12100000
   Using Device Tree in place at 12100000, end 12114a92

CPIO file format

If we manually create the cpio file for the RAM disk image we need to make sure that we use the option "-H newc". Else the kernel will not be able to use the file:

host@machine:~$ find . | cpio -o -H newc | gzip > /tmp/ramdisk.cpio.gz
host@machine:~$ mkimage -A arm -O linux -T ramdisk -a 0 -e 0 -C none -d /tmp/ramdisk.cpio.gz /tmp/ramdisk.cpio.gz.u-boot

Init

The kernel needs to know where to find the init executable. This can be specified with the init= bootarg when not booting an initrd image. In comparison, however, initrd used rdinit= as bootarg. The default is /init. Also, if init should crash we can directly load a shell without using init. We just need to set rdinit=/bin/sh as bootarg.

Missing Kernel Config

We need to make sure the kernel supports initrd by setting the following configs:

CONFIG_BLK_DEV_INITRD=y
CONFIG_RD_GZIP=y