Getting started with memory layouts

So in previous blog post on solving CTF challenge created by me on real alike firmware, I did not touched subject of u-boot bootloader in depth and how such firmware was created in first place. Welcome back here again today you will see steps to create your own firmware for qemu and making it work with u-boot. Many of the stuff you might have encoutered during computer science class and you will be amazed how the concepts of fundamental system architecture and operating system works in action.

First thing you should know is how a typical linux os boots up. Today we will work with emulated ARM Cortex A9 board. Let’s have look at memory mapping for Cortex-A9 CPU.

Address range Size Description
0xE000_0000-0xFFFF_FFFF 512MB External AXI between daughterboards
0xA000_0000-0xDFFF_FFFF 1GB Daughterboard, private
0x8000_0000-0x9FFF_FFFF 512MB Local DDR2
0x8000_0000-0x81FF_FFFF 64MB Remappable memory location
0x6000_0000-0x7FFF_FFFF 512MB Local DDR2 lower
0x5C00-0000-0x5FFF_FFFF 64MB Reserved
0x4000_0000-0x5BFF_FFFF 448MB Motherboard peripherals, typically, memory devices
0x2000_0000-0x3FFF_FFFF 512MB Reserved
0x1002_0000-0x1FFF_FFFF ~256MB Daughterboard, private
0x1000_0000-0x1001_FFFF 128KB Motherboard peripherals, CS7
0x0000_0000-0x0FFF_FFFF 64MB Remappable memory section

As you can see above RAM starts at 0x60000000 ends at 0x7FFFFFFF, Cortex-A9 supports upto 1GB RAM so basically for board will have two 512MB RAM banks all data above 512MB+ will be stored at 0x80000000-0x9FFFFFFF memory range. You can verify this in QEMU source code as well.

static void a9_daughterboard_init(const VexpressMachineState *vms,
                                  ram_addr_t ram_size,
                                  const char *cpu_type,
                                  qemu_irq *pic)
{
    MachineState *machine = MACHINE(vms);
    MemoryRegion *sysmem = get_system_memory();
    MemoryRegion *lowram = g_new(MemoryRegion, 1);
    ram_addr_t low_ram_size;

    if (ram_size > 0x40000000) {
        /* 1GB is the maximum the address space permits */
        error_report("vexpress-a9: cannot model more than 1GB RAM");
        exit(1);
    }

    low_ram_size = ram_size;
    if (low_ram_size > 0x4000000) {
        low_ram_size = 0x4000000;
    }
    /* RAM is from 0x60000000 upwards. The bottom 64MB of the
     * address space should in theory be remappable to various
     * things including ROM or RAM; we always map the RAM there.
     */
    memory_region_init_alias(lowram, NULL, "vexpress.lowmem", machine->ram,
                             0, low_ram_size);
    memory_region_add_subregion(sysmem, 0x0, lowram);
    memory_region_add_subregion(sysmem, 0x60000000, machine->ram); //RAM start address

    /* 0x1e000000 A9MPCore (SCU) private memory region */
    init_cpus(machine, cpu_type, TYPE_A9MPCORE_PRIV, 0x1e000000, pic,
              vms->secure, vms->virt);

    /* Daughterboard peripherals : 0x10020000 .. 0x20000000 */

    /* 0x10020000 PL111 CLCD (daughterboard) */
    sysbus_create_simple("pl111", 0x10020000, pic[44]);

    /* 0x10060000 AXI RAM */
    /* 0x100e0000 PL341 Dynamic Memory Controller */
    /* 0x100e1000 PL354 Static Memory Controller */
    /* 0x100e2000 System Configuration Controller */

    sysbus_create_simple("sp804", 0x100e4000, pic[48]);
    /* 0x100e5000 SP805 Watchdog module */
    /* 0x100e6000 BP147 TrustZone Protection Controller */
    /* 0x100e9000 PL301 'Fast' AXI matrix */
    /* 0x100ea000 PL301 'Slow' AXI matrix */
    /* 0x100ec000 TrustZone Address Space Controller */
    /* 0x10200000 CoreSight debug APB */
    /* 0x1e00a000 PL310 L2 Cache Controller */
    sysbus_create_varargs("l2x0", 0x1e00a000, NULL);
}

https://github.com/qemu/qemu/blob/master/hw/arm/vexpress.c

One more thing to know is address of flash memory from which during boot process we will load things into RAM such as kernel, filesystem/ramfs and device tree blob ((dtb). In qemu Cortex-A9 board will have two Parallel NOR Flash of 64MB. So while making firmware maximum filesize we can use for firmware binary would be 128MB and pass to -pflash option when starting virtual machine. You can check out datasheets of these kind parallel NOR flash chips out in market https://www.micron.com/-/media/client/global/documents/products/data-sheet/nor-flash/parallel/m29ew/m29ew_32mb_128mb.pdf as real example. Enough jumping around lets again peek into Qemu source code.

static hwaddr motherboard_legacy_map[] = {
    [VE_NORFLASHALIAS] = 0,
    /* CS7: 0x10000000 .. 0x10020000 */
    [VE_SYSREGS] = 0x10000000,
    [VE_SP810] = 0x10001000,
    [VE_SERIALPCI] = 0x10002000,
    [VE_PL041] = 0x10004000,
    [VE_MMCI] = 0x10005000,
    [VE_KMI0] = 0x10006000,
    [VE_KMI1] = 0x10007000,
    [VE_UART0] = 0x10009000,
    [VE_UART1] = 0x1000a000,
    [VE_UART2] = 0x1000b000,
    [VE_UART3] = 0x1000c000,
    [VE_WDT] = 0x1000f000,
    [VE_TIMER01] = 0x10011000,
    [VE_TIMER23] = 0x10012000,
    [VE_VIRTIO] = 0x10013000,
    [VE_SERIALDVI] = 0x10016000,
    [VE_RTC] = 0x10017000,
    [VE_COMPACTFLASH] = 0x1001a000,
    [VE_CLCD] = 0x1001f000,
    /* CS0: 0x40000000 .. 0x44000000 */
    [VE_NORFLASH0] = 0x40000000,            //First 64MB parallel NOR flash
    /* CS1: 0x44000000 .. 0x48000000 */
    [VE_NORFLASH1] = 0x44000000,            //Second 64MB parallel NOR flash
    /* CS2: 0x48000000 .. 0x4a000000 */
    [VE_SRAM] = 0x48000000,
    /* CS3: 0x4c000000 .. 0x50000000 */
    [VE_VIDEORAM] = 0x4c000000,
    [VE_ETHERNET] = 0x4e000000,
    [VE_USB] = 0x4f000000,
};

Putting these all information togather we get following ranges for RAM and flash memory

Memory Address Range Size Description
0x60000000-0x7FFFFFFF 512MB RAM <= 512MB
0x80000000-0x9FFFFFFF 512MB RAM > 512MB
0x40000000-0x44000000 64MB NOR FLASH 1
0x48000000-0x4a000000 64MB NOR FLASH 2

In this tutorial we will be only using 512MB RAM and NOR FLASH 1. Now lets see the build process

kernel, file system and u-boot compilation

Whenever I meet new people in infosec most of us are always seem to be affraid of compiling stuff from scratch but trust me with enough of practice you will develop intuition and thanks to age of automation it’s more fun to do now these days.

Kernel compilation steps

Download latest stable kernel from https://www.kernel.org/

wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.12.tar.xz

Now you need to install ARM cross compiler since target CPU we are dealing with is ARM

sudo apt update && sudo apt install gcc-arm-none-eabi -y

Generate intial configuration file

make ARCH=arm CROSS_COMPILE=arm-none-eabi- vexpress_defconfig

You can edit options with menuconfig, this step is optional like if you know what you are doing and which functionalities inside kernel you want to enable. Most of the time for research purposes people like to enable kernel debugging related ones. But for this post u can directly compile and boot into basic system without doing anything extra and skip command below.

make ARCH=arm CROSS_COMPILE=arm-none-eabi- menuconfig
screenshot
menuconfig

After configuring desired options you can start compilation

make -j$(nproc) ARCH=arm CROSS_COMPILE=arm-none-eabi-
screenshot
kernel compilation in progress

Grab cup of coffee because it’s gonna take some few 5-10min (depending on your system specs)

Once finished kernel zImage file should be at /arch/arm/boot in present working directory. Now copy compiled zImage and vexpress-v2p-ca9.dtb (Device Tree Blob) to your desired location.

cp ./arch/arm/boot/zImage ~/my_firmware && cp ./arch/arm/boot/dts/vexpress-v2p-ca9.dtb ~/my_firmware

Test them with qemu

qemu-system-arm -M vexpress-a9 -m 512M -kernel zImage -dtb vexpress-v2p-ca9.dtb -nographic

Output from successful boot

Booting Linux on physical CPU 0x0Linux version 5.6.12cjhackerz-arm-kernel ([email protected]) (gcc version 9.3.0 (Arch Repository)) #1 SMP Tue May 12 04:33:08 IST 2020
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt: Machine model: V2P-CA9
Memory policy: Data cache writeback
Reserved memory: created DMA memory pool at 0x4c000000, size 8 MiB
OF: reserved mem: initialized node [email protected], compatible id shared-dma-pool
cma: Reserved 16 MiB at 0x7f000000
CPU: All CPU(s) started in SVC mode.
percpu: Embedded 19 pages/cpu s45708 r8192 d23924 u77824
Built 1 zonelists, mobility grouping on.  Total pages: 130048
Kernel command line: console=ttyAMA0
printk: log_buf_len individual max cpu contribution: 4096 bytes
printk: log_buf_len total cpu_extra contributions: 12288 bytes
printk: log_buf_len min size: 16384 bytes
printk: log_buf_len: 32768 bytes
printk: early log buf free: 14864(90%)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 491996K/524288K available (7168K kernel code, 511K rwdata, 1732K rodata, 1024K init, 177K bss, 15908K reserved, 16384K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
rcu: Hierarchical RCU implementation.
rcu: 	RCU event tracing is enabled.
rcu: 	RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
GIC CPU mask not found - kernel will fail to boot.
GIC CPU mask not found - kernel will fail to boot.
L2C: platform modifies aux control register: 0x02020000 -> 0x02420000
L2C: DT/platform modifies aux control register: 0x02020000 -> 0x02420000
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 dynamic clock gating disabled, standby mode disabled
L2C-310 cache controller enabled, 8 ways, 128 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x46420001
random: get_random_bytes called from start_kernel+0x2fc/0x498 with crng_init=0
sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
clocksource: arm,sp804: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
Failed to initialize '/[email protected]/motherboard/[email protected],00000000/[email protected]': -22
smp_twd: clock not found -2
Console: colour dummy device 80x30
Calibrating local timer... 92.29MHz.
Calibrating delay loop... 580.40 BogoMIPS (lpj=2902016)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
CPU: Testing write buffer coherency: ok
CPU0: Spectre v2: using BPIALL workaround
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x60100000 - 0x60100060
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
smp: Brought up 1 node, 1 CPU
SMP: Total of 1 processors activated (580.40 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 0
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor ladder
hw-breakpoint: debug architecture 0x4 unsupported.
Serial: AMBA PL011 UART driver
10009000.uart: ttyAMA0 at MMIO 0x10009000 (irq = 29, base_baud = 0) is a PL011 rev1
printk: console [ttyAMA0] enabled
1000a000.uart: ttyAMA1 at MMIO 0x1000a000 (irq = 30, base_baud = 0) is a PL011 rev1
1000b000.uart: ttyAMA2 at MMIO 0x1000b000 (irq = 31, base_baud = 0) is a PL011 rev1
1000c000.uart: ttyAMA3 at MMIO 0x1000c000 (irq = 32, base_baud = 0) is a PL011 rev1
OF: amba_device_add() failed (-19) for /[email protected]/motherboard/[email protected],00000000/[email protected]
OF: amba_device_add() failed (-19) for /[email protected]
OF: amba_device_add() failed (-19) for /[email protected]
OF: amba_device_add() failed (-19) for /[email protected]
irq: type mismatch, failed to map hwirq-75 for [email protected]!
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arm,sp804
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 5 counters available
workingset: timestamp_bits=30 max_order=17 bucket_order=0
squashfs: version 4.0 (2009/01/31) Phillip Lougher
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
9p: Installing v9fs 9p2000 file system support
jitterentropy: Initialization failed with host not compliant with requirements: 2
io scheduler mq-deadline registered
io scheduler kyber registered
i2c i2c-0: Added multiplexed i2c bus 2
drm-clcd-pl111 1001f000.clcd: assigned reserved memory node [email protected]
drm-clcd-pl111 1001f000.clcd: using device-specific reserved memory
drm-clcd-pl111 1001f000.clcd: initializing Versatile Express PL111
drm-clcd-pl111 1001f000.clcd: core tile graphics present
drm-clcd-pl111 1001f000.clcd: this device will be deactivated
Error: Driver 'vexpress-muxfpga' is already registered, aborting...
drm-clcd-pl111 10020000.clcd: initializing Versatile Express PL111
drm-clcd-pl111 10020000.clcd: DVI muxed to daughterboard 1 (core tile) CLCD
drm-clcd-pl111 10020000.clcd: found bridge on endpoint 0
drm-clcd-pl111 10020000.clcd: Using non-panel bridge
[drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[drm] No driver support for vblank timestamp query.
[drm] Initialized pl111 1.0.0 20170317 for 10020000.clcd on minor 0
Console: switching to colour frame buffer device 128x48
drm-clcd-pl111 10020000.clcd: fb0: pl111drmfb frame buffer device
physmap-flash 40000000.flash: physmap platform flash device: [mem 0x40000000-0x43ffffff]
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
physmap-flash 40000000.flash: physmap platform flash device: [mem 0x44000000-0x47ffffff]
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
Concatenating MTD devices:
(0): "40000000.flash"
(1): "40000000.flash"
into device "40000000.flash"
physmap-flash 48000000.psram: physmap platform flash device: [mem 0x48000000-0x49ffffff]
libphy: Fixed MDIO Bus: probed
libphy: smsc911x-mdio: probed
smsc911x 4e000000.ethernet eth0: MAC Address: 52:54:00:12:34:56
isp1760 4f000000.usb: bus width: 32, oc: digital
isp1760 4f000000.usb: NXP ISP1760 USB Host Controller
isp1760 4f000000.usb: new USB bus registered, assigned bus number 1
isp1760 4f000000.usb: Scratch test failed.
isp1760 4f000000.usb: can't setup: -19
isp1760 4f000000.usb: USB bus 1 deregistered
usbcore: registered new interface driver usb-storage
rtc-pl031 10017000.rtc: registered as rtc0
mmci-pl18x 10005000.mmci: Got CD GPIO
mmci-pl18x 10005000.mmci: Got WP GPIO
mmci-pl18x 10005000.mmci: mmc0: PL181 manf 41 rev0 at 0x10005000 irq 25,26 (pio)
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
aaci-pl041 10004000.aaci: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 24
aaci-pl041 10004000.aaci: FIFO 512 entries
oprofile: using arm/armv7-ca9
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
Registering SWP/SWPB emulation handler
rtc-pl031 10017000.rtc: setting system clock to 2020-05-11T23:34:36 UTC (1589240076)
ALSA device list:
  #0: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 24
input: AT Raw Set 2 keyboard as /devices/platform/[email protected]/[email protected]:motherboard/[email protected]:motherboard:[email protected],00000000/10006000.kmi/serio0/input/input0
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/[email protected]/[email protected]:motherboard/[email protected]:motherboard:[email protected],00000000/10007000.kmi/serio1/input/input2
VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
Please append a correct "root=" boot option; here are the available partitions:
1f00          131072 mtdblock0 
 (driver?)
1f01           32768 mtdblock1 
 (driver?)
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.6.12cjhackerz-arm-kernel #1
Hardware name: ARM-Versatile Express
[<801124f8>] (unwind_backtrace) from [<8010cc6c>] (show_stack+0x10/0x14)
[<8010cc6c>] (show_stack) from [<807a48b8>] (dump_stack+0x90/0xa4)
[<807a48b8>] (dump_stack) from [<8012b12c>] (panic+0x104/0x304)
[<8012b12c>] (panic) from [<80a01504>] (mount_block_root+0x1ec/0x280)
[<80a01504>] (mount_block_root) from [<80a016bc>] (mount_root+0x124/0x140)
[<80a016bc>] (mount_root) from [<80a0182c>] (prepare_namespace+0x154/0x190)
[<80a0182c>] (prepare_namespace) from [<807bc05c>] (kernel_init+0x8/0x110)
[<807bc05c>] (kernel_init) from [<801010e8>] (ret_from_fork+0x14/0x2c)
Exception stack(0x9e493fb0 to 0x9e493ff8)
3fa0:                                     00000000 00000000 00000000 00000000
3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

If everything was done correctly till now your kernel should boot and stop at kernel panic. Don’t worry it’s okay because kernel was not able to find root file system. To exit out of qemu press Ctrl+a x (control a and then x).

Creating ramdisk for root filesystem in RAM with buildroot

Get cross compiler for our userspace programs

sudo apt install binutils-arm-linux-gnueabi gcc-arm-linux-gnueabi g++-arm-linux-gnueabi -y

Download latest stable buildroot release https://buildroot.org/download.html

wget https://buildroot.org/downloads/buildroot-2020.02.1.tar.gz && tar -xvf 

Generate .config file

cd buildroot-2020.02.1 && make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- qemu_arm_vexpress_defconfig

Now we need to edit config and tell buildroot to generate gzip compressed ramdisk

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- xconfig
screenshot
File system images > cpio the root file system

You can can also install some software such as Apache web server, tinyssh (SSH server) etc. from Target packages menu. Go to Kernel menu and uncheck it since we already build it seprately. When ready just do make. It will download stuff and start compilation.

make -j$(nproc) ARCH=arm

Time to get another cup of coffee this is gonna take more than hour :D

screenshot
Successful compilation of filesystem

Your prepared filesystem files can be found under output/images

cp ./output/images/rootfs.cpio.gz ~/my_firmware

To test everything now you can directly boot into system from your files

qemu-system-arm -M vexpress-a9 -m 512M \
-kernel zImage -dtb vexpress-v2p-ca9.dtb \
-initrd rootfs.cpio.gz \
-append "root=/dev/ram console=ttyAMA0,38400n8" -nographic

Time to build u-boot and configure it

There are many ways to boot os from u-boot but today we will see single component uImage booting (http://www.denx.de/wiki/pub/U-Boot/Documentation/multi_image_booting_scenarios.pdf). uImage fileformat is wrapper for u-boot. It tells bootloader where to load given uImage file in memory, what kind of file it is (kernel, ramdisk, dtb etc) and is there any compression present or not.

Whenever linux system boots, bootloader puts kernel at start of RAM. After that intial filesystem and dtb is loaded into ram. Bootloader execute kernel by passing required kernel arguments. Think kernel as big binary program just like .ELF file. And like any other program it takes some arguments for example if you type ls -a it shows you all files and for ls -al is shows you all files with permission details. Format for kernel argument is var1=value1 var2=value2

For booting from u-boot we will pass following kernel arguments to kernel root=/dev/ram console=ttyAMA0,38400n8 Above arguments tells kernel to seek root filesystem in ram and set console output to serial port on specified baudrate.

Let’s download and compile u-boot from source, pick any most recent version from https://ftp.denx.de/pub/u-boot/

wget https://ftp.denx.de/pub/u-boot/u-boot-2020.04.tar.bz2 && tar -xvf u-boot-2020.04.tar.bz2
cd u-boot-2020.04 && make ARCH=arm CROSS_COMPILE=arm-none-eabi- vexpress_ca9x4_defconfig && make -j$(nproc) ARCH=arm CROSS_COMPILE=arm-none-eabi-

Finally copy it to our project directory

cp ./u-boot ~/my_firmware

To test it we will pass file to -kernel option in qemu will put it at start of ram so it will directly boot into u-boot bootloader.

qemu-system-arm -M vexpress-a9 -m 512M -kernel u-boot
U-Boot 2020.04 (May 12 2020 - 08:55:56 +0530)

DRAM:  512 MiB
WARNING: Caches not enabled
Flash: 128 MiB
MMC:   MMC: 0
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   smc911x-0
Hit any key to stop autoboot:  0 
MMC Device 1 not found
no mmc device at slot 1
Card did not respond to voltage select!
smc911x: MAC 52:54:00:12:34:56
smc911x: detected LAN9118 controller
smc911x: phy initialized
smc911x: MAC 52:54:00:12:34:56
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (3 ms)
*** Warning: no boot file name; using '0A00020F.img'
Using smc911x-0 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename '0A00020F.img'.
smc911x: MAC 52:54:00:12:34:56
...
TFTP error: trying to overwrite reserved memory...
smc911x: MAC 52:54:00:12:34:56
Wrong Image Format for bootm command
ERROR: can't get kernel image!

If u get output as shown above it means now we can finally create our firmware image file

One last information you should be aware of is that after u-boot executes it will reallocate itself to 0x60800000 (+8MB from start of the RAM).

screenshot
U-boot self reallocation
screenshot
U-boot self reallocation hexdump and RAM comparison

This to make some space free for kernel and filesystem because after boot kernel will go at the start of the RAM. We need to know filesize of all files to avoid overwriting in memory by providing enough space between them. I decided to have approx 2MB distance between each file which gets loaded into RAM.

ls -al --block-size=K zImage rootfs.cpio.gz vexpress-v2p-ca9.dtb u-boot
-rw-r--r-- 1 cjhackerz cjhackerz 6294K May 12 07:59 rootfs.cpio.gz
-rwxr-xr-x 1 cjhackerz cjhackerz 4761K May 12 09:00 u-boot
-rw-r--r-- 1 cjhackerz cjhackerz   14K May 12 04:59 vexpress-v2p-ca9.dtb
-rwxr-xr-x 1 cjhackerz cjhackerz 4653K May 12 04:59 zImage

Our kernel(zImage) is almost 5MB, ramdisk(rootfs.cpio.gz) 6.5MB and device tree blob (vexpress-v2p-ca9.dtb) 14KB so basically we want to put our u-boot after 16MB (Address: 0x61000000) from start of the RAM. So our final memory mapping in RAM should be as table below:

Address Offset (from start of the RAM) Filename Size
0x60010000 0x10000(64KB) zImage 4.5MB
0x60600000 0x600000(6MB) rootfs.cpio.gz 6.1MB

Now create uImage files for u-boot

mkimage -A arm -C none -O linux -T kernel -d zImage -a 0x60010000 -e 0x60010000 kernel.uImage
mkimage -A arm -C gzip -O linux -T ramdisk -d rootfs.cpio.gz -a 0x60600000 -e 0x60600000 rootfs.uImage

Now we need to output them to packed firmware file. To do that we will do is copy files from TFTP server to particular addresses empty NOR flash. And qemu will automatically store output of flash contents of our files once data is copied to it.

sudo apt install -y tftpd-hpa && sudo systemctl enable tftpd-hpa && sudo systemctl restart tftpd-hpa

Copy files to server root

sudo cp *.uImage zImage rootfs.cpio.gz /srv/tftp/

Start u-boot with -kernel option (let autoboot finish with error) and pass empty file to -pflash option

dd if=/dev/zero of=firmware.bin bs=1 count=64M

Note: Please follow my ARM lab setup guide to setup NAT networking in qemu.

sudo qemu-system-arm -M vexpress-a9 -m 512M -kernel u-boot -pflash firmware.bin -nic tap

Follow steps in video given below

So I am loading kernel.uImage at address 0x61000000, rootfs.uImage at 0x62000000 and vexpress-v2p-ca9.dtb at 0x63000000

cp [source address of RAM] [destination address in flash] [number of bytes in hex]

So we will be copying data to RAM from TFTP and from RAM to flash memory. In NOR Flash I leaving 1 sector empty so in flinfo command in u-boot I can visually see from padding that after empty sector new file beings. Before writing to flash you have to turn of write protection and format it with erase command.

=> flinfo

Bank # 1: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
  Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
  Erase timeout: 16384 ms, write timeout: 3 ms
  Buffer write timeout: 3 ms, buffer size: 2048 bytes

  Sector Start Addresses:
  40000000 E RO   40040000   RO   40080000        400C0000        40100000      
  40140000        40180000        401C0000        40200000        40240000      
  40280000        402C0000        40300000        40340000        40380000      
  403C0000        40400000        40440000        40480000        404C0000      
  40500000        40540000        40580000        405C0000        40600000      
  40640000        40680000        406C0000        40700000        40740000      
  40780000        407C0000        40800000        40840000        40880000      
  408C0000        40900000        40940000        40980000        409C0000      
  40A00000        40A40000        40A80000        40AC0000        40B00000      
  40B40000        40B80000        40BC0000        40C00000        40C40000      
  40C80000        40CC0000        40D00000        40D40000        40D80000      
  40DC0000        40E00000        40E40000        40E80000        40EC0000      
  40F00000        40F40000        40F80000        40FC0000        41000000      
  41040000        41080000        410C0000        41100000        41140000      
  41180000        411C0000        41200000        41240000        41280000 E    
  412C0000        41300000        41340000        41380000        413C0000      
  41400000        41440000        41480000        414C0000        41500000      
  41540000        41580000        415C0000        41600000        41640000      
  41680000        416C0000        41700000        41740000        41780000      
  417C0000        41800000        41840000        41880000        418C0000      
  41900000        41940000        41980000        419C0000        41A00000      
  41A40000        41A80000        41AC0000        41B00000        41B40000      
  41B80000        41BC0000        41C00000        41C40000        41C80000      
  41CC0000        41D00000        41D40000        41D80000        41DC0000      
  41E00000        41E40000        41E80000        41EC0000        41F00000      
  41F40000        41F80000        41FC0000        42000000        42040000      
  42080000        420C0000        42100000        42140000        42180000      
  421C0000        42200000        42240000        42280000        422C0000      
  42300000        42340000        42380000        423C0000        42400000      
  42440000        42480000        424C0000        42500000        42540000      
  42580000        425C0000        42600000        42640000        42680000      
  426C0000        42700000        42740000        42780000        427C0000      
  42800000        42840000        42880000        428C0000        42900000      
  42940000        42980000        429C0000        42A00000        42A40000      
  42A80000        42AC0000        42B00000        42B40000        42B80000 E    
  42BC0000        42C00000 E      42C40000 E      42C80000 E      42CC0000 E    
  42D00000 E      42D40000 E      42D80000 E      42DC0000 E      42E00000 E    
  42E40000 E      42E80000 E      42EC0000 E      42F00000 E      42F40000 E    
  42F80000 E      42FC0000 E      43000000 E      43040000 E      43080000 E    
  430C0000 E      43100000 E      43140000 E      43180000 E      431C0000 E    
  43200000 E      43240000 E      43280000 E      432C0000 E      43300000 E    
  43340000 E      43380000 E      433C0000 E      43400000 E      43440000 E    
  43480000 E      434C0000 E      43500000 E      43540000 E      43580000 E    
  435C0000 E      43600000 E      43640000 E      43680000 E      436C0000 E    
  43700000 E      43740000 E      43780000 E      437C0000 E      43800000 E    
  43840000 E      43880000 E      438C0000 E      43900000 E      43940000 E    
  43980000 E      439C0000 E      43A00000 E      43A40000 E      43A80000 E    
  43AC0000 E      43B00000 E      43B40000 E      43B80000 E      43BC0000 E    
  43C00000 E      43C40000 E      43C80000 E      43CC0000 E      43D00000 E    
  43D40000 E      43D80000 E      43DC0000 E      43E00000 E      43E40000 E    
  43E80000 E      43EC0000 E      43F00000 E      43F40000 E      43F80000 E    
  43FC0000 E

So I ended up with following mapping in NOR flash:

  • 40040000 [kernel.uImage]
  • 412C0000 [rootfs.uImage]
  • 42BC0000 [vexpress-v2p-ca9.dtb]

After copying data to flash just directly close your qemu. Changes will now show up in firmware.bin file.

binwalk firmware.bin 

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
262144        0x40000         uImage header, header size: 64 bytes, header CRC: 0xFCAABEFB, created: 2020-05-12 05:37:13, image size: 4763936 bytes, Data Address: 0x60010000, Entry Point: 0x60010000, data CRC: 0x92589585, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: ""
262208        0x40040         Linux kernel ARM boot executable zImage (little-endian)
280824        0x448F8         gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
17039360      0x1040000       uImage header, header size: 64 bytes, header CRC: 0xB41113D6, created: 2020-05-12 05:37:40, image size: 6444900 bytes, Data Address: 0x60600000, Entry Point: 0x60600000, data CRC: 0x6711A199, OS: Linux, CPU: ARM, image type: RAMDisk Image, compression type: gzip, image name: ""
17039424      0x1040040       gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
19660800      0x12C0000       uImage header, header size: 64 bytes, header CRC: 0xB41113D6, created: 2020-05-12 05:37:40, image size: 6444900 bytes, Data Address: 0x60600000, Entry Point: 0x60600000, data CRC: 0x6711A199, OS: Linux, CPU: ARM, image type: RAMDisk Image, compression type: gzip, image name: ""
19660864      0x12C0040       gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
22576367      0x1587CEF       MySQL MISAM index file Version 7
45875200      0x2BC0000       device tree image (dtb)
cd u-boot-2020.04 && make ARCH=arm CROSS_COMPILE=arm-none-eabi- menuconfig

Enable boot arguments (press space to select option) below that option press enter to input our boot argument

screenshot
Entering value for bootargs

Enable a default value for bootcmd

screenshot
Entering value for bootargs
screenshot
Entering value for bootargs

Exit and Save changes finally compile it again

make -j$(nproc) ARCH=arm CROSS_COMPILE=arm-none-eabi-
cp u-boot ~/my_firmware/u-boot-flashboot

Testing for one last time

qemu-system-arm -M vexpress-a9 -m 512M -kernel u-boot-flashboot -pflash firmware.bin -nographic
screenshot
Final outcome

TheEnd

And that’s it. Yes indeeed it is bit hard and time consuming process but one can learn a lot about u-boot bootloader functionalities. To sum up what we did was understood our memomry mappings, compiled our stuff and put everything inside a single file which will be used as emulated NOR Flash inside Qemu. And from flash memory bootloader will load stuff back into the RAM as we decided. If you liked this blog post please share it as much as you can so more people can learn this knowledge. Don’t forget to follow me on twitter as well for latest updates from me @cjhackerz