ARM Foundation Model - Free solution to make your first steps on ARM-64bit and ARM Software stack by Olivier on 24/12/2015 10:11

Get the ARM Foundation Model

Download ARM Foundation Model: http://www.arm.com/products/tools/models/fast-models/foundation-model.php.

Try the model: ./models/Linux64_GCC-4.7/Foundation_Platform --image examples/hello.axf

ERROR: cpu0.gicv3_cpu_if: Virtual LPIs can not be supported in GICv2 only mode. Defaulting virtual lpi support to false.
terminal_0: Listening for serial connection on port 5000
terminal_1: Listening for serial connection on port 5001
terminal_2: Listening for serial connection on port 5002
terminal_3: Listening for serial connection on port 5003
Simulation is started
Hello, 64-bit world!

ARM Software Stack

ARM Architecture has two worlds / execution environment: Trusted World and Non-Trusted World. These two execution environments run at the same time.
Non-Trusted world cannot access Trusted World. It protects trusted information (such as payment details) to be accessed by malicious software running in Non-Trusted world - although risks exist if the Trusted World is not configured properly or the interface between both Worlds are not correctly designed.

Most ARM (and non-ARM) platforms have different types of memory:

The Open-Source ARM Software stack contains these following modules:

Run the full stack from pre-built binaries

  1. Download the archive that contains ARM Trusted Firmware, UEFI binaries, Linux Kernel and Device Tree at http://snapshots.linaro.org/member-builds/armlt-platforms-release/16/fvp-latest-oe-uefi.zip (5.9 MB)

  2. Download OpenEmbedded Minimal filesystem at https://releases.linaro.org/openembedded/juno-lsk/15.09/lt-vexpress64-openembeddedminimal-armv8-gcc-4.920150912-729.img.gz (54.9M)

  3. Launch it!

    export MODEL=$PWD/Foundation_Platformpkg/models/Linux64_GCC-4.7/Foundation_Platform
    export DISK=$PWD/prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img
    ./prebuilt/run_model.sh ./prebuilt/
    

Build and run the full stack

The initial step is to download the AArch64 toolchain archive http://releases.linaro.org/14.11/components/toolchain/binaries/aarch64-linux-gnu/gcc-linaro-4.9-2014.11-x8664aarch64-linux-gnu.tar.xz.
The location where you will unpack the archive will be defined as <your-toolchain-path> in the rest of the document.

Build the software stack in a couple of command lines

Use Google's repo tool. Let's build the software stack with the Linaro recommended instructions.
From this document https://community.arm.com/docs/DOC-10831, we have the choice between:

If we want to run Android later on without to have to switch the Linux kernel, we should go for LSK.

  1. Install repo: https://source.android.com/source/downloading.html#installing-repo

  2. Get the source

    mkdir repo_src && cd repo_src
    repo init -u https://git.linaro.org/landing-teams/working/arm/manifest -b 15.11 -m pinned-lsk.xml
    repo sync -j8
    
  3. Build the sources

    export CROSS_COMPILE=<your-toolchain-path>/gcc-linaro-5.1-2015.08-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
    ./build-scripts/build-all.sh fvp-oe
    ./build-scripts/build-all.sh fvp-oe package
    
  4. Run the newly built binaries

    ./prebuilt/run_model.sh src/output/fvp-oe/uefi
    

Build the software stack components by components

Create the directory that will contain all the ARM Software stack components:

mkdir src && cd src
export CROSS_COMPILE=<your-toolchain-path>/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-

Build UEFI

  1. Clone the UEFI repository

    git clone https://github.com/tianocore/edk2.git
    cd edk2
    
  2. Edit the default Linux command line in ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc to remove the initrd (defined by the argument -f) as we want to boot the Open Embedded disk image and we do not use an initrd for our Linux kernel.

    diff --git a/ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc b/ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc
    index f82de7e..30e3cb5 100644
    --- a/ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc
    +++ b/ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc
    @@ -167,7 +167,7 @@
       #
       gArmPlatformTokenSpaceGuid.PcdDefaultBootDescription|L"Linux from SemiHosting"
       gArmPlatformTokenSpaceGuid.PcdDefaultBootDevicePath|L"Fv(87940482-FC81-41C3-87E6-399CF85AC8A0)/LinuxLoader.efi"
    -  gArmPlatformTokenSpaceGuid.PcdDefaultBootArgument|L"VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/Image -f VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/filesystem.cpio.gz -c \"console=ttyAMA0 earlycon=pl011,0x1c090000 debug user_debug=31 loglevel=9\""
    +  gArmPlatformTokenSpaceGuid.PcdDefaultBootArgument|L"VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/Image -c \"console=ttyAMA0 earlycon=pl011,0x1c090000 debug user_debug=31 loglevel=9 root=/dev/vda2\""
    
       # Use the serial console (ConIn & ConOut) and the Graphic driver (ConOut)
       gArmPlatformTokenSpaceGuid.PcdDefaultConOutPaths|L"VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenPcAnsi();VenHw(407B400
    
  3. Build UEFI

    make -f ArmPlatformPkg/Scripts/Makefile EDK2_TOOLCHAIN=GCC49 EDK2_ARCH=AARCH64 EDK2_DSC=ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc GCC49_AARCH64_PREFIX=${CROSS_COMPILE} EDK2_MACROS="-DUSE_ARM_BDS=1"
    

    USE_ARM_BDS should be used because the default BDS (UEFI Boot Manager) expects Linux kernel to be part of the EFI Boot partition which does not exist on the Linaro file system. The ARM BDS can load the Linux kernel and device tree from semihosting.

    (...)
    Fd File Name:FVP_AARCH64_EFI_SEC
    
    Generate Region at Offset 0x0
       Region Size = 0x80000
       Region Name = FV
    
    Generating FVMAIN_SEC FV
    #####
    Generate Region at Offset 0x0
       Region Size = 0x80000
       Region Name = FV
    Fd File Name:FVP_AARCH64_EFI
    
    Generate Region at Offset 0x0
       Region Size = 0x280000
       Region Name = FV
    
    Generating FVMAIN_COMPACT FV
    ####
    Generating FVMAIN FV
    ####################################
    ########################################
    ########################################
    ########
    Generate Region at Offset 0x0
       Region Size = 0x280000
       Region Name = FV
    
    GUID cross reference file can be found at /home/olivier/src/edk2/Build/ArmVExpress-FVP-AArch64/DEBUG_GCC49/FV/Guid.xref
    
    FV Space Information
    FVMAIN_SEC [8%Full] 524288 total, 45056 used, 479232 free
    FVMAIN_COMPACT [27%Full] 2621440 total, 716984 used, 1904456 free
    FVMAIN [99%Full] 4963200 total, 4963168 used, 32 free
    
    - Done -
    Build end time: 10:18:55, Dec.10 2015
    Build total time: 00:02:11
    
    Warning: 'fip_create' tool is not in the PATH. The UEFI binary will not be added in the Firmware Image Package (FIP).
    

ARM Foundation Model - UEFI

Build ARM Trusted Firmware

  1. Clone the repository

    git clone https://github.com/ARM-software/arm-trusted-firmware.git
    cd arm-trusted-firmware
    
  2. Build ARM Trusted Firmware in DEBUG mode:

    $ make PLAT=fvp all fip BL33=/home/olivier/src/edk2/Build/ArmVExpress-FVP-AArch64/DEBUG_GCC49/FV/FVP_AARCH64_EFI.fd DEBUG=1
    
    (...)
    Firmware Image Package ToC:
    ---------------------------
    - Trusted Boot Firmware BL2: offset=0xB0, size=0x6008
      file: './build/fvp/debug/bl2.bin'
    - EL3 Runtime Firmware BL3-1: offset=0x60B8, size=0x8010
      file: './build/fvp/debug/bl31.bin'
    - Non-Trusted Firmware BL3-3: offset=0xE0C8, size=0x280000
      file: '/home/olivier/src/edk2/Build/ArmVExpress-FVP-AArch64/DEBUG_GCC49/FV/FVP_AARCH64_EFI.fd'
    ---------------------------
    Creating "build/fvp/debug/fip.bin"
    
    Built build/fvp/debug/fip.bin successfully
    

ARM Foundation Model - ARM Trusted Firmware

Build Linux kernel

  1. Clone the repository and use the branch linux-linaro-lsk-v4.1

    git clone https://git.linaro.org/kernel/linux-linaro-stable.git -b linux-linaro-lsk-v4.1
    cd linux-linaro-stable
    
  2. Build the Linux kernel

    $ make ARCH=arm64 defconfig
    $ make ARCH=arm64 -j8 Image
    
    (...)
      LD      vmlinux
      SORTEX  vmlinux
      SYSMAP  System.map
      OBJCOPY arch/arm64/boot/Image
    

ARM Foundation Model - Linux

Run the newly built software stack

  1. Create symbolic link in the current directory to allow UEFI to find the Linux kernel and Device Tree.

    ln -s linux-linaro-stable/arch/arm64/boot/Image
    ln -s arm-trusted-firmware/fdts/fvp-foundation-gicv3-psci.dtb
    
  2. Launch the model

    ../Foundation_Platformpkg/models/Linux64_GCC-4.7/Foundation_Platform --cores=4 --secure-memory --visualization --gicv3 --data="$PWD/arm-trusted-firmware/build/fvp/debug/bl1.bin"@0x0 --data="$PWD/arm-trusted-firmware/build/fvp/debug/fip.bin"@0x08000000 --block-device="/home/olivier/prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img"
    

Some notes:

Linux NFS Boot on the ARM Foundation Model

NFS Boot allows to directly make changes to the file system from your host machine without the need to mount the disk image every time.

Ensure your Linux kernel support NFS

To enable NFS Boot you need to ensure your ARM Foundation Model Linux kernel has support for the Ethernet SMSC 91C111 controller. Check the configuration file of your Linux kernel source tree:

$ grep SMSC .config
CONFIG_NET_VENDOR_SMSC=y
CONFIG_SMSC911X=y
# CONFIG_SMSC911X_ARCH_HOOKS is not set
# CONFIG_SMSC9420 is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_FB_SMSCUFX is not set

CONFIG_SMSC911X is set so it means you can use the ARM Foundation Model network controller with your current kernel binaries. If it is not set then you need to enable it in your kernel configuration and rebuild the kernel.

Configure Network support on the ARM Foundation Model

Start the model with the bridge network:

sudo ../Foundation_Platformpkg/models/Linux64_GCC-4.7/Foundation_Platform --cores=4 --secure-memory --visualization --gicv3 --data="$PWD/arm-trusted-firmware/build/fvp/debug/bl1.bin"@0x0 --data="$PWD/arm-trusted-firmware/build/fvp/debug/fip.bin"@0x08000000 --block-device="/home/olivier/prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img" --network=bridged &
sleep 5
sudo ifconfig ARM0 192.168.1.1 netmask 255.255.255.0

On the target:

sudo ifconfig eth0 192.168.1.2 netmask 255.255.255.0

Note: NAT network with the option --network=nat instead of --network=bridged also works.

More information on the network configuration here.

Setting up the NFS server on your host machine

We will need to copy the file system of your disk image onto your local disk. Let's have a look at the partitions present on your disk image and their locations - fdisk will help us:

$ /sbin/fdisk -lu ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img
Disk ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img: 3221 MB, 3221225472 bytes
255 heads, 63 sectors/track, 391 cylinders, total 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

                      Device Boot      Start         End      Blocks   Id  System
../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img1   *          63      155646       77792    e  W95 FAT16 (LBA)
../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img2          155648     6291455     3067904   83  Linux

You can see the Linux partition starts at the sector number 155648. fidsk says the sector size is 512 bytes.

We use losetup to mount the disk image on /dev/loop0:

$ losetup /dev/loop0 ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img -o $((155648 * 512))

file -s /dev/loop0 confirms us the Linux partition is actually a ext4 partition:

/dev/loop0: Linux rev 1.0 ext4 filesystem data, UUID=a9c947eb-c956-4c39-af5d-1fe3509121a9, volume name "rootfs" (needs journal recovery) (extents) (large files) (huge files)

We finally can mount the device /dev/loop0 to access the partition as part of our file system:

mkdir /tmp/disk
mount /dev/loop0 /tmp/disk

Now the disk image is mounted, we can copy its content to the directory /srv/nfs/lt-vexpress64-openembedded_minimal-armv8:

sudo mkdir /srv/nfs
sudo mount -o loop,offset=$((155648 * 512)) ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img /tmp/disk
sudo cp -ax /tmp/disk /srv/nfs/lt-vexpress64-openembedded_minimal-armv8
umount /dev/loop0
losetup -d /dev/loop0

Setting up NFS server

Check your Linux distribution documentation. For instance a simple installation on the Ubuntu distribution can be found here.
After installing NFS server, make sure the NFS service is started. You can also test your NFS installation is correctly set up by mounting the NFS file system on your host machine:

$ mkdir /tmp/open-embedded
$ sudo mount localhost:/srv/nfs/lt-vexpress64-openembedded_minimal-armv8 /tmp/open-embedded
$ ls /tmp/open-embedded/
bin  boot  dev  EFI  etc  home  lib  lost+found  media  mnt  proc  run  sbin  sys  tmp  usr  var

Next stage is to check whether you can mount the filesystem from the ARM Foundation model.
If you have any issue you can get the NFS error by looking at /var/log/syslog on the host machine: tail -n 20 /var/log/syslog

Once you manage to access the NFS file system from the model, you can now change the Linux kernel command line in UEFI in the file ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc at the line gArmPlatformTokenSpaceGuid.PcdDefaultBootArgument|(..).

Replace root=/dev/vda2 by one of the following arguments:

ARM Foundation Model - NFS

Now, you know how to change and rebuild every components of the ARM Software stack!

Olivier on 24/12/2015 15:06
This blog is also available as a Github document: https://gist.github.com/oliviermartin/1a12b464efd089632c73 to be updated in the future.
Anonymous on 16/02/2016 03:27
Hi,
Thanks for sharing this useful post. I am following your instructions to boot up UEFI and linux kernel. I am successful to boot up UEFI. However, i am unable to boot up linux kernel with the error message " Did not find Linux kernel". I did provide the path as below command:
./Foundation_Platform --cores=4 --no-secure-memory --visualization --gicv3 --data=bl1.bin@0x0 --data=fip.bin@0x8000000 --data=fvp-foundation-gicv2-psci.dtb@0x8300000 --data=Image@0x80080000
I am not sure where else did I miss.
I hope you can advise me on this problem.
Thanks a lot.
Anonymous on 16/02/2016 03:30
Sorry to mention, i am building all the software stack component by component. And if i used the prebuilt binary bl1 and fip then it works.

Login to comment Comment as anonymous