Revision History

Version Date Description

1

2015/12/01

Initial Release.

Introduction

This document builds on the general description of the Tegra boot flow, both in terms of concepts and terminology. It is suggested that document be read first.

The boot flow implemented by the Tegra boot ROM places some single piece of code into RAM and executes it. The boot ROM imposes no restrictions upon exactly how that code implements the rest of the system’s boot flow, nor how it is structured.

L4T and upstream software stacks for chips prior to Tegra210 typically made use of a single bootloader binary which was booted by the boot ROM, in turn loaded the kernel binary, DTB (Device Tree Blob/Binary), and any initrd from disk, and finally jumped directly to the kernel. The overall structure of such as software stack is simple.

The L4T and upstream software stack for Tegra210 is more complex; the boot flow has been split into multiple binaries, and various separate firmware blobs must be loaded and booted in addition to the operating system kernel.

Note that other software stacks, such as ChromeOS, may be structured in a significantly different way.

Components

A brief summary of the components involved in the boot flow is given below:

nvtboot

The bootloader code loaded by the boot ROM and executed on the boot CPU.

nvtboot-cpu

The first code to run on the CCPLEX.

Secure monitor/OS

CCPLEX-based secure monitor and/or secure OS.

Second-stage bootloader

CCPLEX-based final stage bootloader; either U-Boot or cboot.

BPMP-Lite firmware

Power-management firmware that runs on the boot CPU after nvtboot has vacated it.

WB0 resume firmware

Low-level boot firmware used to resume from the SoC-level SC7 low-power mode. This firmware fulfils the same purpose as the LP0 resume firmware in previous versions Tegra.

DTB

nvtboot loads one device-tree to pass to the second-stage bootloader (the "bootloader DTB"), and another to pass on to the operating system that is loaded by the second-stage bootloader (the "kernel DTB").

Boot Flow

nvtboot

As always, the boot ROM loads the first stage bootloader into RAM and executes it on the boot CPU. In this case, nvtboot is this first stage bootloader, and nvtboot expects to be loaded into IRAM rather than SDRAM.

nvtboot loads all other firmware and boot binaries into RAM. Note that this excludes operating system binaries such as the Linux kernel Image file and DTB. Once these binaries are loaded, the CCPLEX is enabled, and nvtboot-cpu executes on the CCPLEX.

nvtboot is responsible for performing:

  • Any required verification of the binaries it loads into RAM.

  • Various SoC initialization that must be performed by secure world or early boot software, e.g. for security reasons. For example, memory controller carve-out configuration, generation of signed configuration data structures used during system resume operations, etc.

  • Any initialization operations common to arbitrary operating systems.

  • Any PMIC/regulator/… programming required for the CCPLEX to operate.

  • Creating RAM-based data structures that are provided to nvtboot-cpu and the second stage bootloaders. These data structures are written over the top of the binaries that will consume them. Those binaries must be structured to expect this by containing a header that will receive the data structure. These data structures include information such as size-of/pointers-to some of the loaded firmware blobs.

nvtboot-cpu

nvtboot-cpu is the first code to run on the CCPLEX, and as such initially runs in EL3 secure state. Early on, nvtboot-cpu jumps to the secure monitor initialization routine, which eventually returns back to nvtboot-cpu in EL2 non-secure state.

After performing various other initialization (e.g. LPDDR4 training), nvtboot-cpu jumps to the second-stage bootloader.

nvtboot-cpu modifies the "kernel" DTB to fill in various properties calculated at run-time, such as LPDDR4 training parameters.

Secure Monitor/OS

As mentioned above, the secure monitor initialization function is called soon after the CCPLEX is enabled. The secure monitor must configure itself, then arrange to return back to its caller in non-secure ARM CPU mode.

The secure monitor is expected to persist and provide PSCI services to the operating system.

L4T does not by default assume the presence of a secure OS. A secure OS may be implemented to provide custom services.

Second-stage bootloader

The second-stage bootloader is primarily responsible for loading, validating, and executing the operating system. It starts execution in EL2 non-secure state.

By default, L4T uses U-Boot for this purpose, although one can manually request the flashing process install cboot (an NVIDIA binary bootloader) instead. Upstream use-cases will always use U-Boot. The rest of this discussion assumes U-Boot, although any generalities apply equally to cboot.

Unlike L4T/upstream software stacks for earlier Tegra SoCs, on Tegra210 U-Boot executes solely on the CCPLEX, and never on the boot CPU. As such, U-Boot SPL is not enabled or used. nvtboot is now responsible for enabling the CCPLEX and causing code to execute on it.

When the L4T Linux kernel boots, it assumes that the BPMP-Lite CPU is already running, and attempts to initialize the communication channel with the BPMP-Lite. Consequently, it is U-Boot’s responsibility to cause the BPMP-Lite firmware to begin execution.

nvtboot/cboot were originally designed to support an Android boot model where nvtboot loads the DTB and edits it to contain various runtime-generated data such as LPDDR4 training parameters, and cboot simply passes this DTB on to the Linux kernel. This data flow can be supported with U-Boot, but would invalidate one of the prime benefits of using such a flexible bootloader; namely, being able to load the kernel, DTB, and initrd from flexible and easily updatable sources such as regular filesystem files, or network servers. Such a data flow would also be rather inconsistent with the flow used on previous Tegra chips.

To work around this, L4T U-Boot includes a feature that allows certain device tree nodes to be copied from the nvtboot "kernel DTB" over the top of the DTB that U-Boot will pass to the kernel it boots. This allows the best of both worlds; U-Boot can still acquire the DTB from a flexible source, yet any data generated by nvtboot can still be passed to the kernel as required for proper operation. This feature is controlled by U-Boot environment variables $fdt_copy_node_names and $fdt_copy_src_addr.

U-Boot also constructs the Linux kernel command-line. While much of the command-line is static and may be hard-coded, some values such as the address of the WB0/SC7 resume firmware may theoretically vary between boots or firmware versions. U-Boot sets various environment variables such as $lp0_vec and $nvdumper_reserved so that relevant portions of the command-line may be dynamically constructed at run-time.

U-Boot uses that de-facto-standard "distro bootcmd" process to locate a boot script or boot configuration file, and apply the instructions contained therein, in order to load and boot the operating system. See doc/README.distro in the U-Boot source tree for more details of this scheme.