Architecture Overview

Requirements & Use Cases

• Convert OCI images generated by the Firebolt SDK into a lightweight, minimal OCI bundle specific to a device that can be run by Dobby on the STB
  • In traditional container solutions (Docker, Podman), this stage is performed by the container runtime itself.
• Parse the OCI Image Manifest/Configuration files to build the final bundle
• Expected to be used both in a cloud environment and on a development machine
  • Unlikely to be used on a STB
• Generated bundles must be able to run on a local machine/STB for use during application development
• Allow creating configuration files per DAC compatible device.
  • This will allow BundleGenerator to generate bundles that will work on a device without having access to that device's hardware/build environment
    

• Allow creating operator configurations to apply business logic to a container image file across all platforms.

• Support hardware variants/revisions of the same platform (e.g. Xi6-A and Xi6-T)

• Process an extended "application capabilities" section to add/remove/change specific sections of the generated bundle according to application requirements

• Prevent creating a bundle for an incompatible device.

  • If the device does not support the required RDK release, then the image cannot be run on that device.

  • However, aim to ensure broad compatibility of applications. The goal of running containerised apps is to allow applications to not be concerned about the platform they are running on

• Allow processing stages to be extended as necessary

• Output should be deterministic - for a given set of inputs, the output should always be identical (ignoring timestamps)
• Should generate the bundle in under 30 seconds to allow on-demand generation if a STB requests a bundle that is not in an operator's cache

Inputs

OCI Image

Description

OCI Image - generated locally or pulled from an OCI Registry

Format

• Directory storing the manifest, layer tarballs and signatures as individual files. Useful for quick debugging of a newly generated image locally (for developer using the SDK)

• OCI Image Layout Path - An image tag in a directory compliant with "Open Container Image Layout Specification" at a given path. See https://github.com/opencontainers/image-spec/blob/master/image-layout.md for more details

  • Allow optionally passing in tag to use a specific tagged image version. Default to latest

Platform Template/Configuration

Description

The template would contain information specific to a single RDK hardware platform/revision. The SDK would ship with configurations for common reference devices such as the Raspberry Pi, Emulator and x86.

Some of this configuration will be generated by hand per device, although this can be based off templates provided for reference devices. Parts of the configuration should be generated automatically - perhaps as a Bitbake stage during the build

Format

Exact format TBD. Potential formats include

• JSON configuration file
• Tarball of device rootfs + config file
• Tarball of specific directories from device + config file
• OCI Image Layer (could then be used as a base layer to build a new image on top of, specific for the device)

Information likely to be required in the template/config would include:

• Device name

• Architecture

• RDK Version (2020Q1, 2020Q2 etc)

  • Include available/installed RDK Services (Thunder NanoServices)

  • Could have multiple configurations for the same device but for different RDK releases if different versions are in use in the field

• GPU Device nodes (and similar - e.g. VPU) that need to be mapped inside the container if graphics is needed

• GPU Libraries

  • Set of tuples containing libraries that must be automatically mounted inside the container for graphics support

  • This allows for adding custom, platform specific graphics libraries into containers (for example libnexus on broadcom platforms)

  • These mounts will override any graphics libraries included with the application

• General Libraries

  • A list of all the libraries available on the device. This will be used when converting the image to bundle to reduce the size of the final output by bind mounting in libraries that are the same major version in the image and on the device

  • This section must be automatically generated for a platform

  • It should be possible for a developer to force the use of a library minor version (e.g. libFoo.so.2.1.5). In this case, unless that exact minor version is available on the STB, the version in the image will be used

    • The mechanism for this should be exposed in the STB publishing tools

• Custom files/paths that must be bind mounted inside the container

• Maximum amount of RAM an application can use (maybe just the amount of RAM available on the platform?)

• Supported resolutions (1080p, 4k)

• Supported Networking options

• Custom environment variables that must be set, specific for the platform

  • XDG_DATA_DIRS

  • XDG_RUNTIME_DIR

• Users/Groups

  • Containerised apps should run as a unique user and group. Define what users/groups a container should run in

  • Allow specifying a range: e.g. AppUser[0-99]:AppGroup

• Non-standard mappings

  • Across RDK, many paths and file names are standardised across builds. If the device uses any non-standard paths or devices, this set of tuples can override the defaults.

Operator Configuration

Description

The operator configuration could be used to do one or more of the following:

• Applying configuration options/settings to all operator devices
• Applying configuration options/settings to a family of devices. I.e. Xi6 would have a single "main" configuration, and each hardware/software variant could have a much smaller platform specific config

Format

TBC. Likely JSON or tarballs, similar to the platform configuration

Output

OCI Bundle

Description

The main output of BundleGen should be an extended OCI bundle (aka Bundle*). This will contain the container's config.json, with an additional section containing the rdkPlugins section to provide additional functionality required by the application - normally specified in the application capabilities

This bundle would be the smallest possible compatible bundle, containing only libraries that are not present on the STB. Any libraries that can be provided by RDK on the STB should be bind-mounted inside the container. This reduces the size of the bundle, necessary to decrease download time and increase the amount of containers that can be stored on the limited flash on an STB. This differs from the current approach in Dobby where the /lib and /usr/lib directory are mounted into the container in their entirety and the application is expected to use only the libraries on the STB - unable to provide its own.

This output format would be used to allow downloading the OCI Bundle* to a STB for execution directly by Dobby. Distribution could take place over any operator specific mechanism - FTP, HTTPS etc.

Format

• Directory

• Tarball of above directory

Processing

Overview (Flowchart)

Description

0. Obtain the OCI Image

Whilst out of the main scope of BundleGen, we must first obtain an OCI image for processing.

One potential candidate tool for retrieving images is skopeo: https://github.com/containers/skopeo

From the README of skopeo:

skopeo is a command line utility that performs various operations on container images and image repositories.

skopeo does not require the user to be running as root to do most of its operations.

skopeo does not require a daemon to be running to perform its operations.

skopeo can work with OCI images as well as the original Docker v2 images.

Skopeo works with API V2 container image registries such as docker.io and quay.io registries, private registries, local directories and local OCI-layout directories. Skopeo can perform operations which consist of:

• Copying an image from and to various storage mechanisms. For example you can copy images from one registry to another, without requiring privilege.

• Inspecting a remote image showing its properties including its layers, without requiring you to pull the image to the host.

• Deleting an image from an image repository.

• When required by the repository, skopeo can pass the appropriate credentials and certificates for authentication.

Skopeo also lets you just pull down the specific architecture variant/OS variant of an image. For example, the hello-world image is available in different architectures and for Windows or Linux (described here https://github.com/docker-library/repo-info/blob/master/repos/hello-world/remote/latest.md). Here, Skopeo pulls the armv8 variant and outputs the image configuration.

[vagrant@localhost ~]$ skopeo --override-arch arm --override-variant v8 inspect docker://hello-world
{
    "Name": "docker.io/library/hello-world",
    "Digest": "sha256:d58e752213a51785838f9eed2b7a498ffa1cb3aa7f946dda11af39286c3db9a9",
    "RepoTags": [
        "latest",
        "linux",
        "nanoserver-1709",
        "nanoserver-1803",
        "nanoserver-1809",
        "nanoserver-sac2016",
        "nanoserver",
        "nanoserver1709"
    ],
    "Created": "2020-01-03T01:02:41.624221437Z",
    "DockerVersion": "18.06.1-ce",
    "Labels": null,
    "Architecture": "arm",
    "Os": "linux",
    "Layers": [
        "sha256:4ee5c797bcd78105083ec883d8e6e8bc58124c65b30444d0110073d603e61190"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

1. Unpack the OCI image

The first stage of processing should be to unpack to the OCI image into an OCI bundle (rootfs + config.json) that we can manipulate to form our final bundle to be delivered to the STB. This is assuming the image provides a variant with a suitable architecture for the target platform.

Existing Tooling

Potential tool: umoci: https://umo.ci/quick-start/workflow/

Umoci is an official opencontainers (formally OpenSUSE) tool that can manipulate downloaded OCI images and convert them to OCI bundles. Written in Go unfortunately, so would not be suitable for running on an STB, but should be fine for use in a cloud environment. 

Umoci is also designed to be a secure mechanism for unpacking images - see here for more info on umoci's security: https://github.com/opencontainers/umoci/blob/master/doc/site/reference/security.md

Example:

% skopeo copy docker://opensuse/amd64:42.2 oci:opensuse:42.2 # Obtain the image from the docker hub using Skopeo
% sudo umoci unpack --image opensuse:42.2 bundle
% ls -l bundle
total 720
-rw-r--r-- 1 root root   3247 Jul  3 17:58 config.json
drwxr-xr-x 1 root root    128 Jan  1  1970 rootfs
-rw-r--r-- 1 root root 725320 Jul  3 17:58 sha256_8eac95fae2d9d0144607ffde0248b2eb46556318dcce7a9e4cc92edcd2100b67.mtree
-rw-r--r-- 1 root root    270 Jul  3 17:58 umoci.json

Here the opensuse image with tag 42.2 has been unpacked to produce a rootfs directory and config.json. As per the documentation:

SYNOPSIS

umoci unpack --image=image[:tag] [--rootless] [--uid-map=value] [--uid-map=value] [--keep-dirlinks] bundle

DESCRIPTION

Extracts all of the layers (deterministically) to an OCI runtime bundle at the path bundle, as well as generating an OCI runtime configuration that corresponds to the image's configuration. In addition, an mtree(8) specification is generated at the time of unpacking to allow filesystem deltas to be generated by umoci-repack(1) and thus allowing for the creation of layered OCI images.

Note that umoci can also be used to repack the modified rootfs back into the original image as a delta layer, or into a completely new image, potentially useful for a minimal OCI image output format. Changes to the config.json are not reflected in a repacked image and must be modified with the umoci config command

umoci doesn't currently support multi-arch images, and doesn't expose a way through the CLI to select which arch to unpack. See:

https://github.com/opencontainers/umoci/issues/10

https://github.com/opencontainers/umoci/issues/313

However, if used with Skopeo, the architecture filtering could be done at that stage, likely negating this issue

There also exists a repo called oci-image-tools, which are also written in Go and perform a similar task. However, this tool is now unmaintained and umoci is recommended as the replacement...:

This project is no longer actively maintained. However, umoci is a much more full-featured tool for manipulating OCI images, and is now an OCI project as a reference implementation of the OCI image-spec. I would strongly suggest people move to using umoci.

https://github.com/opencontainers/image-tools/issues/222

1a. Custom Conversion from Image Config → Config.json

If using existing OCI tools such as umoci, they will do the job of converting between the OCI Image Config and the Runtime Config. However, if there's any custom conversion/mapping to be done, this will need to be performed manually.

See here: Runtime configuration for more info on the mapping between the runtime/image configuration.

2. Validate App is compatible

This should be a first-pass, fail-fast test to ensure that the app can actually be packaged and run on the target device. Check that the specified app RDK version is suitable with the RDK versioned defined in the platform/operator configuration. Check that the platform can supply the capabilities required by the app - such as RAM, storage space and resolutions.

If the app is incompatible, return an error.

3. Library Matching (Potentially skip for initial PoC version if too complex)

Run an ldd on the binary (using a suitable version of ldd for the arch), to work out it's dependencies. Then, based on the information in the platform configuration, determine if a suitable library is available on the STB.

If the STB provides a compatible library - deemed to be one of the same MAJOR version (assuming semver), then edit the config.json in the bundle to add the new bind mount into the container. Delete the library from the bundle rootfs. If the STB does not provide a suitable library, then the library should remain in the rootfs. This step should then remove any libraries that are not needed by the application.

Here be dragons: If the app uses dlopen to manually load dynamic libraries, we somehow need to make sure those aren't accidentally removed at this stage.

Failure here should go back to the safe option of just using the bundled libraries for the app - since that means the app will work, but final size will be larger.

There should also be the option for a developer to lock their application to a specific minor version of a library, or define a minimum minor version if they need to have an exact revision of a library.

Libraries could potentially be matched on:

• Major version (soname)
• Minor version
• SHA hash?

3a. Add GPU Lib Mounts

Based on the libraries specified in the "GPU Libraries" section of the platform config, add bind mounts for the necessary graphics libraries into the container.

Failure here should be fatal.

4. Add Extra Bind Mounts

If any other bind mounts are needed, for example for OCDM or similar, add them now.

5. Add Devices

Any devices set in the platform configuration should now be added to the OCI config in the bind mounts section and the devices section. Note as per the specification, crun will automatically provide the container with:

• /dev/null

• /dev/zero

• /dev/full

• /dev/random

• /dev/urandom

• /dev/tty

• /dev/console

• /dev/ptmx. A bind-mount or symlink of the container's /dev/pts/ptmx.

When starting from a Dobby spec, Dobby also only allows devices in a global whitelist to prevent a user adding a device they shouldn't such as CDI/system device nodes

7. Add Environment Vars

Any environment variables in the platform/operator config should be added to the runtime config now

7. Add Plugins (from application capabilities)

Based on the capabilities required by the application and provided by the platform, generate and write the rdkPlugins section of the runtime config if necessary

8. Write Anything Else

Any other sections of the runtime config that need to be modified should be done now.

9. Validate

Validate the final configuration/bundle to make sure it's sane

10. Generate Bundle

Create a final tarball of the generated OCI bundle fordistribution

Notes/WIP

WIP Notes:

OCI Bundle Generator Requirements.md

A summary of the processing steps (re-order if it helps)

1. Get OCI_Image
2. Parse the subset of OCI_Image_config to get the DAC subsection
3. Filter by architecture (remove all libs and bins that aren't for our arch.
4. Lib matching: lib is in template then remove lib and put in bind mount
5. Config App storage. Write the plugin for storage to oci_bundle_config. (AppID is unique across RDK-M.)
6. Write the plugin ram FS to bundle config if needed
7. Add the devices and bindmount to that required to output config (inc. networking)
8. Copy across the layer for architecture independent assets (images, configs, certificates)
9. Copy across the executable compiles to the correct architecture
10. Translate the the remaining parts of OCI_Image_Config to OCI_Bundle_Config