Key Info

BundleGen is a command-line tool to convert OCI Images into extended OCI Bundles for use with the Dobby container manager. 

• Repository: https://github.com/rdkcentral/BundleGen
• Language: Python
• License: Apache 2.0

Background - Container Images vs Bundles

In OCI, there are two distinct ways of packing applications for containers - Images and Bundles.

Images

At a high-level an OCI implementation would download an OCI Image then convert that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.

An Image is a platform-agnostic distribution format that is designed to store the application and any dependencies it may require such as libraries and configuration files. The specification for images is defined here: https://github.com/opencontainers/image-spec

Iimages contain the following items:

• a manifest

  • The manifest contains metadata about the contents and dependencies of the image - for example md5 hashes and sizes of the image files.

• an image index (optional)

  • A higher-level document which points to specific image manifests. This is ideal for supporting multiple platforms, as each platform would have their own manifest.

• a set of filesystem layers

  • The final container filesystem is built up of layers where one or more layers are applied on top of each other to create a complete filesystem. Layers can be .tar, .tar.gz or .zstd files.

• and a configuration.

  • Each image must have a config file. Note this is different to the runtime spec config.json file.

  • The image config contains basic image info (such as author, creation date) and  information about how the container should be launched, such as the entrypoint, environment variables, networking and volumes.

  • The config in the image is immutable (since the image ID is based on a hash of this config) and changing it results in a derived image instead of changing the existing image

These are all contained in a single archive that can be distributed and downloaded by anyone. By using the image index, a single image can contain multiple variants of an application each compiled for a different architecture (i.e. a single image could contain versions of an app built for x86 and arm). The image filesystem is built up of multiple layers where each image layer represents changes between itself and the parent layer. To construct the final container filesystem, each layer is unpacked on top of the lower layers. Images are constructed this way since whenever an image builder creates a new image, the differences are saved as a layer. This helps improves performance when building images by only needing to re-build what's changed instead of the whole image again.

Once built, an OCI Image can then be uploaded to a registry server and discovered by name, downloaded, verified by hash, trusted through a signature, and converted into the runtime bundle ready to run by the runtime.

Bundles

Once the image is unpacked and converted into a bundle, this bundle can be used to launch a container using an OCI compliant runtime. The bundle consists of two parts:

• config.json - this is the runtime config and contains the fields that enable the creation of a container such as arguments, environment variables and mount points
• rootfs - the root file system of the container.

These files must be in a single directory on the local filesystem. In other words, a tar archive of a bundle will have these artefacts at the root of the archive, not nested within a top-level directory. The specification for the runtime configuration and the bundle format are defined here: https://github.com/opencontainers/runtime-spec. Dobby is fully compliant with the latest version 1.0.2 of the specification.

The bundle contains an OCI configuration file (which is named config.json) which can specify host-independent details such as which executable to launch and host-specific settings such as mount locations, hook paths, Linux namespaces and cgroups. Because the configuration includes host-specific settings, application bundle directories copied between two hosts may require configuration adjustments.

Creating OCI Bundles Manually

It is possible to manually create an OCI bundle from scratch by hand-crafting the runtime configuration and copying files into the rootfs. For more information on how to create OCI bundles from scratch, see the following pages:

• https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
• https://github.com/opencontainers/runc#creating-an-oci-bundle
• https://chromium.googlesource.com/external/github.com/docker/containerd/+/refs/tags/v0.2.0/docs/bundle.md

The Need for BundleGen

As part of the DAC project, Consult Red created a standalone tool to accompany Dobby called BundleGen.

This is a command line tool designed to convert OCI images into OCI bundles that can be run by Dobby.

Since OCI bundles are highly specific to the platform/hardware they are being run on, traditionally the OCI bundles are generated on the device itself. However, this requires downloading the entire OCI image to the device and unpacking it before generating the bundle, which can take some time and it not suitable for embedded devices with limited CPU and flash memory. BundleGen is designed to run off-box in a cloud environment and uses template files to create the bundles tailed for a given RDK platform. These templates tell BundleGen how to generate the bundle for the platform - such as which graphics libraries are needed. The repo includes sample templates for the Dobby Vagrant VM and the RPi.

Roughly, BundleGen performs the following steps:

1. Obtain the OCI Image using Skopeo (open-source tool by RedHat)

2. Unpack the OCI image with Umoci (open-source tool by OCI)

3. Validate App is compatible

4. Modify the OCI config based on the platform template. This includes but is not limited to:

   1. Environment Variables

   2. Mounts for sockets, libraries

   3. Device whitelisting

   4. User and groups

   5. RDK plugins (networking, loopback mounts, logging)

5. Create .tar.gz from bundle with modified config

App Metadata

To create a bundle that is compatible for an application, BundleGen requires a small amount of metadata about the application. This can be thought of as similar to a manifest in Android apps. The app metadata can include information about storage and memory requirements, whether the app needs graphics support and network access such as port forwarding.

The app metadata is a JSON file, an example of which is below:

{
    "id": "com.rdk.wayland-egl-test",
    "type": "application/vnd.rdk-app.dac.native",
    "version": "1.0.0",
    "description": "Simple wayland egl demo, showing green rectangle",
    "graphics": true,
    "network": {
        "type": "open"
    },
    "storage": {},
    "resources": {
        "ram": "128M"
    },
    "features": [],
    "mounts": []
}

The full syntax can be found here: https://github.com/rdkcentral/BundleGen/blob/master/docs/AppMetadata.md

The example images built by the SDK contain metadata files which are the embedded inside the OCI image for BundleGen to use: https://github.com/stagingrdkm/meta-dac-sdk/tree/master/recipes-example/images/metadatas

Non SDK Generated Images

If using images from the Docker hub or other OCI registry that were not generated by the DAC SDK, then you will need to manually write a metadata file for the app and provide the file to BundleGen in the generate command via the optional --appmetadata argument.

Usage

Before you can use BundleGen, you must create an OCI image containing your application using the SDK and upload it to an OCI registry such as the Docker Hub. Once that is complete, follow the below instructions to use BundleGen.

Start by launching the Fedora-based VM provided in the BundleGen repo here: https://github.com/rdkcentral/BundleGen/blob/master/Vagrantfile. If you are not sure how to work with Vagrant, see below:

Once the VM is running, perform the following steps

1. Clone bundle generator repository: git clone https://github.com/rdkcentral/BundleGen.git
2. Go to bundlegen directory: cd BundleGen
3. (Optional) Create virtual environment: python3 -m venv .venv
4. (Optional) Enter virtual environment: source .venv/bin/activate
5. Install bundlegen requirements: pip install -r requirements.txt
6. Install bundlegen: pip install --editable .

Now you can create bundle from images generated by the SDK by running the generate commad: bundlegen generate --platform <platform-name> <img-url> <output-dir>. Example:

$ bundlegen generate --platform rpi3 docker://my-app ~/outputdirectory

Note the image URL should be in the form `docker://image-url`. If the image is on the Docker Hub, the full URL can be omitted - e.g `docker://hello-world`.

For more information and documentation on BundleGen, see the repo's readme file here: https://github.com/rdkcentral/BundleGen/blob/master/README.md and the documentation directory: https://github.com/rdkcentral/BundleGen/tree/master/docs