Architecture Overview

eyJleHRTcnZJbnRlZ1R5cGUiOiIiLCJnQ2xpZW50SWQiOiIiLCJjcmVhdG9yTmFtZSI6IlJhaHVsIE1laHJhIiwib3V0cHV0VHlwZSI6ImJsb2NrIiwibGFzdE1vZGlmaWVyTmFtZSI6IlJhaHVsIE1laHJhIiwibGFuZ3VhZ2UiOiJlbiIsInVpQ29uZmlnIjoie30iLCJkaWFncmFtRGlzcGxheU5hbWUiOiIiLCJzRmlsZUlkIjoiIiwiYXR0SWQiOiIxMDc4Mzk1MzMiLCJkaWFncmFtTmFtZSI6Ik9DSSBCdW5kbGUgR2VuIFByb2Nlc3NpbmciLCJhc3BlY3QiOiIiLCJsaW5rcyI6ImF1dG8iLCJjZW9OYW1lIjoiT0NJIEJ1bmRsZSBHZW5lcmF0b3IiLCJ0YnN0eWxlIjoidG9wIiwiY2FuQ29tbWVudCI6ZmFsc2UsImRpYWdyYW1VcmwiOiIiLCJjc3ZGaWxlVXJsIjoiIiwiYm9yZGVyIjp0cnVlLCJtYXhTY2FsZSI6IjEiLCJvd25pbmdQYWdlSWQiOjEwNzgzOTUwMiwiZWRpdGFibGUiOmZhbHNlLCJjZW9JZCI6MTA5MDIwMjE1LCJwYWdlSWQiOiIiLCJsYm94Ijp0cnVlLCJzZXJ2ZXJDb25maWciOnsiZW1haWxwcmV2aWV3IjoiMSJ9LCJvZHJpdmVJZCI6IiIsInJldmlzaW9uIjoxLCJtYWNyb0lkIjoiMzUwOWQ5ZTItOTEzOS00NTExLWI0MDktNmIzOTVkMjRlMzkxIiwicHJldmlld05hbWUiOiJPQ0kgQnVuZGxlIEdlbiBQcm9jZXNzaW5nLnBuZyIsImxpY2Vuc2VTdGF0dXMiOiJPSyIsInNlcnZpY2UiOiIiLCJpc1RlbXBsYXRlIjoiIiwid2lkdGgiOiI5MTEiLCJzaW1wbGVWaWV3ZXIiOmZhbHNlLCJsYXN0TW9kaWZpZWQiOjE1OTI5MTc1NTgwMDAsImV4Y2VlZFBhZ2VXaWR0aCI6ZmFsc2UsIm9DbGllbnRJZCI6IiJ9

Requirements & Use Cases

• Convert OCI images generated by the Firebolt SDK into a lightweight, minimal OCI bundle specific to a device that can be run by Dobby on the STB
  • In traditional container solutions (Docker, Podman), this stage is performed by the container runtime itself.
• Parse the OCI Image Manifest/Configuration files to build the final bundle
• Expected to be used both in a cloud environment and on a development machine
  • Unlikely to be used on a STB
• Generated bundles must be able to run on a local machine/STB for use during application development
• Allow creating configuration files per DAC compatible device.
  • This will allow BundleGenerator to generate bundles that will work on a device without having access to that device's hardware/build environment
    

• Allow creating operator configurations to apply business logic to a container image file across all platforms.

• Support hardware variants/revisions of the same platform (e.g. Xi6-A and Xi6-T)

• Process an extended "application capabilities" section to add/remove/change specific sections of the generated bundle according to application requirements

• Prevent creating a bundle for an incompatible device.

  • If the device does not support the required RDK release, then the image cannot be run on that device.

  • However, aim to ensure broad compatibility of applications. The goal of running containerised apps is to allow applications to not be concerned about the platform they are running on

• Allow processing stages to be extended as necessary

Inputs

OCI Image

Description

OCI Image - generated locally or pulled from an OCI Registry

Format

• Directory storing the manifest, layer tarballs and signatures as individual files. Useful for quick debugging of a newly generated image locally (for developer using the SDK)

• OCI Image Layout Path - An image tag in a directory compliant with "Open Container Image Layout Specification" at a given path. See https://github.com/opencontainers/image-spec/blob/master/image-layout.md for more details

  • Allow optionally passing in tag to use a specific tagged image version. Default to latest

Platform Template/Configuration

Description

The template would contain information specific to a single RDK hardware platform/revision. The SDK would ship with configurations for common reference devices such as the Raspberry Pi, Emulator and x86.

Format

Exact format TBD. Potential formats include

• JSON configuration file
• Tarball of device rootfs + config file
• Tarball of specific directories from device + config file
• OCI Image Layer (could then be used as a base layer to build a new image on top of, specific for the device)

Information likely to be required in the template/config would include:

• Device name

• Architecture

• RDK Version (2020Q1, 2020Q2 etc)

  • Include available/installed RDK Services (Thunder NanoServices)

  • Could have multiple configurations for the same device but for different RDK releases if different versions are in use in the field

• GPU Device nodes (and similar - e.g. VPU) that need to be mapped inside the container if graphics is needed

• GPU Libraries

  • Set of tuples containing libraries that must be automatically mounted inside the container for graphics support

  • This allows for adding custom, platform specific graphics libraries into containers (for example libnexus on broadcom platforms)

  • These mounts will override any graphics libraries included with the application

• General Libraries

  • A list of all the libraries available on the device. This will be used when converting the image to bundle to reduce the size of the final output by bind mounting in libraries that are the same major version in the image and on the device

  • This section must be automatically generated for a platform

  • It should be possible for a developer to force the use of a library minor version (e.g. libFoo.so.2.1.5). In this case, unless that exact minor version is available on the STB, the version in the image will be used

    • The mechanism for this should be exposed in the STB publishing tools

• Custom files/paths that must be bind mounted inside the container

• Maximum amount of RAM an application can use (maybe just the amount of RAM available on the platform?)

• Supported resolutions (1080p, 4k)

• Supported Networking options

• Custom environment variables that must be set, specific for the platform

  • XDG_DATA_DIRS

  • XDG_RUNTIME_DIR

• Users/Groups

  • Containerised apps should run as a unique user and group. Define what users/groups a container should run in

  • Allow specifying a range: e.g. AppUser[0-99]:AppGroup

• Non-standard mappings

  • Across RDK, many paths and file names are standardised across builds. If the device uses any non-standard paths or devices, this set of tuples can override the defaults.

Markdown Notes

WIP Notes:

OCI Bundle Generator Requirements.md

Summary

An overview of Bundle gen in context

STB Specific Template (3)

• Where does this come from?
  • today its : /sys, /proc, /tmp, /lib, /bin, /sbin, /usr, /dev
  • PLUS an manually generated optional file list - a list of additional files/devices/groups... i.e. psudeo FS nodes that are available
  • PLUS an manually generated optional MAPPING file that maps any actual node from any of the above lists into the RDK-M namespace. i.e tells you what non standard nodes name are in RDK-M std terms. (e.g. XXx)
• What format is it in?
  • tar of above directories
  • the lists is JSON list
  • the MAPPING is JSON tuple { expected path, actual path}
• Define REference ones for :
  • RasbPi 
  • PC dev

Operator Rules (3)

• What does it have:
  • Meory, cpu, networking, resolutions, RDK services upported, ....
• What format:
  • JSON
• Where:
  • ?
• Who :
  • Operator

OCI Image Spec (3)

Standard OCI image 

plus RDK  extensions to define application requirements 

Overview processing steps  (4)

A summary of the processing steps (re-order if it helps)

1. Get OCI_Image
2. Parse the subset of OCI_Image_config to get the DAC subsection
3. Filter by architecture (remove all libs and bins that aren't for our arch.
4. Lib matching: lib is in template then remove lib and put in bind mount
5. Config App storage. Write the plugin for storage to oci_bundle_config. (AppID is unique across RDK-M.)
6. Write the plugin ram FS to bundle config if needed
7. Add the devices and bindmount to that required to output config (inc. networking)
8. Copy across the layer for architecture independent assets (images, configs, certificates)
9. Copy across the executable compiles to the correct architecture
10. Translate the the remaining parts of OCI_Image_Config to OCI_Bundle_Config

Library matching

Options:

• https://en.wikipedia.org/wiki/Soname
• Full Name matching
• Base name matching, exclude version numbering
• Hash of the lib
  • how to handle to handle compiler option diffs e.g. strip?
• Are LD commands to get version number?

/Dev Matching