Overview

Downloadable Application Containers (DAC) is a pan-RDK initiative to develop a container solution that allows binary applications to be downloaded and run on STBs without modification. DAC is initially targeting RDK-V set-top boxes but the aim is to extend DAC to suitable RDK-B devices in due course. DAC is a co-operation effort between Comcast, Sky, Liberty Global, Metrological and Consult Red.

DAC builds upon open-source specifications, components and tools from the Open Container Initiative (OCI). In particular, it uses crun, a fast, small, OCI-runtime compliant engine that can configure and launch applications containers. As part of the DAC project, a number of new open-source components have been contributed to RDK including:

• Dobby - an embedded-focused OCI container manager https://github.com/rdkcentral/Dobby
• OCIContainer - a Thunder plugin for interacting with Dobby to start/stop and inspect containers https://github.com/rdkcentral/rdkservices/tree/sprint/2009/OCIContainer
• BundleGen - a command-line tool that runs in the cloud to prepare containers for use on specific platforms https://github.com/rdkcentral/BundleGen

These are the first components of an end-to-end system that operators can use to adapt, manage and control containers to be suitable for their platform.

Other components within the end-to-end system that are currently in development include RDK-V SDK extensions that allow DAC Apps to be: built, published, and then adapted before loading and running on RDK-V STBs. The work for this is co-ordinated by RDK DAC Special interest Group (RDKDACSIG)

Background

What are Containers?

Used heavily in modern cloud software, containers are a standardised way to run the software in a self-contained, isolated and secure environment. Containers contain everything an application needs to run, including code, libraries and other dependencies. This means that there is no need to install specific libraries and dependencies on the host for each application. Unlike more traditional virtual machines, containers are lightweight and fast to create and destroy and don't have the overhead of virtualising an entire operating system. By sharing the OS kernel between containers, running applications inside containers adds very little performance or footprint overhead.

The most popular containerisation solution currently in use is Docker, although there are a number of other solutions such as LXC, Singularity and Podman. LXC containers have been available within RDK for a number of years, using a container generation tool at build time to define and create the container configurations.

Image source: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

Why use containers in RDK?

There are a number of advantages of using containers to run applications in RDK deployments, including:

• Allow developers to easily write applications to run on any RDK devices
• Consistent behaviour across all RDK operators and devices
• Write once, deploy on many devices
• Increase security without impacting performance

As part of the DAC initiative, containers are used to reduce the difficulty of developing native applications that can be run on many different RDK-V devices from different operators by creating an isolated environment for each application. This means the app becomes platform agnostic and can be run on devices the develop may not have physical access to.

Open Container Initiative (OCI)

From the Open Container Initiative (OCI) website (https://opencontainers.org/):

The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Formed in 2015 by Docker and other companies in the container industry and now part of the Linux Foundation, OCI define a number of specifications that allow developers to define containers. These specifications are followed by almost all major containerisation platforms.

OCI define both a runtime specification and an image specification. The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk. The OCI image is used for packaging containers in a platform-agnostic way that can be easily distributed. At a high-level, an OCI implementation would download an OCI Image then unpack that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.

OCI Runtimes

An OCI runtime is a CLI tool that allows for spawning and running containers according to the OCI specification. There are two main OCI runtimes in production use:

Runc

Repo: https://github.com/opencontainers/runc/

Runc is the reference implementation of an OCI runtime and is developed directly by the OCI group. This is the runtime used by Docker, Kubernetes and others. However, being written in Go, it is less suitable for embedded STB environments due to a relatively large footprint.

Crun

Repo: https://github.com/containers/crun

Crun is a different implementation of an OCI runtime, this time written in C and optimised for performance and a low memory footprint. It is developed at RedHat and is currently in use by Podman in Fedora, and is planned to be used in a main RHEL release soon. This is the runtime supported by Dobby and will be used as the default runtime across RDK.

Dobby Overview

Dobby is a container management tool, originally developed by Sky and open-sourced to the RDK community. Dobby is a daemon-based program that makes use of the crun runtime and provides a more user-friendly experience for starting/stopping containers, handles container lifecycle management and enhancing the base functionality provided by crun.

Dobby is designed to be used for starting, stopping and monitoring all containerised applications on an RDK-V device. To start a container with Dobby, provide Dobby with a path to an OCI bundle and a container ID, and Dobby will handle the rest.

Written in C++ and with a plugin-based architecture, Dobby is lightweight, highly expandable and customisable, offering the ability to run custom code at various stages in the container lifecycle to add additional functionality. Out of the box, Dobby provides a number of plugins ready for use to provide commonly needed functionality including:

• Advanced container networking support with NAT and both IPv4 and IPv6 support. Allows for easily adding iptables rules to allow/prevent traffic flow in and out of container
• GPU memory limiting (providing the kernel has the appropriate support)
• Container log management to either files or directly to journald
• Loopback storage mounts to add persistent, isolated storage to containers
• IPC support between containers/host by allowing access to the host dbus inside containers

Plugins are C++ code written against the plugin interface, allowing for operators to easily add additional functionality, or modify existing functionality easily. All the plugins for Dobby can be found in the RDKPlugins directory in the repo here: https://github.com/rdkcentral/Dobby/tree/master/rdkPlugins. Each plugin contains a README file with documentation on its usage. The TestPlugin is designed as a minimal reference plugin that can be used as an example for your own development. Note there is a directory in the Dobby repo called plugins which containers legacy plugins required for some platforms. These should not be used as a reference for new plugins.

Usage

The core component of Dobby is the DobbyDaemon. This should be started at STB boot and then listens over dbus for commands. The full dbus API of Dobby can be found here: https://github.com/rdkcentral/Dobby/blob/master/protocol/include/DobbyProtocol.h, although it is not recommended to communicate with Dobby manually over dbus. Instead, Dobby provides abstractions over dbus which are discussed later in this document.

Dobby ships with a systemd unit file called dobby.service, which allows Dobby to be started/stopped by systemd.

Command-Line

For debug builds of Dobby, the DobbyTool binary is installed on the STB. This is a simple command line app that communicates with Dobby directly over dbus to issue common commands. This is very useful for troubleshooting and testing Dobby.

Integration with Thunder

OCIContainers Plugin

To control Dobby using a JSON-RPC API, then use the OCIContainer Thunder NanoService. This exposes the same functionality as DobbyTool, but in a manner more suited for integration into other code. Example of OCIContainer to start a container from an OCI bundle:

$ curl -X POST http://127.0.0.1:9998/jsonrpc/ -d '{
   "jsonrpc":"2.0",
   "id":3,
   "method":"org.rdk.OCIContainer.1.startContainer",
   "params":{
      "containerId": "testContainer",
      "bundlePath": "[-INSERT BUNDLE PATH-]"
   }
}'

{
   "jsonrpc":"2.0",
   "id":3,
   "result":{
      "descriptor":257,
      "success":true
   }
}

The OCIContainer code  contains a README.md file with detailed documentation on its usage -  see here: https://github.com/rdkcentral/rdkservices/blob/sprint/2009/OCIContainer/README.md

ProcessContainers

Thunder includes the ability to run plugins in containers as well as the traditional in/out of process modes. Thunder offers different back-ends for running Thunder plugins in containers, one of which is Dobby. This mode will communicate with Dobby to start/stop Thunder plugins inside Dobby containers. This provides the advantage of meaning Dobby is responsible for running all types of container on the STB, from native apps to Thunder plugins.

To enable this, compile Thunder with the following CMake flags:

• -DPROCESS_CONTAINERS=ON
• -DPROCESSCONTAINERS_DOBBY=ON

Then for the plugin(s) you wish to run in a container, set the execution mode to "container" in the plugin configuration file.

Creating OCI bundles for Dobby

To run an application as a Dobby container, the application must be packed as an OCI bundle. This bundle can come from many different sources - it could be manually created or generated from an OCI image.

An OCI bundle consists of two key components:

• Root file system - a directory containing the root filesystem of the container
• Config.json - the container configuration

The exact structure of this bundle is defined in the OCI runtime specification here: https://github.com/opencontainers/runtime-spec/blob/master/spec.md. Dobby is fully compliant with version 1.0.2 of the specification.

For more information on how to create OCI bundles from scratch, see the following pages:

• https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
• https://github.com/opencontainers/runc#creating-an-oci-bundle
• https://chromium.googlesource.com/external/github.com/docker/containerd/+/refs/tags/v0.2.0/docs/bundle.md

BundleGen

As part of the DAC project, Consult Red created a standalone tool to accompany Dobby called BundleGen. This is a command line tool designed to convert OCI images into OCI bundles that can be run by Dobby. BundleGen is designed to run off-box in a cloud environment and uses template files to create the bundles tailed for a given RDK platform. The repo includes sample templates for the Dobby Vagrant VM and the RPi.

For more information and documentation on BundleGen, see the repo here: https://github.com/rdkcentral/BundleGen/tree/master/docs

Source Code/Development

Dobby source code lives on the RDKCentral GitHub here: https://github.com/rdkcentral/Dobby and is Apache 2.0 licenced.

To develop Dobby, it is recommended to use Ubuntu 16.04. Dobby ships with a Vagrant file to create a development environment. More information is available here: https://github.com/rdkcentral/Dobby/tree/master/develop

If you would like to contribute code to this project you can do so through GitHub by forking the repository and sending a pull request. Before RDK accepts your code into the project you must sign the RDK Contributor License Agreement (CLA).