Overview

Downloadable Application Containers (DAC) is a pan-RDK initiative to develop a container solution that allows binary applications to be downloaded and run on STBs without modification. DAC is initially targeting RDK-V set-top boxes but the aim is to extend DAC to suitable RDK-B devices in due course. DAC is a co-operation effort between Comcast, Sky, Liberty Global, Metrological and Consult Red.

DAC builds upon open-source specifications, components and tools from the Open Container Initiative (OCI). In particular, it uses crun, a fast, small, OCI-runtime compliant engine that can configure and launch applications containers. The first RDK contributions to be released are Dobby  and OCIContainers. These are the first two components of an end-to-end system that operators can use to adapt, manage and control containers to be suitable for their platform. Other components within the end-to-end system that are currently in development include RDK-V SDK extensions that allow DAC Apps to be: built, published, and then adapted before loading and running on RDK-V STBs. The work for this is co-ordinated by RDK DAC Special interest Group (RDKDACSIG)

Background

What are Containers?

Used heavily in modern cloud software, containers are a standardised way to run the software in a self-contained, isolated and secure environment. Containers contain everything an application needs to run, including code, libraries and other dependencies. This means that there is no need to install specific libraries and dependencies on the host for each application. Unlike more traditional virtual machines, containers are lightweight and fast to create and destroy and don't have the overhead of virtualising an entire operating system. By sharing the OS kernel between containers, running applications inside containers adds very little performance or footprint overhead.

The most popular containerisation solution currently in use is Docker, although there are a number of other solutions such as LXC, Singularity and Podman. LXC containers have been available within RDK for a number of years, using a container generation tool at build time to define and create the container configurations.

Image source: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

Why use containers in RDK?

There are a number of advantages of using containers to run applications in RDK deployments, including:

• Allow developers to easily write applications to run on any RDK devices
• Consistent behaviour across all RDK operators and devices
• Write once, deploy on many devices
• Increase security without impacting performance

Open Container Initiative (OCI)

From the Open Container Initiative (OCI) website (https://opencontainers.org/):

The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Formed in 2015 by Docker and other companies in the container industry and now part of the Linux Foundation, OCI define a number of specifications that allow developers to define containers. These specifications are followed by almost all major containerisation platforms.

OCI define both a runtime specification and an image specification. The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk. The OCI image is used for packaging containers in a platform-agnostic way that can be easily distributed. At a high-level, an OCI implementation would download an OCI Image then unpack that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.

OCI Runtimes

An OCI runtime is a CLI tool that allows for spawning and running containers according to the OCI specification. There are two main OCI runtimes in production use:

Runc

Repo: https://github.com/opencontainers/runc/

Runc is the reference implementation of an OCI runtime and is developed directly by the OCI group. This is the runtime used by Docker, Kubernetes and others. However, being written in Go, it is less suitable for embedded STB environments due to a relatively large footprint.

Crun

Repo: https://github.com/containers/crun

Crun is a different implementation of an OCI runtime, this time written in C and optimised for performance and a low memory footprint. It is developed at RedHat and is currently in use by Podman in Fedora, and is planned to be used in a main RHEL release soon. This is the runtime supported by Dobby and will be used in RDK.

Dobby Overview

Dobby is a container management tool, originally developed by Sky and open-sourced to the RDK community. Dobby is a daemon-based program that sits above the crun runtime, providing a more user-friendly experience and enhancing the base functionality provided by crun.

Dobby is designed to be used for starting, stopping and monitoring all containerised applications on the STB. To start a container with Dobby, simply provide Dobby with a path to an OCI bundle and a container ID, and Dobby will handle the rest.

Written in C++ and with a plugin-based architecture, Dobby is highly expandable and customisable, offering the ability to run custom code at various stages in the container lifecycle to add additional functionality. Out of the box, Dobby provides a number of plugins ready for use to provide commonly needed functionality including:

• Advanced container networking support with NAT and both IPv4 and IPv6 support
• Container log management to either files or directly to journald
• Loopback storage mounts to add persistent, isolated storage to containers
• Allowing access to the host dbus inside containers

Plugins are simply C++ code written against the plugin interface, allowing for operators to easily add additional functionality, or modify existing functionality easily.

Usage

The core component of Dobby is the DobbyDaemon. This should be started at STB boot and then listens over dbus for commands. Dobby ships with a systemd unit file called sky-dobby.service, which allows Dobby to be started/stopped by systemd. DobbyDaemon logs are sent to journald.

Once the daemon is running, there are a number of ways to interact with Dobby

Command-Line

For debug builds of Dobby, the DobbyTool is installed on the STB. This is a simple command line app that communicates with Dobby directly over dbus to issue common commands. DobbyTool offers the following commands:

vagrant@dobby-vagrant:~/srcDobby/build$ DobbyTool help
quit              quit
help              help [command]
shutdown          shutdown
start             start [options...] <id> <specfile/bundlepath> [command]
stop              stop <id> [options...]
pause             pause <id>
resume            resume <id>
exec              exec [options...] <id> <command>
list              list
info              info <id>
dumpspec          dumpspec <id> [options...]
bundle            bundle <id> <specfile> [options...]
set-dbus          set-dbus <private>|<public> <address>

Some commands offered by DobbyTool (mainly dumpspec) are to allow compatibility with the legacy Dobby Spec input format (instead of OCI bundles), which are deprecated and maintained only for backwards compatibility with existing deployments.

Thunder

To control Dobby using a JSON-RPC API, then use the OCIContainer Thunder NanoService. The source code for OCIContainer can be found here TODO:: UPDATE THIS ONCE OCICONTAINER IS MERGED: https://github.com/rdkcentral/rdkservices/pull/116.

OCIContainer code contains a README.md file with detailed documentation on its usage, and includes a test script (test/ociContainerTest.sh) that can be used to ensure Dobby/OCIContainer is working as expected.

Creating OCI bundles for Dobby

To run an application as a Dobby container, the application must be packed as an OCI bundle. This bundle can come from many different sources - it could be manually created or generated from an OCI image.

For demonstration purposes, a sample bundle is attached to this page that creates a simple container. This container runs the sleep command to sleep for 30 seconds. It also configured Dobby's logging to log the container stdout/err to a file in /tmp/container.log

TODO:: Attach sleepy container bundle

For more information on how to create OCI bundles from scratch, see the following pages:

• https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
• https://github.com/opencontainers/runc#creating-an-oci-bundle
• https://chromium.googlesource.com/external/github.com/docker/containerd/+/refs/tags/v0.2.0/docs/bundle.md

Source Code/Development

Dobby source code lives on the RDKCentral GitHub here: https://github.com/rdkcentral/Dobby and is Apache 2.0 licenced.

To develop Dobby, it is recommended to use Ubuntu 16.04. You cannot develop/test Dobby inside a Docker container as you cannot run Dobby containers inside a Docker container. TODO:: ADD LINK TO VAGRANT VM IF/WHEN WE MAKE IT PUBLIC

If you would like to contribute code to this project you can do so through GitHub by forking the repository and sending a pull request. Before RDK accepts your code into the project you must sign the RDK Contributor License Agreement (CLA).