RDK-C : RDK Integrating libcamera into PipeWire

Why libcamera in PipeWire?

• Standardized interface: Libcamera provides a unified and standardized interface for interacting with different types of camera devices. By integrating libcamera into pipewire, camera applications can use a single API to access and control camera hardware.
• Flexibility and modularity: Pipewire is designed to be a modular multimedia framework. 
• Improved performance: Libcamera is designed to provide better performance and lower latency than previous camera APIs for Linux-based systems.
• Cross-platform compatibility: Both libcamera and pipewire are open-source projects that can run on a variety of Linux-based platforms.
• Integrating libcamera into pipewire helps to simplify and streamline the process of working with camera devices on Linux-based systems.

Architecture PipeWire-libcamera

Integration of RDKC with PipeWire-libcamera

Below list of changes set for PipeWire-libcamera

Libcamera Recipes:
https://code.rdkcentral.com/r/plugins/gitiles/rdk/components/generic/rdk-oe/meta-cmf-raspberrypi/+/refs/heads/rdk-next/recipes-multimedia/libcamera/
https://code.rdkcentral.com/r/plugins/gitiles/rdk/components/opensource/oe/meta-openembedded/+/refs/heads/dunfell/meta-multimedia/recipes-multimedia/libcamera/

Enabling Libcamera SPA Plugin Pipewire:
https://code.rdkcentral.com/r/c/components/generic/rdk-oe/meta-cmf-camera/+/81195
https://code.rdkcentral.com/r/c/rdk/components/generic/rdk-oe/meta-cmf-raspberrypi/+/81171

PipeWire Overview and Flow in RDK-C

• It can create audio and video server
• Apps and devices are represented by node
• Each node has the input and output port to share media between node
• The session manager is to configure nodes, ports and linking nodes
• SPA: It will link to the driver to control the driver parameter and send and receive media.
• SPA plugins: libcamera, ALSA, v4l2, JACK and etc...
• WebRTC Live Streaming- Can View from Web Application
• RMS live streaming with RTSP protocol – Can View live streaming in VLC
• CVR with KVS SDK – Can view/record data in AWS KVS service
• Session manager will control these app nodes and ports and it will link nodes along with ports for communication
• Used libcamera SPA to fetch data from the v4l2 driver

Overview of PipeWire-Libcamera Internals

SPA Plugin Creation (pipewire/spa/plugins)

1. Implemented each SPA plugins with a group of factories.
2. Each factory should have the below details
       * Version ( SPA_VERSION_HANDLE_FACTORY )
       * Factory Name ( SPA_NAME_API_* - eg.SPA_NAME_API_LIBCAMERA_ENUM_UDEV)
       * SPA dictionary information ( struct spa_dict )
       * Get size callback  - impl_get_size()
           - Each factory has its own impl structure for to maintain some specific details.
           - With this callback, we can get the size of this factory impl structure
       * Initialization callback – impl_init()
           - This will initialize this factory-specific activity
       * Enum interface info callback - impl_enum_interface_info()
3. Each SPA plugin's factories are associated with the spa_handle_factory_enum() function.
4. Libcamera SPA plugin’s factories(libcamera.c)

Libcamera API sequence

PipeWire API sequence in RDK-C:

Supported Usecases using Pipewire-Libcamera:

1. WebRTC Streaming
2. RTSP Streaming
3. Continuous Video Recording

Libcamera Roadmap

Features planned down the lane: 

• Configuration and control algorithms using libcamera in Raspberry Pi Platform with advanced camera module.
  • 3A algorithm
  • AEC/AGC control                                                                                                                   
  • AWB control
  • AF control

                                                                                                                                                                                Sony IMX477

                                                                                                                                                                               Zooming Camera.