OpenTelemetry Integration into the RDK-E Framework

1. Overview

   1. This document captures architecture, integration flow, macros, build steps, and troubleshooting details for OpenTelemetry integration into RDK-E Thunder plugins.

2. Architecture & Components

   1. **OTLP Logger Library** (`libotlp_logger.so.1`): Wrapper over OTel C++ SDK exporting logs to OTLP gRPC endpoint.

   2. **Thunder Plugins**: LOGINFO/LOGWARN/LOGERR redirected to OTel logger.

   3. **OTel Collector**: Receives OTLP logs and exports to backend.

   4. **Build System**: Yocto + CMake for library and plugins.

   5. **Splunk Collector**: Build steps for Splunk Collector

3. Logging API Migration

   1. PoC code is available here: https://github.com/rdkcentral/meta-rdk-video/pull/2292/

      1. #define LOGINFO(...) do {otlp_logger::log_msgf(opentelemetry::logs::Severity::kInfo, "" __VA_ARGS__);}while(0)

      2. #define LOGWARN(...) do {otlp_logger::log_msgf(opentelemetry::logs::Severity::kWarn, "" __VA_ARGS__);}while(0)

      3. #define LOGERR(...) do {otlp_logger::log_msgf(opentelemetry::logs::Severity::kError, "" __VA_ARGS__);}while(0)

4. OTel Logger Library — High-Level Internal Flow

   1. PoC code is available here: https://github.com/rdkcentral/meta-rdk/pull/423/files

   2. The logger library follows these steps:

      1. Initialize the library (once per program):

         1. Create the OTLP gRPC log exporter (OtlpGrpcLogRecordExporter).

         2. Create a log record processor using the exporter.

         3. Create a LoggerProvider from the processor.

         4. Set the global logger provider so all loggers use this pipeline.

      2. Logging API at runtime:

         1. Retrieve the logger from the global logger provider.

         2. Call EmitLogRecord() with severity + formatted message.

   3. This design ensures minimal overhead in plugins and centralizes all OTLP configuration in one shared library.

5. Build & Link (CMake)

   1. find_package(opentelemetry-cpp CONFIG REQUIRED)

   2. add_library(otlp_logger SHARED otlp_logger.cpp)

   3. target_link_libraries(otlp_logger PRIVATE opentelemetry_exporter_otlp_grpc_logs opentelemetry_logs opentelemetry_common)

   4. target_link_libraries(DisplaySettings PRIVATE otlp_logger)

   5. target_link_libraries(DeviceInfo PRIVATE otlp_logger)

6. Yocto/BitBake Recipes

   1. SOLIBS = ".so"

   2. SOLIBSDEV = ""

   3. FILES:${PN} += "${libdir}/*.so ${libdir}/*.so.*"

   4. RDEPENDS:${PN} += "otlp-logger"

   5. Ensure logger and its dependencies are present in rootfs so plugins can resolve them at runtime.

7. Runtime Configuration

   1. The default configuration for the OTLP endpoint is as below:

   2. OTEL_EXPORTER_OTLP_ENDPOINT default = http://localhost:4317

   3. The default value can be changed with the environment variable as below:

   4. export OTEL_EXPORTER_OTLP_ENDPOINT=grpc://<collector-ip>:4317

   5. LD_LIBRARY_PATH=/opt/otlp_libraries2:/usr/lib

8. Observation

   1. The otlp logger is integrated with the entservices plugins. These plugins send the logs at the time of starting. There are no frequent logs observed after that.

   2. The logs are forwarded to the local splunk collector. The splunk collector was cross compiled to arm7 architecture and it was compressed to 38MB. The uncompressed binary is about 255MB.

   3. When the compressed binary is started, it will be uncompressed first and then run. So it will take up the space of 255MB in RAM

   4. There is no noticeable increase in CPU load, as there is not many logs from the processes.

9. Building Splunk Collector

   1. The below steps are used to build the splunk collector

      1. git clone https://github.com/signalfx/splunk-otel-collector.git

      2. go build -o splunk-otel-collector ./cmd/otelcol # with CGO_ENABLED

      3. CGO_ENABLED=0 go build -o splunk-otel-collector -ldflags="-s -w" ./cmd/otelcol #CGO disabled

      4. sudo snap install upx

      5. sudo apt-get install upx

      6. upx --best --lzma splunk-otel-collector #compression

      7. Attached the Cross-compiled Splunk Collector

   2. In STB:

      1. mkdir /var/lib/otelcol/metrics_disk_queue

      2. mkdir /var/lib/otelcol/logs_disk_queue

      3. ./splunk-otel-collector --config ./otel_collector_config.yaml

      4. Attached the config file

10. Troubleshooting Notes

    1. Missing library: Use readelf -d, objdump -p.

    2. nostd vs std mismatch: Always use std::shared_ptr inside logger.

    3. Use LD_DEBUG=libs to inspect loader decisions

splunk-otel-collector-armotel_collector_config.yaml.