Harvester

Harvester is an RDK-B telemetry component responsible for collecting and reporting WiFi metrics and statistics from the gateway device to cloud management systems. The component gathers data about associated WiFi devices, neighboring access points, and radio interface statistics at configurable intervals and transmits these metrics using Avro serialization format through the WebPA/Parodus framework for upstream analytics and monitoring.

The component operates as a scheduled data collection agent that periodically polls WiFi HAL APIs to retrieve device association information, radio traffic metrics, and neighboring AP scan results. Harvester packages this telemetry data into structured Avro-encoded messages and transmits them to cloud-based data processing systems through the CcspWebPA messaging infrastructure. The component supports configurable polling and reporting intervals for each telemetry report type with persistent storage of configuration parameters through CcspPsm integration.

Harvester implements TR-181 data model parameters under the Device.X_RDKCENTRAL-COM_Report namespace enabling dynamic configuration and control of telemetry collection behavior. The architecture supports both traditional WiFi HAL-based data collection and R-BUS-based data retrieval when OneWifi feature is enabled providing flexibility across different RDK-B platform configurations.

 Key Features & Responsibilities:

• Interface Devices WiFi Reporting: Collects and reports detailed information about WiFi devices associated with each SSID including MAC addresses, signal strength, connection timestamps, operating mode, frequency band, and data rate information at configurable polling intervals
• Radio Interface Statistics Collection: Gathers radio-level performance metrics including channel utilization, noise floor, transmission statistics, retry counts, and error rates for each WiFi radio interface supporting 2.4GHz and 5GHz bands
• Neighboring Access Point Scanning: Performs periodic WiFi channel scans to detect and report neighboring access points including SSID, BSSID, channel, signal strength, security mode, and operating standards for spectrum analysis and channel optimization
• Avro Serialization and Encoding: Packages collected telemetry data using Apache Avro binary serialization format with schema validation ensuring compact data representation and efficient transmission to cloud analytics platforms
• WebPA Integration and Upstream Messaging: Establishes persistent connection with Parodus daemon through a dedicated handler thread that continuously monitors connectivity status and performs connection initialization using libparodus APIs over nanomsg IPC transport. When reporting intervals expire, collection threads invoke message transmission by constructing WRP event message structures containing Avro-encoded telemetry payloads with message type, source MAC address, destination routing, and content-type metadata. The transmission function calls libparodus_send with mutex synchronization to deliver messages to Parodus daemon which forwards them to cloud management systems. When transmission to Parodus fails, the component implements synchronous retry logic with exponential backoff delay calculated as 2^n - 1 seconds, attempting up to 5 retries before abandoning the message. The Parodus daemon handles subsequent message queuing and cloud delivery while Harvester confirms only that Parodus successfully accepted the outbound telemetry report

Design

Harvester follows an event-driven, multi-threaded architecture organized around independent data collection threads for each telemetry report type. The design separates data collection responsibilities into distinct modules corresponding to Interface Devices WiFi (IDW), Radio Interface Statistics (RIS), and Neighboring AP (NAP) reporting each operating with independent polling cycles and reporting intervals configured through TR-181 parameters.

The component implements a producer-consumer pattern where dedicated polling threads periodically invoke WiFi HAL APIs or R-BUS queries to retrieve metrics and package the collected data into Avro-encoded binary payloads. These telemetry messages are transmitted to Parodus daemon through libparodus IPC interface with Harvester implementing retry logic and exponential backoff to ensure Parodus accepts the messages. Parodus is subsequently responsible for reliable cloud delivery including message queuing and cloud-level connection management.

Configuration parameters for polling periods, reporting intervals, and harvesting enable flags are persisted in CcspPsm ensuring configuration survives system reboots and component restarts. The TR-181 data model implementation provides standardized northbound interfaces for external components and management systems to configure harvester behavior dynamically. The component supports a default configuration mechanism with override TTL support allowing temporary parameter changes that automatically revert to defaults after expiration.

When OneWifi feature is enabled, Harvester utilizes R-BUS APIs for data collection providing an abstraction layer over direct WiFi HAL access and enabling better integration with unified WiFi management architectures. The dual-mode operation supports both traditional HAL-based collection and R-BUS-based queries determined at build time through conditional compilation.

 Prerequisites and Dependencies

Build-Time Flags and Configuration:

RDK-B Platform and Integration Requirements:

• Build Dependencies: ccsp-common-library, rdk-logger, avro-c, trower-base64, hal-wifi, msgpackc, dbus, utopia, wrp-c, nanomsg, libparodus
• RDK-B Components: CcspPsm, CcspWiFiAgent/OneWifi, CcspWebPA
• HAL Dependencies: WiFi HAL APIs for device association queries, radio statistics, and neighboring AP scans
• Systemd Services: CcspPsm services must be active before harvester initialization for configuration parameter retrieval
• Message Bus: Component registers with component name com.cisco.spvtg.ccsp.harvester or with name harvester when OneWifi is enabled
• TR-181 Data Model: Implements Device.X_RDKCENTRAL-COM_Report object hierarchy for telemetry configuration and control
• Configuration Files: Harvester.XML for TR-181 parameter definitions; Avro schema files in /usr/ccsp/harvester/ including InterfaceDevicesWifi.avsc, RadioInterfacesStatistics.avsc, GatewayAccessPointNeighborScanReport.avsc
• Startup Order: Initialize after CcspPsm, CcspWiFiAgent/OneWifi, and CcspWebPA services are running to ensure configuration retrieval and message transmission capabilities

Threading Model:

Harvester implements a multi-threaded architecture with independent worker threads for each telemetry report type enabling concurrent data collection without blocking operations.

• Threading Architecture: Multi-threaded with main event loop and dedicated worker threads for each telemetry report
• Main Thread: Handles TR-181 parameter requests, component lifecycle management, and message bus event processing
• Worker Threads:
  • IDW Thread: Polls associated device information at configured intervals and triggers Avro encoding and transmission (thread function: StartAssociatedDeviceHarvesting)
  • RIS Thread: Collects radio-level performance metrics periodically and packages data for upstream reporting
  • NAP Thread: Performs WiFi channel scans to detect neighboring access points and generates telemetry reports
  • Parodus Handler Thread: Maintains WebPA/Parodus connection and handles upstream message transmission with retry logic (handle_parodus thread function)
• Synchronization: Uses pthread mutexes for configuration parameter access, pthread condition variables for polling interval timing, and semaphores for thread lifecycle management

Component State Flow

Initialization to Active State

Harvester follows a structured initialization sequence establishing message bus connectivity, loading persisted configuration from CcspPsm, and spawning worker threads for telemetry collection based on enabled report types.
 
 

Runtime State Changes and Context Switching

During operation, Harvester responds to configuration changes, network events, and CcspWebPA connectivity status affecting telemetry collection behavior.

State Change Triggers:

• TR-181 parameter changes for Enabled flags triggering thread creation or termination for specific report types
• Polling period or reporting period modifications requiring reconfiguration of worker thread timers without thread restart
• Override TTL expiration causing automatic reversion to default configuration parameters retrieved from CcspPsm
• CcspWebPA connection loss triggering buffering of telemetry messages and retry attempts with exponential backoff

Context Switching Scenarios:

• OneWifi mode transitions between WiFi HAL direct access and R-BUS-based query mechanisms based on build-time configuration
• Collection mode switches between active polling and suspended state when Enabled parameter changes from true to false
• Transmission context switches between immediate send and retry queue when CcspWebPA/Parodus connection availability changes

Call Flow

Initialization Call Flow:

Request Processing Call Flow:

%%{init: {  "sequence": { "useMaxWidth": false }}}%%
sequenceDiagram
    participant Client as WebPA/TR-181 Client
    participant DM as Data Model Handler
    participant IDW as IDW Module
    participant Avro as Avro Encoder
    participant WebPA as WebPA Interface
    participant Cloud as Cloud Server
    
    Client->>DM: Set Enabled=true
    DM->>PSM: Store Parameter
    DM->>IDW: StartAssociatedDeviceHarvesting()
    IDW->>IDW: Create Collection Thread
    loop Polling Interval
        IDW->>WiFiHAL: wifi_getAssociatedDeviceDetail()
        WiFiHAL-->>IDW: Device Data Array
        IDW->>Avro: Package Avro Message
        Avro-->>IDW: Encoded Binary Payload
        IDW->>WebPA: sendWebpaMsg()
        WebPA->>Cloud: WRP Message (Avro/Binary)
        Cloud-->>WebPA: ACK
    end
    DM-->>Client: Success

TR‑181 Data Models

Supported TR-181 Parameters

Harvester implements vendor-specific TR-181 data model extensions under the Device.X_RDKCENTRAL-COM_Report namespace providing configuration and control of WiFi telemetry collection and reporting functionality. The implementation follows TR-181 Issue 2 architectural patterns with support for dynamic parameter updates, validation, commit/rollback operations, and persistent storage integration.

Object Hierarchy

Device.

└── X_RDKCENTRAL-COM_Report.
    ├── InterfaceDevicesWifi.
    │   ├── Enabled (boolean, R/W)
    │   ├── ReportingPeriod (unsignedInt, R/W)
    │   ├── PollingPeriod (unsignedInt, R/W)
    │   ├── Schema (string, R)
    │   ├── SchemaID (string, R)
    │   └── Default.
    │       ├── ReportingPeriod (unsignedInt, R/W)
    │       ├── PollingPeriod (unsignedInt, R/W)
    │       └── OverrideTTL (unsignedInt, R)
    ├── RadioInterfaceStatistics.
    │   ├── Enabled (boolean, R/W)
    │   ├── ReportingPeriod (unsignedInt, R/W)
    │   ├── PollingPeriod (unsignedInt, R/W)
    │   ├── Schema (string, R)
    │   ├── SchemaID (string, R)
    │   └── Default.
    │       ├── ReportingPeriod (unsignedInt, R/W)
    │       ├── PollingPeriod (unsignedInt, R/W)
    │       └── OverrideTTL (unsignedInt, R)
    └── NeighboringAP.
        ├── Enabled (boolean, R/W)
        ├── ReportingPeriod (unsignedInt, R/W)
        ├── PollingPeriod (unsignedInt, R/W)
        ├── Schema (string, R)
        ├── SchemaID (string, R)
        └── Default.
            ├── ReportingPeriod (unsignedInt, R/W)
            ├── PollingPeriod (unsignedInt, R/W)
            └── OverrideTTL (unsignedInt, R)

Internal Modules

Harvester is organized into specialized modules responsible for telemetry data collection, encoding, transmission, and configuration management with clear separation between report-specific collection logic and shared infrastructure components.

Component Interactions

Harvester maintains interactions with RDK-B middleware components for configuration management and telemetry transmission, WiFi subsystems for data collection, and cloud management systems for upstream reporting.

Interaction Matrix

Events Published by Harvester:

IPC Flow Patterns

Primary IPC Flow - Telemetry Collection with WebPA Messaging:
 
 

Event Notification Flow:

sequenceDiagram
    participant Client as TR-181 Client
    participant DM as Data Model
    participant IDW as IDW Thread
    participant PSM as PSM
    
    Client->>DM: Set Enabled=true
    DM->>PSM: Store Parameter
    PSM-->>DM: Success
    DM->>IDW: Signal Thread Start
    IDW->>IDW: Initialize Polling Timer
    IDW-->>DM: Thread Running
    DM-->>Client: Parameter Set Success

Implementation Details

Major HAL APIs Integration

Harvester integrates with WiFi HAL APIs to retrieve telemetry data for associated devices, radio statistics, and neighboring access point information with conditional R-BUS based access when OneWifi is enabled.

Core HAL APIs:

RBus Integration for OneWifi:

When OneWifi feature is enabled, Harvester uses RBus queries instead of direct HAL calls through the RBus API module:

Avro Schema and Serialization

Harvester uses Apache Avro for efficient binary serialization of telemetry data with schema validation:

• Schema Files: Avro schema definitions stored in /usr/ccsp/harvester/ directory including InterfaceDevicesWifi.avsc, RadioInterfacesStatistics.avsc, GatewayAccessPointNeighborScanReport.avsc
• Schema Loading: Avro schemas loaded at runtime during component initialization and cached for encoding operations
• Message Format: Binary encoded messages prefixed with magic number (0x85) and 32-byte schema ID hash for validation
• Encoding Process: Telemetry data structures marshalled into Avro format using writer buffers with automatic memory management

Configuration Persistence

Harvester stores configuration parameters in PSM with the following key patterns:

• eRT.com.cisco.spvtg.ccsp.harvester.InterfaceDevicesWifiEnabled
• eRT.com.cisco.spvtg.ccsp.harvester.InterfaceDevicesWifiPollingPeriod
• eRT.com.cisco.spvtg.ccsp.harvester.InterfaceDevicesWifiReportingPeriod
• Similar patterns for RadioInterfaceStatistics and NeighboringAP report types
• Default values stored separately with override TTL mechanism for temporary configuration changes

Key Implementation Logic

• Telemetry Collection Engine: Multi-threaded polling system with independent collection threads for each report type implementing producer-consumer pattern with configurable polling intervals validated against predefined acceptable values. Thread lifecycle management with conditional variables for timer-based polling and graceful shutdown handling. State machine implementation tracking collection thread status transitions between idle, active, and suspended states.

• Event Processing: Asynchronous message transmission through WebPA/Parodus with retry logic and exponential backoff for failed transmissions. Queue management for telemetry reports when cloud connectivity is unavailable with persistent buffering during extended outages. Parameter validation engine ensuring polling period less than or equal to reporting period and restricting values to supported intervals defined in source code arrays.

• Error Handling Strategy: Comprehensive error detection and recovery mechanisms ensuring system stability during WiFi HAL failures, WebPA connection losses, and resource constraints. Automatic retry logic with configurable backoff for transient HAL errors. Graceful degradation when telemetry transmission fails with local buffering and notification logging. Health monitoring logging telemetry collection success rates and transmission statistics for operational visibility.

Key Configuration Files