Harvester

What is Harvester ?

• Harvester is a CCSP component, which runs on an RDKB gateway to collect data from the Wi-Fi driver in three areas: Associated Devices, Radio Traffic, and Neighboring Wireless Networks.

Repository 

• PATH : rdkb/components/generic/harvester
• CLONE : git clone https://code.rdkcentral.com/r/rdkb/components/generic/harvester

Architecture

• The Harvester Agent is a  CCSP component to handle configuration parameters to control data extraction from the device.
• The harvester operates by polling the WiFi HAL (Hardware Access Layer) for information for each of the three reports and may store multiple polls for bulk transmission to WebPA and subsequent components.

TR-181

TR-181 Object is X_RDKCENTRAL-COM_Report. A Device Report is an Encoded Avro Schema Payload

1. InterfaceDevicesWifi

   • The InterfaceDevicesWifi Report sends WiFi Metrics Data for the devices connected to the Gateway harvested from the WiFi HAL.
   • This report is generated by the Harvester Component

   • It supports various parameters to control the Harvesting Periodicity, Schema Identification and Default Values for the various intervals.
     

     TR-181	Type	Default value	Range	Description
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.Enabled	Boolean	False		Enables or disables InterfaceDevicesWifi data collection. A value of true means that the harvester will collect data about devices connected to all the SSID's which are Enabled on the system and send them to the Cloud
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.PollingPeriod	unsigned integer	900	1, 5, 10, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400	Stores the InterfaceDevicesWifi polling time period in seconds  An unsigned integer in seconds  denotes the time the harvester will wait between querying associated device data on all SSID's which are enabled in the System. Must be less than or equal to Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.ReportingPeriod
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.ReportingPeriod	unsigned integer	900	1, 5, 10, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400 Must be greater than or equal to the PollingPeriod	Stores the InterfaceDevicesWifi reporting time period in seconds  An unsigned integer in seconds  denotes the time the harvester will wait between uploading associated device data on all SSID's which are enabled in the System to the cloud. Must be greater than or equal to Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.PollingPeriod
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.Schema	string			Stores the Schema used to encode the Associated Devices Scan Data A String consisting of the Avro Schema used to Encode the InterfaceDevicesWifi data from the HAL
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.SchemaID	string			Stores the UUID/HASH of the Schema used to encode the Scan data UUID/HASH as a string of the Schema used to Encode the data from HAL
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.Default.PollingPeriod	unsigned integer	900	1, 5, 10, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400	Stores the InterfaceDevicesWifi default polling time period in seconds  An unsigned integer in seconds denotes the default time the harvester will wait between querying associated device data on all SSID's which are enabled in the System.
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.Default.ReportingPeriod	unsigned integer	900	1, 5, 10, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400 Must be greater than Polling Period	Stores the InterfaceDevicesWifi default reporting time period in seconds  An unsigned interger in seconds denotes the default time the harvester will wait between uploading associated device data on all SSID's which are enabled in the System to the cloud.
     Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.Default.OverrideTTL	unsigned integer	300		Stores the time which an accelerated Polling or Reporting period will remain active An unsigned integer in seconds denotes the time that a request to accelerate a polling interval or reporting interval will remain active. Defaults to 300 seconds if a value is not specified

2. RadioInterfaceStatistics

   • The RadioInterfacesStatistics Report sends WiFi Radio Data.
   • This report is generated by the Harvester Component

   • It supports various parameters to control the Harvesting Periodicity, Schema Identification and Default Values for the various periods

     TR-181	Type	Default value	Range	Description
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.Enabled	Boolean	False		Enables or disables RadioInterfaceStatistics data collection. A value of true means that the harvester will collect data about devices connected to all the SSID's which are Enabled on the system and send them to the Cloud
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.PollingPeriod	unsigned integer	900	1, 5, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400	Stores the RadioInterfaceStatistics polling time period in seconds  An unsigned integer in seconds  denotes the time the harvester will wait between querying associated device data on all SSID's which are enabled in the System. Must be less than or equal to Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.ReportingPeriod
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.ReportingPeriod	unsigned integer	900	1, 5, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400 Must be greater than or equal to the PollingPeriod	Stores the RadioInterfaceStatistics reporting time period in seconds An unsigned integer in seconds  denotes the time the harvester will wait between uploading associated device data on all SSID's which are enabled in the System to the cloud. Must be greater than or equal to Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.PollingPeriod
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.Schema	string			Stores the Schema used to encode the Associated Devices Scan Data A String consisting of the Avro Schema used to Encode the RadioInterfaceStatistics data from the HAL
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.SchemaID	string			Stores the UUID/HASH of the Schema used to encode the Scan data UUID/HASH as a string of the Schema used to Encode the data from HAL
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.Default.PollingPeriod	unsigned integer	900	1, 5, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400	Stores the RadioInterfaceStatistics default polling time period in seconds  An unsigned integer in seconds denotes the default time the harvester will wait between querying radio traffic data on all SSID's which are enabled in the System.
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.Default.ReportingPeriod	unsigned integer	900	1, 5, 15, 30, 60, 300, 900, 1800, 3600, 10800, 21600, 43200, 86400 Must be greater than Polling Period	Stores the RadioInterfaceStatistics default reporting time period in seconds  An unsigned interger in seconds denotes the default time the harvester will wait between uploading radio traffic data on all SSID's which are enabled in the System to the cloud.
     Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.Default.OverrideTTL	unsigned integer	300		Stores the time which an accelerated Polling or Reporting period will remain active An unsigned integer in seconds denotes the time that a request to accelerate a polling interval or reporting interval will remain active. Defaults to 300 seconds if a value is not specified

3. NeighboringAP

   • The NeighboringAP Report sends data about all the neighboring access points
   • This report is generated by the Harvester Component

   • It supports various parameters to control the Harvesting Periodicity, Schema Identification and Default Values for the various periods

     TR-181	Type	Default value	Range	Description
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.Enabled	Boolean	False		Enables or disables NeighboringAP data collection. A value of true means that the harvester will collect data about devices connected to all the SSID's which are Enabled on the system and send them to the Cloud
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.PollingPeriod	unsigned integer	21600	300, 900, 1800, 3600, 10800, 21600, 43200, 86400	Stores the NeighboringAP polling time period in seconds  An unsigned integer in seconds  denotes the time the harvester will wait between querying associated device data on all SSID's which are enabled in the System. Must be less than or equal to Device.X_RDKCENTRAL-COM_Report.NeighboringAP.ReportingPeriod
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.ReportingPeriod	unsigned integer	43200	300, 900, 1800, 3600, 10800, 21600, 43200, 86400 Must be greater than or equal to the PollingPeriod	Stores the NeighboringAP reporting time period in seconds  An unsigned integer in seconds  denotes the time the harvester will wait between uploading NeighboringAP data on all SSID's which are enabled in the System to the cloud. Must be greater than or equal to Device.X_RDKCENTRAL-COM_Report.NeighboringAP.PollingPeriod
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.Schema	string			Stores the Schema used to encode the Associated Devices Scan Data A String consisting of the Avro Schema used to Encode the NeighboringAP data from the HAL
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.SchemaID	string			Stores the UUID/HASH of the Schema used to encode the Scan data UUID/HASH as a string of the Schema used to Encode the data from HAL
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.Default.PollingPeriod	unsigned integer	21600	300, 900, 1800, 3600, 10800, 21600, 43200, 86400	Stores the NeighboringAP default polling time period in seconds  An unsigned integer in seconds denotes the default time the harvester will wait between querying associated device data on all SSID's which are enabled in the System.
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.Default.ReportingPeriod	unsigned integer	43200	300, 900, 1800, 3600, 10800, 21600, 43200, 86400 Must be greater than Polling Period	Stores the NeighboringAP default reporting time period in seconds  An unsigned interger in seconds denotes the default time the harvester will wait between uploading NeighboringAP device data on all SSID's which are enabled in the System to the cloud.
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.Default.OverrideTTL	unsigned integer	43200		Stores the time which an accelerated Polling or Reporting period will remain active An unsigned integer in seconds denotes the time that a request to accelerate a polling interval or reporting interval will remain active. Defaults to 300 seconds if a value is not specified
     Device.X_RDKCENTRAL-COM_Report.NeighboringAP.LastScanData	string			Returns the last scan data from previous Neighbor AP Scan
     Device.X_RDKCENTRAL- COM_Report.NeighboringAP.OnDemandScan	string			Initiates a Neighbor AP Scan and immediately returns the result

SAMPLE OUTPUT AND LOGS

• In Rpi , by default harvester process will be not running.
• To launch the process

Go to /usr/ccsp/harvester folder
/usr/bin/harvester –DEBUG –LOGFILE /tmp/logfile.txt &

• Sample Log 
  

• TR-181 output from Rpi on enabling below harvester parameters 
  dmcli eRT setv Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.Enabled bool true
  dmcli eRT setv Device.X_RDKCENTRAL-COM_Report.RadioInterfaceStatistics.Enabled bool true
  dmcli eRT setv Device.X_RDKCENTRAL-COM_Report.NeighboringAP.Enabled bool true
  
  
• Sample debug logs from Rpi logfile.txt

• Harvester logs are available under /rdklogs/logs/Harvesterlog.txt.<0/1>
  Sample:
  

  220228-10:52:18.395433 [mod=Harvester, lvl=WARN] [tid=31156] Register empty object : com.cisco.spvtg. 220228-10:52:18.395571 [mod=Harvester, lvl=WARN] [tid=31156] Register empty object : com.cisco.spvtg.ccsp. 220228-10:52:18.396881 [mod=Harvester, lvl=WARN] [tid=31156] DM Library module name = CCSP Base Data Model 220228-10:52:18.398988 [mod=Harvester, lvl=WARN] [tid=31156] DM Library module name = COSA_TR181_Harvester 220228-10:52:18.400870 [mod=Harvester, lvl=WARN] [tid=31156] Loading DM library libHarvesterSsp.so... 220228-10:52:18.448430 [mod=Harvester, lvl=WARN] [tid=31156] Search for empty object in 'dslhCpeCheckEmptyObject' 220228-10:52:18.452440 [mod=Harvester, lvl=INFO] [tid=31156]  rdk-logger initialzed! 220228-10:52:18.457414 [mod=Harvester, lvl=INFO] [tid=31156] RDK_LOG_WARN, HARV : /tmp/harvester_initialized created 220228-10:52:18.457656 [mod=Harvester, lvl=WARN] [tid=31156]   HARV : /tmp/harvester_initialized created 220228-10:52:18.463001 [mod=Harvester, lvl=INFO] [tid=31168]  Init for parodus Success..!! 220228-10:52:19.195747 [mod=Harvester, lvl=INFO] [tid=31161]  Ethernet WAN is enabled 220228-10:52:19.196703 [mod=Harvester, lvl=INFO] [tid=31161] deviceMAC is b827eb221636 220228-10:52:19.216732 [mod=Harvester, lvl=INFO] [tid=31161]  Sent message successfully to parodus 220228-10:53:25.934303 [mod=Harvester, lvl=WARN] [tid=31159] DslhObjcoValidate: Enabled, bsUpdate = 3 220228-10:53:25.934586 [mod=Harvester, lvl=INFO] [tid=31159] RDKB_HARVESTER_VALUE_CHANGED - WiFiInterfaceDevice Report Enabled 220228-10:53:25.934742 [mod=Harvester, lvl=WARN] [tid=31159] Before committing:Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi. 220228-10:53:25.935239 [mod=Harvester, lvl=WARN] [tid=31159]  Harvester InterfaceDeviceWifi Report STARTED SetIDWHarvestingStatus 220228-10:53:25.944784 [mod=Harvester, lvl=WARN] [tid=31159] After committing:Device.X_RDKCENTRAL-COM_Report.InterfaceDevicesWifi.:0 220228-10:53:26.184683 [mod=Harvester, lvl=INFO] [tid=680]  Harvester GetWiFiApGetAssocDevicesData : SSID RPi3_RDKB-AP1 is NOT ENABLED  or ERROR retured 0 220228-10:54:10.478550 [mod=Harvester, lvl=WARN] [tid=31159] DslhObjcoValidate: Enabled, bsUpdate = 3 220228-10:54:10.478789 [mod=Harvester, lvl=INFO] [tid=31159] RDKB_HARVESTER_VALUE_CHANGED - WiFiNeighbourAP Report Enabled 220228-10:54:10.478947 [mod=Harvester, lvl=WARN] [tid=31159] Before committing:Device.X_RDKCENTRAL-COM_Report.NeighboringAP. 220228-10:54:10.484989 [mod=Harvester, lvl=WARN] [tid=31159] After committing:Device.X_RDKCENTRAL-COM_Report.NeighboringAP.:0 220228-10:54:10.638449 [mod=Harvester, lvl=ERROR] [tid=1958]  Harvester GetRadioNeighboringAPData : Radio wifi1 is NOT ENABLED  or ERROR retured 0 220228-10:54:10.681146 [mod=Harvester, lvl=INFO] [tid=1958]  Sent message successfully to parodus