Telemetry in Emulator with Cron job support

Introduction:

Telemetry is the automatic recording and transmission of data from remote or inaccessible sources to an IT system in a different location for monitoring and analysis. A cron utility is used to schedule the Telemetry feature with a configured frequency and / or configured date and time.

RDKB Telemetry Components:

1. Xconf Server
2. An virtual box Emulator with RDK Broadband image
3. Tftp Server

RDKB Telemetry Architecture:

XConf Server Configuration procedure: RDK-B Emulator will connect to Xconf Server to get the device configuration information, telemetry profile information, telemetry upload schedule and log upload repository (ip address of tftp server). It will search for the markers (contents mentioned in the permanent profile) in the mentioned log file. If it is found in the log file, it will process those information and create a text file and upload that file in the tftp server. This way all the critical data related to device health will be available on timely basis in the tftp server for further analysis.

1. Login to Xconf Server and go to DCM menu item and click on “Formulas” sub menu

2. Click on Create button

3. i) Provide a unique “Name”, along with “Description”, and build conditions and save the formula.
ii) Verify Percentage and Priority field(normally it displays the sequence number of formula, can be changed to make the formula higher/lower priority) is auto field up with their respective values
iii) Go to “Build condition” section and click on the first empty field to get all the available field displayed there
iv) Select estbMacAddress and provide the MAC address of the Emulator.

4. Verify the created formula (ex. RDKB_EMU_RDKM) displayed in the formula list.

5. Click on Edit formula for the created formula (ex. RDKB_EMU_RDKM).

6. Edit formula page opens.

7. Click on “Create Device Settings” and fill up the fields as displayed in the screen below and click on save button:

8. Click on DCM menu then “Upload repository” submenu and click on “Create” button

9. Provide inputs as mentioned below screen and save the upload repository.

10. Click on Formula and go back to the Edit formula page of the created formula (ex. RDKB_EMU_RDKM) and click on “Create Log Upload settings” button. Upload repository drop down menu field will show all the upload repositories including the created one (ex. Chennai_22Aug_repo)

11. Provide inputs as mentioned in the screen below:

12. Save the “Edit Formula” page

13. Verify the Formula page for the created formula (ex. RDKB_EMU_RDKM) where View Settings column displays the “Device” and “LogUpload”

14. Go to Telemetry menu and click on “Permanent Profile”

15. Click on “Create” button to create permanent profile with different markers.

Name – Give an unique name for the Telemetry permanent profile

Schedule – This gives the information about the schedule log upload frequency and / or date and time for the telemetry upload schedule.

For ex.

*/10 * * * * :- This will schedule the telemetry log upload at every 10 min.

Below is the format for this field.

Upload repository – This gives the URL for the server where the Telemetry logs need to be uploaded.

Telemetry profile entries - 

First field is “Header” - This is the name for the particular logs collection. This is a custom name and can be decided based on the use case or error condition or any other scenario.
Second field is “Content” - This is the "log message string" that appears in the RDK logs.
Third field is “type” - This is the name of the log file where the above log message string needs to be searched.
Fourth field is “Skip frequency” - This is a skip frequency. It can have values like 0, 1, 2 etc.
                     0 - This particular log message is never skipped.
                     1 - Skip every alternate occurrence of this log message.
                     2 - Skip 2 consecutive log message occurrences and then collect 1 occurrence and then again skip 2 occurrences.

Based on the Emulator supported markers available, we have selected few of them as mentioned below:

16. Save Permanent profile (ex. Perm_Prof_22Aug) and it will display in the Permanent profiles page

17. Click on “Telemetry” and then click on “Targeting Rules”

18. Click on “Create” button, provide Rule name and select different rule parameters and map the Permanent profile with this rule by selecting “Bound profile” drop down menu. Save the Targeting rule.

19. Verify the created Targeting rule displayed on the Targeting rule page

Emulator side configuration and testing process:

Below steps needed after flashing for the first time:

1. Flash the Telemetry image and load it on Emulator
2. Configure the CLOUDURL parameter and Log Server/Http Server parameter by providing IP address of XConf server in the two mentioned files /etc/include.properties and /etc/dcm.properties respectively.

For ex. In include.properties file:

CLOUDURL=http://35.155.171.121:9092/xconf/swu/stb?eStbMac=

In dcm.properties file:

DCM_LOG_SERVER_URL=http://35.155.171.121:9092/loguploader/getSettings
DCM_HTTP_SERVER_URL=http://35.155.171.121/xconf/telemetry_upload.php
DCM_LA_SERVER_URL=http://35.155.171.121/xconf/logupload.php

3. After board boot up, disable log rotation by using the command: 

Log Monitor Disable

4. Verify the image version: cat /version.txt
5. Perform reboot: reboot -f

Important Note:

Disabling log monitor is required because log rotation feature will wipe out logs when threshold reached, due to which telemetry feature can't able to identify the event count.

Telemetry functionality  requires rdk logs.

To bring both log rotation and telemetry feature at same time, user needs to increase the threshold size of max size which is used by log rotation feature  . Please follow Log Rotation Support in RPI User manual - Broadband - 2019 M8 to change threshold size of log rotation.

Below steps are to check telemetry log whenever box boots up ( after first time boot ) :

1. Verify tftp server is running and XConf DCM and Telemetry Permanent profile is configured properly (see Telemetry User manual for XConf side configuration for Telemetry)
2. After reboot, verify Telemetry log: cat /rdklogs/logs/telemetry.log
3. Verify that rtl_json.txt and zipped log files for all other logs displayed in tftp server
4. Verify that rtl_json file and zipped log files for all other logs displayed in AWS(http) server(http://35.155.171.121/xconf/upload)
5. Verify that the rtf_json file and zipped log files are getting uploaded at the frequency as configured in the permanent profile using "Schedule".