Required equipment's


  • Raspberry pi device

  • Ethernet cable

  • SD card

  • Standard USB keyboard
  • TV/Monitor with HDMI input

Telemetry


Telemetry is the automatic recording and transmission of data from remote or inaccessible sources to an IT system in a different location for monitoring and analysis.

Telemetry Components

  • XConf Server
  • RPI Board with RDK Broadband image
  • Tftp Server

Steps to build code 


  • mkdir <workspace dir>
  • cd <workspace dir>
  • repo init -u https://code.rdkcentral.com/r/manifests -b kirkstone -m rdkb-extsrc.xml
  • repo sync -j`nproc` --no-clone-bundle
  • MACHINE=raspberrypi4-rdk-broadband source meta-cmf-raspberrypi/setup-environment      //For 32 Bit Image
  • MACHINE=raspberrypi4-64-rdk-broadband source meta-cmf-raspberrypi/setup-environment   //For 64 Bit Image
  • bitbake rdk-generic-broadband-image

Steps to flash image


  • The image will be available under build-raspberrypi-rdk-broadband/tmp/deploy/images/raspberrypi-rdk-broadband
  • Copy the image "rdkb-generic-broadband-image_default_<xx>.rootfs.rpi-sdimg" to your server.
  • To flash the image
        dmesg
        mount
        umount <partition-mountpoint>
        sudo dd if=<rpi sdimg> of=/dev/sdb bs=1
  • Remove the SD card and insert it to the Raspberry Pi SD card slot
  • Bring up the device

XConf server


Define Environments   

       

Define Models 

       

Define MAC List

     

Configuring Telemetry on XConf server


Log upload

  • Change Application to stb in top right corner of the window

         
  • Defining the formula

URL

https://xconf.rdkcentral.com:19093/admin/ux/#/formulas/all

Flow

XConf-server ->DCM -> Formulas -> Create

Name

<Unique name for log upload >

Description

<Enter a short description>

Percentage

100

Priority

4

Build condition

estbMacAddress is <MAC-Address> 

               Sample 

     
     

            As soon as the formula is saved, displays the device settings and Log upload settings options
     

  • Device Settings

Name 

RDKB_telemetry

CheckOnReboot

true

Are active

true

Expression

2 1 2 1 1

   

  • Defining the Upload repositories

URL

https://xconf.rdkcentral.com:19093/admin/ux/#/uploadrepository

Flow

XConf-server ->DCM -> Upload repositories

Name

RDKB_telemetry

Description

RDKB_telemetry

URL

tftp   : xconf.rdkcentral.com

HTTP: https://xconf.rdkcentral.com/xconf/logupload.php

   

  • Log upload settings

Name

<Enter the name given during Log upload formula creation>

Upload On Reboot

true

Number of Days

1

Are settings active

true

Upload Repository

RDKB_telemetry

Cron Expression

2 1 1 1 1

     

  • Test Page

URL

https://xconf.rdkcentral.com:19093/admin/ux/#/formulas/all

Path

Xconf-server-> DCM -> Test Page

     


Telemetry

  • Defining Telemetry profile

URL

https://xconf.rdkcentral.com:19093/admin/ux/#/formulas/all

Path

XConf-server-> Telemetry->Permanent Profiles -> Create

                Sample :
     

     

  • Create Targeting rule 

URL

https://xconf.rdkcentral.com:19093/admin/ux/#/formulas/all

Path

XConf-server-> Telemetry->Targeting rule -> Create

  1. Give the Unique rule name
  2. Select the rule. Here, used estbMacAddress as the rule
  3. Select the Bound profile (Telemetry Permanent Profile Name)

         Sample

     

  • Test Page

URL

https://xconf.rdkcentral.com:19093/admin/ux/#/formulas/all

Path

XConf-server-> Telemetry->Test Page

  1. To test whether the configured profile details are retrieved properly from XConf-Server
  2. Give the rule specified in “Targeting Rule” section i.e., estbMacAddress here   

       Sample:

     

Configuring Telemetry on Raspberry pi


  • Once the board is up, go to /etc/dcm.properties and add the below details
  • Restart the dcm-log service using “systemctl restart dcm-log
  • To check the status of the service “systemctl status dcm-log
  • By triggering the service, RPI starts uploading the logs to the XConf-Server
  • The log upload and Telemetry markers can be uploaded via tftp and http
  • The profile can be verified using curl, conf file, through logs and in server as below

1.  CURL

  • Syntax : curl -i 'http://<IP>:19092/loguploader/getSettings?estbMacAddress=<MAC>’
Sample: 1  TFTP

root@RaspberryPi-Gateway:~# ifconfig erouter0
erouter0  Link encap:Ethernet  HWaddr <MAC>
          inet addr:<Device IP>  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::ba27:ebff:fe22:1636/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18748 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3081 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2614475 (2.4 MiB)  TX bytes:590848 (577.0 KiB)

root@RaspberryPi-Gateway:~# curl -i 'https://xconf.rdkcentral.com:19092/loguploader/getSettings?estbMacAddress=<MAC>'
HTTP/1.1 200 OK

Date: Fri, 31 Jan 2020 10:59:09 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(9.2.1.v20140609)

{"urn:settings:GroupName":"RDKB_telemetry","urn:settings:CheckOnReboot":true,"urn:settings:CheckSchedule:cron":"2 1 2 1 1","urn:settings:CheckSchedule:DurationMinutes":0,"urn:settings:LogUploadSettings:Message":null,"urn:settings:LogUploadSettings:Name":"RDKB_telemetry","urn:settings:LogUploadSettings:NumberOfDays":0,"urn:settings:LogUploadSettings:UploadRepositoryName":"RDKB_telemetry","urn:settings:LogUploadSettings:RepositoryURL":"tftp://xconf.rdkcentral.com","urn:settings:LogUploadSettings:UploadOnReboot":true,"urn:settings:LogUploadSettings:UploadImmediately":false,"urn:settings:LogUploadSettings:upload":true,"urn:settings:LogUploadSettings:UploadSchedule:cron":"2 1 1 1 1","urn:settings:LogUploadSettings:UploadSchedule:levelone:cron":null,"urn:settings:LogUploadSettings:UploadSchedule:leveltwo:cron":null,"urn:settings:LogUploadSettings:UploadSchedule:levelthree:cron":null,"urn:settings:LogUploadSettings:UploadSchedule:DurationMinutes":0,"urn:settings:VODSettings:Name":null,"urn:settings:VODSettings:LocationsURL":null,"urn:settings:VODSettings:SRMIPList":null,"urn:settings:TelemetryProfile":{"id":"72ac2ca9-b248-4a5e-b120-27ff37382564","telemetryProfile":[{"header":"Firewall","content":"starting firewall service","type":"FirewallDebug.txt","pollingFrequency":"1"}],"schedule":"3","expires":0,"telemetryProfile:name":"RDKB_tele","uploadRepository:URL":"xconf.rdkcentral.com","uploadRepository:uploadProtocol":"TFTP"}}

Sample: 2  HTTP

root@RaspberryPi-Gateway:~# curl -i 'https://xconf.rdkcentral.com:9092/loguploader/getSettings?estbMacAddress=<MAC>'
HTTP/1.1 200 OK

Date: Fri, 31 Jan 2020 12:15:26 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(9.2.1.v20140609)

{"urn:settings:GroupName":"RDKB_telemetry","urn:settings:CheckOnReboot":true,"urn:settings:CheckSchedule:cron":"2 1 2 1 1","urn:settings:CheckSchedule:DurationMinutes":0,"urn:settings:LogUploadSettings:Message":null,"urn:settings:LogUploadSettings:Name":"RDKB_telemetry","urn:settings:LogUploadSettings:NumberOfDays":0,"urn:settings:LogUploadSettings:UploadRepositoryName":"RDKB_telemetry","urn:settings:LogUploadSettings:RepositoryURL":"http://xconf.rdkcentral.com/xconf/logupload.php","urn:settings:LogUploadSettings:UploadOnReboot":true,"urn:settings:LogUploadSettings:UploadImmediately":false,"urn:settings:LogUploadSettings:upload":true,"urn:settings:LogUploadSettings:UploadSchedule:cron":"2 1 1 1 1","urn:settings:LogUploadSettings:UploadSchedule:levelone:cron":null,"urn:settings:LogUploadSettings:UploadSchedule:leveltwo:cron":null,"urn:settings:LogUploadSettings:UploadSchedule:levelthree:cron":null,"urn:settings:LogUploadSettings:UploadSchedule:DurationMinutes":0,"urn:settings:VODSettings:Name":null,"urn:settings:VODSettings:LocationsURL":null,"urn:settings:VODSettings:SRMIPList":null,"urn:settings:TelemetryProfile":{"id":"72ac2ca9-b248-4a5e-b120-27ff37382564","telemetryProfile":[{"header":"Firewall","content":"starting firewall service","type":"FirewallDebug.txt","pollingFrequency":"1"}],"schedule":"3","expires":0,"telemetryProfile:name":"RDKB_tele","uploadRepository:URL":"https://xconf.rdkcentral.com/xconf/telemetry_upload.php","uploadRepository:uploadProtocol":"HTTP"}}


2. Configuration file

  • Once the profile details are successfully retrieved , the conf file will be created under /tmp which has the server configurations 
  • The telemetry markers are compared with the corresponding log files and if the match is found then the markers are uploaded to the http://xconf.rdkcentral.com/xconf/upload/
Sample 1: tftp
root@RaspberryPi-Gateway:~# cat /tmp/DCMSettings.conf

urn:settings:GroupName=RDKB_telemetry
urn:settings:CheckOnReboot=true
urn:settings:CheckSchedule:cron=2 1 2 1 1
urn:settings:CheckSchedule:DurationMinutes=0
urn:settings:LogUploadSettings:Message=null
urn:settings:LogUploadSettings:Name=RDKB_telemetry
urn:settings:LogUploadSettings:NumberOfDays=0
urn:settings:LogUploadSettings:UploadRepositoryName=RDKB_telemetry
urn:settings:LogUploadSettings:RepositoryURL=tftp://xconf.rdkcentral.com
urn:settings:LogUploadSettings:UploadOnReboot=true
urn:settings:LogUploadSettings:UploadImmediately=false
urn:settings:LogUploadSettings:upload=true
urn:settings:LogUploadSettings:UploadSchedule:cron=2 1 1 1 1
urn:settings:LogUploadSettings:UploadSchedule:levelone:cron=null
urn:settings:LogUploadSettings:UploadSchedule:leveltwo:cron=null
urn:settings:LogUploadSettings:UploadSchedule:levelthree:cron=null
urn:settings:LogUploadSettings:UploadSchedule:DurationMinutes=0
urn:settings:VODSettings:Name=null
urn:settings:VODSettings:LocationsURL=null
urn:settings:VODSettings:SRMIPList=null
"urn:settings:TelemetryProfile":{"id":"72ac2ca9-b248-4a5e-b120-27ff37382564","telemetryProfile":[{"header" : "Firewall","content" : "staring firewall service","type" : "FirewallDebug.txt","pollingFrequency":"1"}],"schedule":"3","expires":0,"telemetryProfile:name":"RDKB_tele,"uploadRepository:URL":"xconf.rdkcentral.com","uploadRepository:uploadProtocol":"TFTP"}


Sample 2:  http
root@RaspberryPi-Gateway:~# cat /tmp/DCMSettings.conf

urn:settings:GroupName=RDKB_telemetry
urn:settings:CheckOnReboot=true
urn:settings:CheckSchedule:cron=2 1 2 1 1
urn:settings:CheckSchedule:DurationMinutes=0
urn:settings:LogUploadSettings:Message=null
urn:settings:LogUploadSettings:Name=RDKB_telemetry
urn:settings:LogUploadSettings:NumberOfDays=0
urn:settings:LogUploadSettings:UploadRepositoryName=RDKB_telemetry
urn:settings:LogUploadSettings:RepositoryURL= https://xconf.rdkcentral.com/xconf/logupload.php 
urn:settings:LogUploadSettings:UploadOnReboot=true
urn:settings:LogUploadSettings:UploadImmediately=false
urn:settings:LogUploadSettings:upload=true
urn:settings:LogUploadSettings:UploadSchedule:cron=2 1 1 1 1
urn:settings:LogUploadSettings:UploadSchedule:levelone:cron=null
urn:settings:LogUploadSettings:UploadSchedule:leveltwo:cron=null
urn:settings:LogUploadSettings:UploadSchedule:levelthree:cron=null
urn:settings:LogUploadSettings:UploadSchedule:DurationMinutes=0
urn:settings:VODSettings:Name=null
urn:settings:VODSettings:LocationsURL=null
urn:settings:VODSettings:SRMIPList=null
"urn:settings:TelemetryProfile":{"id":"72ac2ca9-b248-4a5e-b120-27ff37382564","telemetryProfile":[{"header" : "Firewall","content" : "starting firewall service","type" : "FirewallDebug.txt","pollingFrequency":"1"}],"schedule":"3","expires":0,"telemetryProfile:name":"RDKB_tele","uploadRepository:URL":"http://xconf.rdkcentral.com/xconf/telemetry_upload.php","uploadRepository:uploadProtocol":"HTTP"}


 3. Logs

  • Log file to monitor and analyze the files which are uploaded to server
  • /rdklogs/logs/dcmscript.log
    TFTP
Uploading Logs with DCM UploadOnReboot set to true
200131-09:23:25.892859 dca : Direct Connection HTTP RESPONSE CODE:
200131-09:23:25.899693 dca: Direct connection success - ret:0
200131-09:23:25.908621 dca: Json message successfully submitted.                                                                  
01-31-20-09-23AM-ArmConsolelog.txt.0
01-31-20-09-23AM-BootTime.log
01-31-20-09-23AM-CRlog.txt.0
01-31-20-09-23AM-Consolelog.txt.0
01-31-20-09-23AM-FirewallDebug.txt
01-31-20-09-23AM-LM.txt.0
01-31-20-09-23AM-MnetDebug.txt
01-31-20-09-23AM-PAMlog.txt.0
01-31-20-09-23AM-PARODUSlog.txt.0
01-31-20-09-23AM-PSMlog.txt.0
01-31-20-09-23AM-TDMlog.txt.0
01-31-20-09-23AM-TR69log.txt.0
01-31-20-09-23AM-WEBPAlog.txt.0
01-31-20-09-23AM-WiFilog.txt.0
01-31-20-09-23AM-dcmscript.log
01-31-20-09-23AM-swupdate.log
01-31-20-09-23AM-version.txt
01-31-20-09-23AM-wifihealth.txt
Uploading logs 01-31-20-09-23AM.tgz onto xconf.rdkcentral.com
Uploading logs 01-31-20-09-23AM.tgz onto xconf.rdkcentral.com
Moving to Previous Logs Backup Folder

          HTTP   

Uploading Logs with DCM UploadOnReboot set to true
200131-11:22:00.729982 dea: Sleeping for 10 before upload.
200131-11:22:11.392670 dca: Direct connection success - ret:0
200131-11:22:11.399931 dca: Json message successfully submitted.
01-31-20-11-22AM-Consolelog.txt.0
01-31-20-11-22AM-dcmscript.log
01-31-20-11-22AM-version.txt
01-31-20-11-22AM-wifihealth.txt
Uploading logs 01-31-20-11-21AM.tgz onto https://xconf.rdkcentral.com/xconf/logupload.php
Moving to Previous Logs Backup Folder
  • /rdklogs/logs/telemetry.log

    TFTP
TFTPIP:xconf.rdkcentral.com
200131-09:23:15.165431 dea: Using Direct commnication
number of proUPdell:42
Upload protocol telemetry is:TFTP
before TFTP Load-----------
number of delim:17
tftp ip is :xconf.rdkcentral.com
rcl_json.txt available, going for tftp upload
0
TFTP Telemetry succeded !!!
call uploadLogOnReboot
Sleeping for seven minutes
Done sleeping prev logpath /zdklogs/logs//PreviousLogs
ckp100----------prev log path------------/rdkloga/logs//PreviousLoga
ckp101------------upload log--------------true
total 390

drwxr-xr-x    2 root    root             1024 Jan 31 09:23 .                             
drwxr-xr-x    4 root    root             1024 Jan 31 09:23 . .
-rw-r--r--    1 root    root            15510 Jan 31 09:48 01-31-20-09-23AM-ArmConsolelog.txt.Ο  
-rw-r--r--    1 root    root             214  Jan 31 09:12 01-31-20-09-23AM-BootTime.log                             
-rw-r--r--    1 root    root            18632 Jan 31 09:12 01-31-20-09-23AM-CRlog.txt.0 
-rw-r--r--    1 root    root            73531 Jan 31 09:23 01-31-20-09-23AM-Consolelog.txt.0
-rw-r--r--    1 root    root            50543 Jan 31 09:12 01-31-20-09-23AM-FirewallDebug.txt 
-rw-r--r--    1 root    root            20141 Jan 31 09:23 01-31-20-09-23AM-LM.txt.0
-rw-r--r--    1 root    root            17514 Jan 31 09:12 01-31-20-09-23AM-MnetDebug.txt
-rw-r--r--    1 root    root            46715 Jan 31 09:48 01-31-20-09-23AM-PAMlog.txt.0 
-rw-r--r--    1 root    root            16562 Jan 31 09:20 01-31-20-09-23AM-PARODUSiog.txt.0   
-rw-r--r--    1 root    root            50662 Jan 31 09:48 01-31-20-09-23AM-P5Mlog.txt.0
-rw-r--r--    1 root    root             1478 Jan 31 09:12 01-31-20-09-23AM-TDMlog.txt.0
-rw-r--r--    1 root    root            12187 Jan 31 09:12 01-31-20-09-23AM-TR69log.txt.0 
-rw-r--r--    1 root    root             6687 Jan 31 09:23 01-31-20-09-23AM-WEBPAlog.txt.0                  
-rw-r--r--    1 root    root            29083 Jan 31 09:12 01-31-20-09-23AM-WiFilog.txt.0
-rw-r--r--    1 root    root              460 Jan 31 09:23 01-31-20-09-23AM-domscript.log 
-rw-r--r--    1 root    root             2079 Jan 31 09:16 01-31-20-09-23AM-dcmscript.log
-rw-r--r--    1 root    root              176 Jan 31 09:23 01-31-20-09-23AM-version.txt     
-rw-r--r--    1 root    root            12410 Jan 31 09:22 01-31-20-09-23AM-wifihealth.txt
number of proUPde12:42
Upload protocol logupload is TFTP
0
Done Uploading Logs and removing rtl_json.txt file

       HTTP

Upload Protocol is:HTTP
call uploadLogOnReboot
Sleeping for seven minutes 
Done sleeping prev logpath /rdklogs/logs//PreviousLogs
ckp100-------------prev log path-------------/rdklogs/logs//PreviousLogs
ckp101---------------------upload log-----true
total 13
drwxr-xr-x    2 root    root             1024 Jan 31 09:23 .                             
drwxr-xr-x    4 root    root             1024 Jan 31 09:23 . .
-rw-r--r--    1 root    root            73531 Jan 31 09:23 01-31-20-09-23AM-Consolelog.txt.0
 w-r--r--    1 root    root            2079 Jan 31 09:16 01-31-20-09-23AM-dcmscript.log
-rw-r--r--    1 root    root              176 Jan 31 09:23 01-31-20-09-23AM-version.txt  
-rw-r--r--    1 root    root            12410 Jan 31 09:22 01-31-20-09-23AM-wifihealth.txt
200131-09:23:15.165431 dea: Using Direct communication
number of proUPdell:42
Upload protocol telemetry is:HTTP
before HTTP upload-----------
number of HTTP is:42
HTTPLOGUPLOADURL:https://xconf.rdkcentral.com/xconf/logupload.php
------CURL_CMD2:curl -w '%{http_code}\n' -F filename=@/rdklogs/logs//PreviousLogs/E4:5F:01:F4:EE:2B-Logs-05-03-24-12-11PM.tgz 'https://xconf.rdkcentral.com/xconf/logupload.php' --connect-timeout 30 -m 30
http_code is :200
HTTP log upload succeded!!!!!!!!!!!!!!!!!
ckp100-----------------------------------
number of proUPdell:42
Upload protocol telemetry is:HTTP
before HTTP log upload
HTTPLOGUPLOADURL:https://xconf.rdkcentral.com/xconf/logupload.php
upload log file is:05-03-24-12-11PM.tgz
------CURL_CMD2:curl -w '%{http_code}\n' -F filename=@/rdklogs/logs//PreviousLogs/E4:5F:01:F4:EE:2B-Logs-05-03-24-12-11PM.tgz 'https://xconf.rdkcentral.com/xconf/logupload.php' --connect-timeout 100 -m 100
http_code is :200
HTTP log upload succeded!!!!!!!!!!!!!!!!!
Done Uploading Logs and removing rtl_json.txt file

4. XConf-Server

   

           HTTP   
   

Configuring more entries to a profile


  • Go to Telemetry -> Permanent Profile
  • Under “Telemetry Profile entries” add your new required markers


   

  • Here, added 2 new markers for wifihealth.txt file

                1. WIFI_MAC_2 with the profile name as “Wifihealth_1” and polling frequency as 1
                2. WIFI_BYTESSENTCLIENTS with the profile name as “Wifihealth_2” and polling frequency as 1

  • Save
  • In RPI, make sure that you stopped the rdkbLogMontior service, in order to stop the log rotation process (systemctl stop rdkbLogMonitor)
  • Now restart the dcm-log service using systemctl restart dcm-log
  • With successful response, tar file and JSON file gets uploaded to the XConf-Server
  • Open the upload JSON file in the browser
  • We could see the profile entries with the number of times the specified marker has appeared in the file.

    Sample:
    
    {"searchResult":[{"Firewall":"1"},{"Wifihealth_1":"2"},{"Wifihealth_2":"2"},{"Profile":"RDKB"},{"mac":"<MAC>"},{"erouterIpv4":"192.168.30.125"},{"erouterIpv6":"null"},{"PartnerId":"RDKM"},{"AccountId":"Unknown"},{"Version":"rdkb-generic-broadband-image_default_20200224150713"},{"Time":"2020-02-26 12:23:51"}]}

    NOTE: JSON file holds only the number of times the particular marker has appeared in the respective log file. To see the complete log file, need to download the tar file uploaded and extract the files to see the complete logs or value of the parameters


Observations


  • Pushing script from XConf

Currently pushing the script from XConf to the target device is not possible due to the security issue, as it may lead to downloading the licensed file to the vulnerable device. The pushing may also lead to removal of file from rootfs, so this is not advised to do for now. Whereas, we can initiate the download from device through tftp/curl command either from the script or from console.

  • telemetry target type

 Target type is provided to handle dynamic parameters change. We have 4 target types.

        1.   # 0 if as part of normal execution

        2.   # 1 if initiated due to an XCONF update

        3.  # 2 if forced execution before log upload

        4.  # 3 if modify the cron schedule

During boot-up or service start, target type update will be initiated.

On each initiation, whenever it finds the markers in the log files, it provides the search results with marker details, when it doesn't find, it just provides the basic RDKB profile details (without markers). It all depends on the markers availability in the log files.

Currently Type 1 and 2 are supported.

  • Empty JSON message

On having the Schedule type in DCM either as "ActNow" or "CronExpression", with the expression in schedule under telemetry profile. JSON messages are seen with generic RDKB profile details on every 5 min / 10 mins (Expression: /5 * * * * or */10 * * * *). Once in a day uploads JSON with markers to the server.

This is seen due to the schedule expression. As a workaround, we can give the schedule in numbers (Example: 2 / 5 / 10)

The ticket is in place to implement the cron expression support in RPI. Once it is in place, the empty JSON messages will not be seen.


  • No labels