View Source

WIP

Intro

As a Field Triage Engineer and Developer responsible for debugging field issues, having the ability to collect diagnostic/ debug data from the device without requiring SSH access would be beneficial. The Triage Engineer can use WebPA communication to initiate the data reporting process and retrieve the report from the S3 log service

Architecture

Please refere https://github.com/rdkcentral/remote_debugger/tree/develop

Use case diagram

Samples

Upload scripts location

/lib/rdk/uploadRRDLogs.sh
/rdklogger/uploadRDKBLogs.sh

Report generation location
- /tmp/rrd
Log files
- /rdklogs/logs/remote-debugger.log.0

static json file location

Path in CPE
   /etc/rrd/remote_debugger.json
Path in code base
   https://github.com/rdkcentral/remote_debugger/blob/develop/remote_debugger.json

Datamodel command

dmcli eRT setv Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType string "Device.Uptime"

rdm agent download url path

/tmp/.xconfssrdownloadurl

Removed existing rdm in rdkb builds

meta-cmf/recipes-core/packagegroups/packagegroup-rdk-ccsp-broadband.bbappend # Comment out or remove the line: RDEPENDS_packagegroup-rdk-ccsp-broadband_append = " rdm"

Distro

rrd to be enabled

RFC

DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.Enable

Enable service from yocto

remote-debugger.service

Image install

Install remote debugger & rdm-agent in monolithic image

Code base

remote debugger
- Source repo
  - https://github.com/rdkcentral/remote_debugger/tree/develop
- meta layer
  - https://github.com/rdkcentral/meta-rdk/blob/develop/recipes-extended/remotedebugger/remotedebugger.bb
rdm agent
- Source repo
  - https://github.com/rdkcentral/rdm-agent/tree/develop
- meta layer
  - https://github.com/rdkcentral/meta-rdk/blob/develop/recipes-support/rdmagent/rdmagent.bb
- Dependencies
- Additional changes
  - rdm_downloadutils.c #ifdef IARMBUS_SUPPORT #include "rdmMgr.h" #else #define RDM_PKG_INVALID_INPUT 11 #endif

Customizations to support cash report uploading

Note: This is only for testing in BPI R4.For production you need to follow the actual signing process.
https://github.com/cpokuru/remote_debugger/commit/d2f717039216809a7e137b0a0ec94e783248c1ba

Dependencies removal

remove utopia dependencies from remote debugger service file

Roles and responsibilities of ISP

Develop your own triaging tool on top of webpa/usp/tr69

Use case for production

static profile using remote debugger (Works)
dynamic profile using remote debugger and rdm

Test logs for static profile

1. Set up the upload server(Ex:Apache server) on the local Machine for BPIR4 to upload the debug reports

sudo apt install apache2 
sudo a2enmod dav dav_fs

# have the below content in /etc/apache2/sites-available/upload.conf
<VirtualHost *:8080>
    ServerAdmin admin@example.com
    DocumentRoot /srv/upload

    <Directory /srv/upload>
        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
        DAV On
    </Directory>

</VirtualHost>

sudo a2ensite upload.conf
sudo mkdir /srv/upload && sudo chown www-data:www-data /srv/upload

# add the ports to listen  in /etc/apache2/ports.conf
Listen 0.0.0.0:80
Listen 0.0.0.0:8080
<IfModule ssl_module>
	Listen 443
</IfModule>
<IfModule mod_gnutls.c>
	Listen 443
</IfModule>

#Start Apache2 server 
 sudo systemctl enable apache2 
 sudo systemctl start apache2
 chtslbcp515@chtslbcp515-Latitude-3400:~$ sudo netstat -lputnu |     grep apa
 tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1038/apache2        
 tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN              1038/apache2

Testing logs

~# dmcli eRT setv Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType string "Device.Uptime"
CR component name is: eRT.com.cisco.spvtg.ccsp.CR
subsystem_prefix eRT.
Execution succeed.

root@Filogic-GW:~# cat /rdklogs/logs/remote-debugger.log.0 
220428-17:44:16.829555 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6844] [isRRDEnabled:112]:RFC is enabled, starting remote-debugger
220428-17:44:16.889345 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [RRDEventThreadFunc:44]:Waiting for for TR69/RBUS Events... 
260122-19:38:52.063885 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6889] [_remoteDebuggerEventHandler:322]: Received event for RRD_SET_ISSUE_EVENT Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType 
260122-19:38:52.063962 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6889] [pushIssueTypesToMsgQueue:395]: SUCCESS: Message sending Done, ID=0 MSG=Device.Uptime Size=13 Type=1 AppendMode=0! 
260122-19:38:52.064019 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [getIssueInfo:191]:  Received Main Node= Device, SubNode= Uptime
260122-19:38:52.064049 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [processIssueTypeInStaticProfile:296]: Checking Static Profile... 
260122-19:38:52.067965 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [processIssueTypeInStaticProfile:312]: Issue Data Node: Device and Sub-Node: Uptime found in Static JSON File /etc/rrd/remote_debugger.json... 
260122-19:38:52.068239 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [executeCommands:350]: No MACRO found, proceeding with commands:"uptime" 
260122-19:38:52.068315 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [executeCommands:376]: Executing following commands using systemd-run:
 ""uptime""
260122-19:38:52.069350 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [executeCommands:390]: Starting remote_debugger_Device.Uptime service success...
260122-19:38:52.168889 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [executeCommands:396]: Using journalctl to log command output...
260122-19:38:52.169811 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [executeCommands:404]: journalctl remote_debugger_Device.Uptime service success...
260122-19:38:52.176470 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [executeCommands:414]: Sleeping with timeout 10 sec before stopping service...
260122-19:39:02.176595 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [executeCommands:418]: Stopping remote_debugger_Device.Uptime service...
260122-19:39:02.189241 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [uploadDebugoutput:46]: Starting Upload Debug output Script: /lib/rdk/uploadRRDLogs.sh... 
2026-01-22T19:39:02.339Z: /lib/rdk/uploadRRDLogs.sh: Executing uploadRRDLogs.sh script to upload Debug info of ISSUETYPE=DEVICE_UPTIME
2026-01-22T19:39:02.345Z: /lib/rdk/uploadRRDLogs.sh: Using upstream RRD URL: http://10.2.166.120:8080/upload
2026-01-22T19:39:02.350Z: /lib/rdk/uploadRRDLogs.sh: Checking /tmp/rrd/Device-DebugReport-2026-01-22-19-38-52 size and contents
2026-01-22T19:39:02.357Z: /lib/rdk/uploadRRDLogs.sh: Creating 020100B65218_DEVICE_UPTIME_2026-01-22-19-39-02PM_RRD_DEBUG_LOGS.tgz tarfile from Debug Commands output
2026-01-22T19:39:02.367Z: /lib/rdk/uploadRRDLogs.sh: Invoking uploadRDKBLogs.sh script to upload 020100B65218_DEVICE_UPTIME_2026-01-22-19-39-02PM_RRD_DEBUG_LOGS.tgz file
2026-01-22T19:39:02.941Z: /lib/rdk/uploadRRDLogs.sh: RRD DEVICE_UPTIME Debug Information Report upload Success
2026-01-22T19:39:02.946Z: /lib/rdk/uploadRRDLogs.sh: Removing uploaded report 020100B65218_DEVICE_UPTIME_2026-01-22-19-39-02PM_RRD_DEBUG_LOGS.tgz
260122-19:39:02.949497 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [checkIssueNodeInfo:552]: RRD Upload Script Execution Success...
260122-19:39:02.949609 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=6937] [RRDEventThreadFunc:44]:Waiting for for TR69/RBUS Events... 
root@Filogic-GW:~#

Test logs for dynamic profile

1. Install the Download Server ( Ex: Apache)

2 . Host the Dynamic profile in Download server path ( in below format)

chtslbcp515@chtslbcp515-Latitude-3400:/var/www/html$ sudo tar -tf RDK-RRD-Device_1.0-signed.tar
RDK-RRD-Device.tar

chtslbcp515@chtslbcp515-Latitude-3400:/var/www/html$ sudo tar -tf RDK-RRD-Device.tar
etc/
etc/rrd/
etc/rrd/remote_debugger.json

chtslbcp515@chtslbcp515-Latitude-3400:/var/www/html$ cat etc/rrd/remote_debugger.json
{
    
    "Device" : {
        "wifi" : {
            "Commands": "iw dev;uname -r",
            "Timeout" : 10
        }
    }
}

3 Configure the Download Server URL on CPE

root@Filogic-GW:~# dmcli eRT setv Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.CDLDM.CDLModuleUrl  string http://192.168.2.135
CR component name is: eRT.com.cisco.spvtg.ccsp.CR
subsystem_prefix eRT.
Execution succeed.

4 Test the dynamic profile

root@Filogic-GW:~# dmcli eRT setv Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType string "Device.wifi"
CR component name is: eRT.com.cisco.spvtg.ccsp.CR
subsystem_prefix eRT.
Execution succeed.

# Remote-debugger logs
root@Filogic-GW:~# cat /rdklogs/logs/remote-debugger.log.0 
root@Filogic-GW:/tmp/RDK-RRD-Device/etc/rrd# cat /rdklogs/logs/remote-debugger.log.0 
260217-11:51:55.933866 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13750] [_remoteDebuggerEventHandler:322]: Received event for RRD_SET_ISSUE_EVENT Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType 
260217-11:51:55.933927 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13750] [pushIssueTypesToMsgQueue:395]: SUCCESS: Message sending Done, ID=0 MSG=Device.wifi Size=11 Type=1 AppendMode=0! 
260217-11:51:55.933978 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13755] [getIssueInfo:191]:  Received Main Node= Device, SubNode= wifi
260217-11:51:55.934010 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13755] [processIssueTypeInStaticProfile:296]: Checking Static Profile... 
260217-11:51:55.934210 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13755] [processIssueTypeInInstalledPackage:480]: Checking Dynamic Profile... 
260217-11:51:55.934255 [mod=REMOTEDEBUGGER, lvl=ERROR] [tid=13755] [readJsonFile:85]: Json File /tmp/RDK-RRD-Device/etc/rrd/remote_debugger.json Read Failed!!! 
260217-11:51:55.934272 [mod=REMOTEDEBUGGER, lvl=ERROR] [tid=13755] [readAndParseJSON:120]: Reading json file failed, Skipping Parse!!
260217-11:51:55.934287 [mod=REMOTEDEBUGGER, lvl=ERROR] [tid=13755] [processIssueTypeInInstalledPackage:484]: Dynamic Profile Parse/Read failed... /tmp/RDK-RRD-Device/etc/rrd/remote_debugger.json
260217-11:51:55.934301 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13755] [processIssueTypeInInstalledPackage:485]: Going to RDM Request... 
260217-11:51:55.941092 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13755] [append_item:113]: Append Item with PkgData: RDK-RRD-Device and issue Type: Device.wifi to Cache 
260217-11:51:55.941185 [mod=REMOTEDEBUGGER, lvl=INFO] [tid=13755] [RRDEventThreadFunc:44]:Waiting for for TR69/RBUS Events... 

# Rdm-agent logs
root@Filogic-GW:~# cat /rdklogs/logs/rdm-status.log.0 
260217-11:51:55.959722 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] main[349]: Install App from custom path: RDK-RRD-Device:1.0
260217-11:51:55.959843 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] main[359]: pkg_name_signed = RDK-RRD-Device_1.0-signed.tar
260217-11:51:55.960485 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmRbusGetRfc[139]: RDM URL: http://192.168.2.135
260217-11:51:55.960588 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlUpdateURL[105]: Download URL is not available in /path/to/my/url/file
260217-11:51:55.960605 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlUpdateURL[106]: Using RDM Default URL to download from RFC: http://192.168.2.135
260217-11:51:55.960623 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[943]: App Name               : RDK-RRD-Device
260217-11:51:55.960637 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[944]: PKG Name               : RDK-RRD-Device_1.0-signed.tar
260217-11:51:55.960652 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[945]: PKG Type               : 
260217-11:51:55.960665 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[946]: PKG Ver                : 1.0
260217-11:51:55.960679 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[947]: App Size               : 
260217-11:51:55.960693 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[948]: App Size in kb         : 0
260217-11:51:55.960707 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[949]: App Size in mb         : 0
260217-11:51:55.960721 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[950]: App Home Path          : /tmp/RDK-RRD-Device
260217-11:51:55.960734 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[951]: App mnt  Path          : /tmp
260217-11:51:55.960749 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[952]: App download Home      : /tmp/rdm/downloads
260217-11:51:55.960763 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[953]: App download Path      : /tmp/rdm/downloads/RDK-RRD-Device
260217-11:51:55.960777 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[954]: App download File Path : 
260217-11:51:55.960790 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[955]: App download URL       : http://192.168.2.135/RDK-RRD-Device_1.0-signed.tar
260217-11:51:55.960804 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[956]: App info Path          : /nvram/persistent/rdmDownloadInfo.txt
260217-11:51:55.960818 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[957]: Download On Demond     : 0
260217-11:51:55.960832 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[958]: Download Method Control: 
260217-11:51:55.960846 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmPrintAppDetails[959]: App Versioned          : 0
260217-11:51:55.960862 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadApp[112]: Downloading RDK-RRD-Device...
260217-11:51:55.960882 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadApp[130]: HOME PATH for APP = /tmp/RDK-RRD-Device
260217-11:51:55.961058 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlGetCert[241]: RDM download certificate success. cert=MyCertName, cert type=MyCertType
260217-11:51:55.962251 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlDirect[292]: Downloading The Package /tmp/rdm/downloads/RDK-RRD-Device/RDK-RRD-Device_1.0-signed.tar 
260217-11:51:55.977771 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlDirect[297]: curl_ret_code:0 httpCode:200
260217-11:51:55.982015 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlExtract[74]: Extracting PKG File : /tmp/rdm/downloads/RDK-RRD-Device/RDK-RRD-Device_1.0-signed.tar to dwnd path /tmp/rdm/downloads/RDK-RRD-Device
260217-11:51:55.986036 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlExtract[80]: Extraction of /tmp/rdm/downloads/RDK-RRD-Device/RDK-RRD-Device_1.0-signed.tar is Successful
260217-11:51:55.986103 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlExtract[95]: Extracting /tmp/rdm/downloads/RDK-RRD-Device/RDK-RRD-Device_1.0.tar to /tmp/rdm/downloads/RDK-RRD-Device
260217-11:51:55.989862 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlExtract[104]: Extraction of /tmp/rdm/downloads/RDK-RRD-Device/RDK-RRD-Device_1.0.tar is Successful 
260217-11:51:55.993789 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadMgr[298]: Download and Extraction Completed
260217-11:51:55.993829 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadMgr[304]: Signature validation for Versioned APP
260217-11:51:55.993877 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadMgr[318]: RDM package download success: RDK-RRD-Device_1.0-signed.tar 
260217-11:51:55.993895 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDwnlRunPostScripts[442]: Running Scripts after RDM Download
260217-11:51:55.993926 [mod=RDMAGENT, lvl=ERROR] [tid=14419] [RDMAGENT] rdmDwnlRunPostScripts[451]: Error opening directory: /tmp/RDK-RRD-Device/etc/rdm/post-services/
260217-11:51:55.993943 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadApp[159]: rdmDownloadMgr is sucess
260217-11:51:55.994158 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadApp[182]: Package info path /nvram/persistent exists
260217-11:51:55.994222 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] rdmDownloadApp[194]: Opened /nvram/persistent/rdmDownloadInfo.txt file successfully
260217-11:51:55.994419 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] main[369]: Download completed for App RDK-RRD-Device with status=0
260217-11:51:55.994437 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] main[377]: App download success, sending status as 0
260217-11:51:55.994452 [mod=RDMAGENT, lvl=INFO] [tid=14419] [RDMAGENT] main[380]: Post Installation Successful for RDK-RRD-Device


#Check whether dynamic json is created in below path /tmp/RDK-RRD-Device/etc/rrd/remote_debugger.json
root@Filogic-GW:~# cat /tmp/RDK-RRD-Device/etc/rrd/remote_debugger.json
{
    
    "Device" : {
        "wifi" : {
            "Commands": "iw dev;uname -r",
            "Timeout" : 10
        }
    }
}