Introduction

Logging and Log Upload are critical aspects when considering large-scale RDK deployment. CPEs have a space constraint to store device logs for future use so either we store them for a limited number of days in CPE or in cloud servers.

Logs play a crucial role in RDK deployments by providing data for deriving field metrics and assessing the criticality of situations, such as determining whether an issue needs immediate attention or can wait until the next day. Controlled log uploads, ranging from 100% to very small numbers (2-3%) based on device maturity, help manage the volume of data. Configurable upload intervals, typically set to once a day, and the right level of abstraction ensure a balance between available storage space and the logs generated. Additionally, rotate-backup-upload policies are implemented to maintain efficient log management. Log Upload uses XConf as the primary endpoint, allowing the configuration to enable or disable log upload for a device or group of devices. During bootup, the CPE retrieves configuration information from XConf, and based on rules defined in the XConf server, it can control whether a particular device needs to perform log uploads. This setup allows for the sampling of devices from a larger population. The configuration server is responsible for retrieving and delivering settings to any client, and it supports configurable upload methods such as HTTPS, HTTP, and TFTP.

Architecture

RDK Log Upload Mechanism

draw.io Diagram
border true diagramName LogUploadProcess simpleViewer false width links auto tbstyle top lbox true diagramWidth 547 height 348 revision 1

Log uploader (/usr/bin/logupload) is responsible for gathering local logs, packaging them, and uploading them to a remote URL. A client-side scheduler would control when it executes, but it could also be activated via immediate triggers from the log upload checker or other local actions.

The Uploader is responsible for sending CPE log files to a remote file store. It performs the following:

• Responds to triggers (i.e. from cron)

• Checks local Device Settings for the URL of the file store, the logs it should send, and the repository it should use

• Gets connection credentials

• Makes a connection to the remote server and sends the files

Repository

https://github.com/rdkcentral/dcm-agent

How CPEs Upload Logs and Metrics

CPEs check with XConf for their logging and telemetry configuration through a query. Part of their query string includes several (about four) parameters of this particular CPE, such as the MAC address, model, current firmware version, and environment. The XConf server looks at these parameters and runs them through a rules engine that finds the closest targeting rule match to the CPE’s profile. Once it finds a match, it retrieves the configuration for that rule. The configuration includes things like an upload schedule, repository URL, upload protocol, path to the file server, protocol indicator, upload time smear, and more. Once the CPE gets the response, it parses it and sets the specified random schedule that delays uploading the files. At that time, the CPE packages up all of its log files, compresses the bundle and uploads the archive to the destination.

1) Boot-time initialization and configuration fetch

1. Device boots and core services start (dcmd, telemetry2.0, reboot helpers, service plugins).

2. DCM/XConf configuration is fetched using device identity attributes (for example: MAC, model, firmware, environment) by telemetry2.0 process.

   Code Block

   2025-11-28T19:06:47.125Z telemetry2_0[1177]: 251128-19:06:47.123393 [mod=T2, lvl=INFO] [tid=2488] T2: Curl Using XCONF URI : https://xconf.rdkcentral.com/loguploader/getT2Settings?estbMacAddress=D8%3a3A%3aDD%3a0D%3a16%3aB7&firmwareVersion=lib32-application-test-image-RPI4-20251030114955&model=RDKERPi4&manufacturer=RaspberryPi&partnerId=community&accountId=%221234%22&ecmMacAddress=D8%3a3A%3aDD%3a0D%3a16%3aB7&env=dev&controllerId=2504&channelMapId=2345&vodId=15660&timezone=America%2fNew_York&version=2 with Length :379 2025-11-28T19:06:47.125Z telemetry2_0[1177]: 251128-19:06:47.125776 [mod=T2, lvl=INFO] [tid=2488] doHttpGet with url https://xconf.rdkcentral.com/loguploader/getT2Settings?estbMacAddress=D8%3a3A%3aDD%3a0D%3a16%3aB7&firmwareVersion=lib32-application-test-image-RPI4-20251030114955&model=RDKERPi4&manufacturer=RaspberryPi&partnerId=community&accountId=%221234%22&ecmMacAddress=D8%3a3A%3aDD%3a0D%3a16%3aB7&env=dev&controllerId=2504&channelMapId=2345&vodId=15660&timezone=America%2fNew_York&version=2

3. XConf returns log-upload policy, typically including:

   • upload schedule (cron-like)

   • upload endpoint URL

   • upload protocol (HTTP/HTTPS)

   • behavior toggles (e.g., reboot upload, mTLS controls via RFC)

     Code Block

     root@raspberrypi4-64-rdke:~# cat /opt/.t2persistentfolder/DCMresponse.txt {"urn:settings:GroupName":"MW_RPi45","urn:settings:CheckOnReboot":true,"urn:settings:TimeZoneMode":"UTC","urn:settings:CheckSchedule:cron":"2 2 * * *", "urn:settings:CheckSchedule:DurationMinutes":0,"urn:settings:LogUploadSettings:Message":null,"urn:settings:LogUploadSettings:Name":"RPi45", "urn:settings:LogUploadSettings:NumberOfDays":0,"urn:settings:LogUploadSettings:UploadRepositoryName":"m8RPiFullTest45", "urn:settings:LogUploadSettings:UploadRepository:URL":"https://xconf.rdkcentral.com/logTelemetryServer/logportal.php", "urn:settings:LogUploadSettings:UploadRepository:uploadProtocol":"HTTP","urn:settings:LogUploadSettings:UploadOnReboot":true, "urn:settings:LogUploadSettings:UploadImmediately":false,"urn:settings:LogUploadSettings:upload":true, "urn:settings:LogUploadSettings:UploadSchedule:cron":"5 0 * * *","urn:settings:LogUploadSettings:UploadSchedule:levelone:cron":null, "urn:settings:LogUploadSettings:UploadSchedule:leveltwo:cron":null,"urn:settings:LogUploadSettings:UploadSchedule:levelthree:cron":null, "urn:settings:LogUploadSettings:UploadSchedule:TimeZoneMode":"UTC","urn:settings:LogUploadSettings:UploadSchedule:DurationMinutes":0, "urn:settings:VODSettings:Name":null,"urn:settings:VODSettings:LocationsURL":null,"urn:settings:VODSettings:SRMIPList":null, "urn:settings:TelemetryProfile":{"@type":"PermanentTelemetryProfile","id":"fcda6fb8-953d-4699-85b2-70e6f4824c9e", "telemetryProfile":[{"header":"NW_DNSMASQ_STARTED","content":"dnsmasq: starting /usr/bin/dnsmasq","type":"NetworkManager.log","pollingFrequency":"0"}, {"header":"SYST_ERR_LogUpload_Failed","content":"sysint","type":"<event>","pollingFrequency":"0"}, {"header":"Xi_wifiMAC_split","content":"sysint","type":"<event>","pollingFrequency":"0"},{"header":"Uptime","content":"Device.DeviceInfo.UpTime","type":"dataModel","pollingFrequency":"0"}], "schedule":"3","expires":0,"telemetryProfile:name":"RDKMA_Test", "uploadRepository:URL":"http://xconf.rdkcentral.com/xconf/logupload.php","uploadRepository:uploadProtocol":"HTTP"}}

4. Effective values are resolved into local runtime config used by callers/uploader - /tmp/DCMSetting.conf

Common config sources used at runtime

• XConf/DCM settings:

  • LogUploadSettings:UploadRepository:URL

  • LogUploadSettings:UploadRepository:uploadProtocol

  • LogUploadSettings:UploadOnReboot

• Local files:

  • /etc/dcm.properties or /opt/dcm.properties (e.g., LOG_SERVER)

  • /tmp/DCMSettings.conf (resolved runtime values)

2) How log upload is triggered after boot

After boot, upload may start from any of these trigger paths:

A) Scheduled trigger (DCM scheduler)

• dcmd executes scheduled job (DCM_LOGUPLOAD_SCHED).

• Calls uploadstblogs_run() with trigger type TRIGGER_SCHEDULED.

B) Maintenance window trigger

• MaintenanceManager runs maintenance task chain.

• Upload task invokes upload binary/wrapper (/usr/bin/logupload) from /lib/rdk/Start_MaintenanceTasks.sh

C) On-demand app/API trigger

• org.rdk.SystemServices.uploadLogsAsync initiates immediate upload path.

• Caller resolves protocol/URL and invokes uploadstblogs_run () with trigger type TRIGGER_MANUAL.

D) TR69hostif immediate trigger (UploadLogsNow RFC)

• RFC: Device.DeviceInfo.X_RDKCENTRAL-COM_xOpsDeviceMgmt.Logging.xOpsDMUploadLogsNow

• Invokes uploadstblogs_run()with uploadlogsnow_mode=true ON_DEMAND

E) Reboot upload trigger

• Boot sequence waits on marker synchronization (backup_logs, reboot info, telemetry markers).

• Then uploader runs with TRIGGER_REBOOT (subject to timeout guards).

3) Endpoint and how it is chosen

The final upload destination is selected from resolved config:

1. Primary: XConf/DCM upload repository values (URL, uploadProtocol)

2. Fallback/auxiliary: local properties (LOG_SERVER) depending on caller path

3. Security mode:

   • plain HTTP/HTTPS as configured

   • mTLS/xPKI path

So the uploader reaches the cloud URL from DCM/XConf policy (or caller fallback), using configured protocol/security mode.

4) How it executes on-device (what runs)

Regardless of trigger source, execution pattern is:

1. Acquire concurrency lock: /tmp/.log-upload.lock

2. Collect eligible logs

3. Apply exclusions/filters

4. Create compressed archive (tar.gz)

5. Upload archive to configured endpoint

6. Record completion:

   • maintenance path may emit completion/error signaling

   • uploadlogsnow writes /tmp/logUploadStatus.txt

5) “Downloads and executes” clarification

In this flow, the device typically downloads configuration, not a new upload executable.

• Downloaded from server: policy/config (schedule, URL, protocol, flags)

• Executed locally: existing on-device upload engine (uploadstblogs/logupload path via caller)

• In short: backend tells the device where/when/how to upload; device uses its preinstalled binaries to perform upload.

6) Quick reference table (post-boot perspective)

7) Runtime monitoring

Use:

tail MaintanenceManager invokes the wrapper sysint script /lib/rdk/StartMaintenanceTasks.sh LOGUPLOAD which invokes /usr/bin/logupload. To monitor the log upload process, you can use the command $tail -f /opt/logs/dcmscript.og.log

For UploadLogsNow status:

cat /tmp/logUploadStatus.txt

Configuring server details

The value set for Devicefor Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.Telemetry.ConfigURL determines ConfigURL determines where the device will fetch its DCM/Telemetry settings. Those settings, in turn, provide the URL for log upload and other related parameters. 

This can be retrieved from xconf server as a response to the RFC query: https://xconf.rdkcentral.com/featureControl/getSettings?<parameters>

For RDK7 release in community devices, this is set from https://github.com/rdkcentral/meta-image-assembler-rdke/blob/main/conf/community-rfc-configs.ini

 mTLS support

• RDKE LogUpload functionality expects mTLS connection by default to Download & upload logs to the server.

• Support to add mTLS to RDKM XConf server is under investigation

  Jira
  server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 11deff04-0380-3a3d-a916-0849d4e573f7 key RDKCLOUD-74

   
  

  Jira
  server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 11deff04-0380-3a3d-a916-0849d4e573f7 key RDKECOREMW-870

  
  

• Until then, logUpload functionality is broken in RDKE builds

Configuration Details

⁠To know how XConf server-side configuration and feature validation are done please refer to User guide for configuration and feature validation .

Validation details