View Source

Overview

Maintenance Manager is one of the WPEFramework Thunder Plugin that manages the daily maintenance activities on RDK device.

Source Code

RDKV	MaintenanceManager (rdkservices main)
RDKE	MaintenanceManager (entservices-softwareupdate main)

Terminology

Daily Maintenance	Set of Activities (or tasks) that needs to be executed daily without impacting the UX. Executed when the user is not actively using the CPE
Maintenance Activity	Activity (or Task) that is a part of Daily Maintenance. They are considered critical and wake up the Device from DEEPSLEEP power state to be executed
DCM	Device Configuration Manager, it fetches and updates configuration for Device
RFC	RDK Runtime Feature Control, is used for remotely controlling CPE software features on RDK platforms. Release Management uses RFC for staged rollout of new features.
Log Upload	Uploads all the logs at the time specified in `DCMSettings.conf` or when the size reaches predefined limit
FW Download (swupdate/ rdkfwupgrader)	Checkis if there is a new firmware update is required and performing the update on the device (if needed)

Types of Maintenance

Unsolicited Maintenance (Maintenance on Boot)
Solicited Maintenance (Scheduled Maintenance)

Unsolicited Maintenance

Also known as Maintenance on Bootup. RDK automatically performs a Maintenance as soon as the device boots up. Unsolicited maintenance occurs on each and every bootup. Device will look for critical updates on boot and performs maintenance of critical activities without application triggering maintenance. The isRebootPending will be set to false unless there is an RFC/FWUpdate needed. Below are the tasks executed as part of unsolicited maintenance.

Solicited Maintenance

Also known as Scheduled Maintenance/ Daily Maintenance. This is executed daily based on Maintenance Start Time triggered by Application. The isRebootPending is set to true irrespective of XConf configuration. Below are the tasks executed as part of Solicited Maintenance

Note: Both Unsolicited and Solicited maintenance cannot be triggered at the same time. If a Solicited Maintenance is started while an Unsolicited Maintenance is in progress, then the Solicited Maintenance will be notified and postponed to the next day.

Functionality Observations

Maintenance Manager notifies the upper layer components when the next daily maintenance activity is required to take place (based on the start time scheduled by XConf for the specific device)
Maintenance Manager starts the daily maintenance upon request from the upper layer components
Capable of running the maintenance activity in the background or aborting or skipping a daily activity upon request of the upper layers
Capable for running the list of the activity in time order, one at a time. If multiple activities have the same start time, it will run one after the other one, not concurrently.
A default Maintenance Start Time will be maintained by Maintenance Manager on first initialization, with initial status as MAINTENANCE_IDLE
Maintenance Start Time will be updated by the Maintenance Plugin by calling getMaintenanceStartTime jsonPRC call refer JsonPRC section for more info
Maintenance Manager Plugin, interacts with Network Plugin polling to get the Network status (if its connected to active internet or not). The polling happens for 4 times with a retry delay of 30 seconds.
On an un-successful polling session, Maintenance Manager will subscribe to onInternetStatusChange event and set the Maintenance Status to MAINTENANCE_ERROR, on successful event reception, MaintenanceMaintenance will trigger Critical Tasks ()

WhoAmI

WhoAmI is used by device to determine the operating context (partner, product/experience, Regional configService) and load the right firmware so that we can activate and use the service that user signed up for. It happens only in Unsolicited Maintenance. It is not defaulted in RDKE yet (as of 25Q2). WhoAmI has dependency on components such as AuthService which has to be refactored before making it suitable for community builds usage.

Maintenance Activity/ Task

Activity/ Task Name	RDKV		RDKE
Activity/ Task Name	Wrapper	Task	Wrapper	Task
RFC	`/lib/rdk/Start_RFC.sh`	`/lib/rdk/RFCbase.sh`	`/lib/rdk/StartMaintenanceTasks.sh RFC`	ENABLE_RFC_MANAGER flag Enabled
				`/usr/bin/rfcMgr`
				ENABLE_RFC_MANAGER flag Disbled
				`/lib/rdk/RFCbase.sh`

SWUPDATE	`/lib/rdk/swupdate_utility.sh`	FWDOWNLOAD RFC Enabled	`/lib/rdk/StartMaintenanceTasks.sh SWUPDATE`	`/usr/bin/rdkvfwupgrader 0 1`
		`/usr/bin/rdkvfwupgrader 0 1`
		FWDOWNLOAD RFC Disabled
		`/lib/rdk/deviceInitiatedFWDnld.sh`

LOGUPLOAD	`/lib/rdk/Start_uploadSTBLogs.sh`	`/lib/rdk/uploadSTBLogs.sh`	`/lib/rdk/StartMaintenanceTasks.sh LOGUPLOAD`	`/lib/rdk/uploadSTBLogs.sh`

XConf Settings

The DCMScript.sh (in RDKV) or dcmd (in RDKE) will parse the /tmp/DCMSettings.conf file to get XConf Data fetched by Telemetry. Data likeurn:settings:CheckSchedule:cron is used to get start_hr and start_min values and urn:settings:TimeZoneMode is used to get the tz_mode data. Device is expected to receive the time based on the time zone confiigured in the device. This parsed data is then stored to /opt/rdk_maintenance.conf file by DCM. The data can be used to calculate the Maintenance Start Time for the next scheduled Maintenance (SOLICITED). The Maintenance Start Time is calculated using /lib/rdk/getMaintenanceStartTime.sh (in RDKV) or using CalculateStartTime() API in MaintenanceManager internally.

Architecture Overview (HLA)

Activity Diagram

@startuml
start
:Initialize();
:InitializeIARM();
:MAINTENANCE_IDLE;
if (isDeviceConnectedToInternet?) then (yes)
else (Not Connected)
    repeat
        :Sleep 30s;
    repeat while (Retry Count < 4)
endif
if (Network Plugin Active?) then (yes)
    :Subscribe to Internet Status Event;
else (no)
    :Internet Status Event Subscription Failed;
endif

:task_execution_thread;
if (isDeviceOnline?) then (yes)
else (no)
    :MAINTENANCE_ERROR;
    stop
endif

group Task Execution (RFC/ SWUPDATE/ LOGUPLOAD)
    :Start RFC/ SWUPDATE/ LOGUPLOAD Task;
    :Start Timer;
    repeat
        :Execute system() for RFC/ SWUPDATE/ LOGUPLOAD Task;
        if (RFC/ SWUPDATE/ LOGUPLOAD success? before timer ends) then (yes)
            :Stop Timer;
            :MAINTENANCE_<RFC/FWDOWNLOAD/LOGUPLOAD>_COMPLETE;
        else (no, fail or timeout)
            if (System Call Failed) then (yes)
                if (Retry Count < TASK_RETRY_COUNT) then (yes)
                    :Retry after TASK_RETRY_DELAY;
                    :Decrement Retry Count;
                    :Restart Timer;
                else (no more retries)
                    :MAINTENANCE_<RFC/FWDOWNLOAD/LOGUPLOAD>_ERROR;
                    :Stop Timer;
                    break
                endif
            else (RFC/ SWUPDATE/ LOGUPLOAD Task Timed Out)
                :MAINTENANCE_<RFC/FWDOWNLOAD/LOGUPLOAD>_ERROR;
                :Stop Timer;
                break
            endif
        endif
    repeat while (Retry Count > 0)
end group

if (Any at least One Task of RFC/ FWDOWNLOAD/ LOGUPLOAD failed) then (yes)
    :MAINTENANCE_ERROR;
else (no)
    :MAINTENANCE_COMPLETE;
endif
:Task Execution Completed;
:Thread Join;
:Deinitialize();
if (Timer Exists) then (yes)
    :Delete Timer;
    if (Signal Registered) then (yes)
        :Unregister Signal;
    endif
endif
stop
@enduml

High Level Sequence Diagram

@startuml
participant "Maintenance Manager" as mm
participant "Network Manager" as nm
participant "WPEFramework" as wpe
participant "RFC service" as rfc
participant "swupdate" as swu
participant "Log Upload" as lu

wpe -> mm: Activate Maintenance Manager Plugin
note over mm: MAINTENANCE_IDLE

== Network check ==
mm -> nm: isDeviceConnectedToInternet?
alt Success case
    nm --> mm: Connected
else Failure case
    nm --> mm: Not Connected
    loop 4 times max
        mm -> mm: Sleep 30s
        mm -> nm: isDeviceConnectedToInternet?
    end
end

alt No internet?
    mm --> mm: MAINTENANCE_ERROR
else Internet available
    mm -> mm: proceed to execution thread
end

note over mm: MAINTENANCE_STARTED

== Maintenance tasks ==
mm -> rfc: Start RFC
rfc --> mm: RFC Complete/ Error

mm -> swu: Start SWU
swu --> mm: SWU Complete/ Error

mm -> lu: Start Log upload
lu --> mm: Log upload Complete/ Error

note over mm: MAINTENANCE_COMPLETE/ MAINTENANCE_ERROR
@enduml

Maintenance Status Notifications

MAINTENANCE_IDLE - Initial state
MAINTENANCE_STARTED - Sent immediately on maintenance startup either scheduled or onboot
MAINTENANCE_ERROR - Sent after receiving error notification while executing any of the maintenance activities
MAINTENANCE_COMPLETE - Sent after receiving *_COMPLETE notification from all critical maintenance tasks.
MAINTENANCE_INCOMPLETE - Sent whenever Maintenance service doesn't execute any of the tasks. MAINTENANCE_ERROR is returned even if one task returns error while others weren't executed.

Distros and RFCs used in Maintenance Manager

Maintenance Manager feature is controlled by distro "enable_maintenance_manager"
stopMaintenance support is controlled by RFC "Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.StopMaintenance.Enable" which is enabled by default
whoAmI feature is controlled by both distros "whoami_enabled" and "sec_manager_whoami_enabled" and a compile time flag "ENABLE_WHOAMI".

Notable features

Network check - Before triggering maintenance tasks, first it will check for internet connectivity in the device. If the device is connected with internet, it will proceed to execute maintenance tasks otherwise it will wait for 2 minutes and if still it has no internet connection, it will exit from maintenance and status will be MAINTENANCE_ERROR
Maintenance Manager will subscribe for network event if device is not able to acquire network in 2mins before exiting from maintenance activities. Once device got internet, Maintenance Manager will be notified, and it will trigger RFC and xconfImageCheck.sh
StopMaintenance is allowed only when the maintenance is in progress and it will set the status as MAINTENANCE_ERROR.
StartMaintenance is allowed only when the maintenance is already not in progress.
Who Am I integration changes are done in Maintenance Manager to acquire the device Initialization context

Logs

Maintenance Manager logs can be found in wpeframework log.

Grep string: grep -inr "MaintenanceManager.cpp" /opt/logs/maintenance.log*