RDK Maintenance Manager

Overview

Maintenance Manager is one of the WPEFramework Thunder Plugin that manages the daily maintenance activities on RDK device.

Source Code

RDKV	MaintenanceManager (rdkservices main)
RDKE	MaintenanceManager (entservices-softwareupdate main)

Terminology

Daily Maintenance	Set of Activities (or tasks) that needs to be executed daily without impacting the UX. Executed when the user is not actively using the CPE
Maintenance Activity	Activity (or Task) that is a part of Daily Maintenance. They are considered critical and wake up the Device from DEEPSLEEP power state to be executed
DCM	Device Configuration Manager, it fetches and updates configuration for Device
RFC	RDK Runtime Feature Control, is used for remotely controlling CPE software features on RDK platforms. Release Management uses RFC for staged rollout of new features.
Log Upload	Uploads all the logs at the time specified in `DCMSettings.conf` or when the size reaches predefined limit
FW Download (swupdate/ rdkfwupgrader)	Checkis if there is a new firmware update is required and performing the update on the device (if needed)

Types of Maintenance

Unsolicited Maintenance (Maintenance on Boot)
Solicited Maintenance (Scheduled Maintenance)

Unsolicited Maintenance

Also known as Maintenance on Bootup. RDK automatically performs a Maintenance as soon as the device boots up. Unsolicited maintenance occurs on each and every bootup. Device will look for critical updates on boot and performs maintenance of critical activities without application triggering maintenance. The isRebootPending will be set to false unless there is an RFC/FWUpdate needed.

Solicited Maintenance

Also known as Scheduled Maintenance/ Daily Maintenance. This is executed daily based on Maintenance Start Time triggered by Application. The isRebootPending is set to true irrespective of XConf configuration.

Note: Both Unsolicited and Solicited maintenance cannot be triggered at the same time. If a Solicited Maintenance is started while an Unsolicited Maintenance is in progress, then the Solicited Maintenance will be notified and postponed to the next day.

Functionality Observations

Maintenance Manager notifies the upper layer components when the next daily maintenance activity is required to take place (based on the start time scheduled by XConf for the specific device)
Maintenance Manager starts the daily maintenance upon request from the upper layer components
Capable of running the maintenance activity in the background or aborting or skipping a daily activity upon request of the upper layers
Capable for running the list of the activity in time order, one at a time. If multiple activities have the same start time, it will run one after the other one, not concurrently.
A default Maintenance Start Time will be maintained by Maintenance Manager on first initialization, with initial status as MAINTENANCE_IDLE
Maintenance Start Time will be updated by the Maintenance Plugin by calling getMaintenanceStartTime jsonPRC call refer JsonPRC section for more info
Maintenance Manager Plugin, interacts with Network Plugin polling to get the Network status (if its connected to active internet or not). The polling happens for 4 times with a retry delay of 30 seconds.
On an un-successful polling session, Maintenance Manager will subscribe to onInternetStatusChange event and set the Maintenance Status to MAINTENANCE_ERROR, on successful event reception, MaintenanceMaintenance will trigger Critical Tasks ()

WhoAmI

WhoAmI is used by device to determine the operating context (partner, product/experience, Regional configService) and load the right firmware so that we can activate and use the service that user signed up for. It happens only in Unsolicited Maintenance. It is not defaulted in RDKE yet (as of 25Q2). WhoAmI has dependency on components such as AuthService which has to be refactored before making it suitable for community builds usage.

Maintenance Activity/ Task

Activity/ Task Name	RDKV		RDKE
Activity/ Task Name	Wrapper	Task	Wrapper	Task
RFC	`/lib/rdk/Start_RFC.sh`	`/lib/rdk/RFCbase.sh`	`/lib/rdk/StartMaintenanceTasks.sh RFC`	ENABLE_RFC_MANAGER flag Enabled
				`/usr/bin/rfcMgr`
				ENABLE_RFC_MANAGER flag Disbled
				`/lib/rdk/RFCbase.sh`

SWUPDATE	`/lib/rdk/swupdate_utility.sh`	FWDOWNLOAD RFC Enabled	`/lib/rdk/StartMaintenanceTasks.sh SWUPDATE`	`/usr/bin/rdkvfwupgrader 0 1`
		`/usr/bin/rdkvfwupgrader 0 1`
		FWDOWNLOAD RFC Disabled
		`/lib/rdk/deviceInitiatedFWDnld.sh`

LOGUPLOAD	`/lib/rdk/Start_uploadSTBLogs.sh`	`/lib/rdk/uploadSTBLogs.sh`	`/lib/rdk/StartMaintenanceTasks.sh LOGUPLOAD`	`/lib/rdk/uploadSTBLogs.sh`

XConf Settings

The DCMScript.sh (in RDKV) or dcmd (in RDKE) will parse the /tmp/DCMSettings.conf file to get XConf Data fetched by Telemetry. Data likeurn:settings:CheckSchedule:cron is used to get start_hr and start_min values and urn:settings:TimeZoneMode is used to get the tz_mode data. Device is expected to receive the time based on the time zone confiigured in the device. This parsed data is then stored to /opt/rdk_maintenance.conf file by DCM. The data can be used to calculate the Maintenance Start Time for the next scheduled Maintenance (SOLICITED). The Maintenance Start Time is calculated using /lib/rdk/getMaintenanceStartTime.sh (in RDKV) or using CalculateStartTime() API in MaintenanceManager internally.

Architecture Overview (HLA)

Activity Diagram

High Level Sequence Diagram

Maintenance Status Notifications

MAINTENANCE_IDLE - Initial state
MAINTENANCE_STARTED - Sent immediately on maintenance startup either scheduled or onboot
MAINTENANCE_ERROR - Sent after receiving error notification while executing any of the maintenance activities
MAINTENANCE_COMPLETE - Sent after receiving *_COMPLETE notification from all critical maintenance tasks.
MAINTENANCE_INCOMPLETE - Sent whenever Maintenance service doesn't execute any of the tasks. MAINTENANCE_ERROR is returned even if one task returns error while others weren't executed.

Distros and RFCs used in Maintenance Manager

Maintenance Manager feature is controlled by distro "enable_maintenance_manager"
stopMaintenance support is controlled by RFC "Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.StopMaintenance.Enable" which is enabled by default
whoAmI feature is controlled by both distros "whoami_enabled" and "sec_manager_whoami_enabled" and a compile time flag "ENABLE_WHOAMI".

Notable features

Network check - Before triggering maintenance tasks, first it will check for internet connectivity in the device. If the device is connected with internet, it will proceed to execute maintenance tasks otherwise it will wait for 2 minutes and if still it has no internet connection, it will exit from maintenance and status will be MAINTENANCE_ERROR
Maintenance Manager will subscribe for network event if device is not able to acquire network in 2mins before exiting from maintenance activities. Once device got internet, Maintenance Manager will be notified, and it will trigger RFC and xconfImageCheck.sh
StopMaintenance is allowed only when the maintenance is in progress and it will set the status as MAINTENANCE_ERROR.
StartMaintenance is allowed only when the maintenance is already not in progress.
Who Am I integration changes are done in Maintenance Manager to acquire the device Initialization context

Logs

Maintenance Manager logs can be found in wpeframework log.

Grep string: grep -inr "MaintenanceManager.cpp" /opt/logs/maintenance.log*

Child Pages

Methods, RPCs. Events and Functions

Methods

subscribeForInternetStatusEvent

Description	A MaintenanceManager method that does a proper subscription to Network state changes coming from Network Plugin, we use internetStatusChangeEventHandler() Event handler
Prototype	bool MaintenanceManager::subscribeForInternetStatusEvent(string event)
Input Args	string event
Return	true	returned if Thunder Plaugin Handle is obtained and Network Plugin Event is subscribed with ERROR_NONE
Return	false	returned if there is a failure to get Thunder Plugin Handle or if the Event Subscription fails (with Status not ERROR_NONE)

task_execution_thread

Description	This is triggered by the MaintenanceonBootup and startMaintenance to execute all maintenance tasks in a specific order with all preconditions asserted like Network Status, reboot flags, critical update, etc
Prototype	void MaintenanceManager::task_execution_thread()
Input Args	None
Return	void

getThunderPluginHandle

Description	A Plugin method to get a plugin handler for a Plugin with a Callsign. It creates a Toke on runtime to get the Plugin handle
Prototype	WPEFramework::JSONRPC::LinkType<WPEFramework::Core::JSON::IElement>* MaintenanceManager::getThunderPluginHandle(const char* callsign)
Input Args	const char* callsign
Return	WPEFramework::JSONRPC::LinkType<WPEFramework::Core::JSON::IElement>*

internetStatusChangeEventHandler

Description	This is an event handler method that is triggered subscribeForInternetStatusEvent() when there is a network state change from the Network Plugin.
Prototype	void MaintenanceManager::internetStatusChangeEventHandler(const JsonObject& parameters)
Input Args	const JsonObject& parameters
Return	void

startCriticalTasks

Description	A Maintenance Manager method to trigger Critical tasks via 3 Scripts StartDCM_maintenance.sh RFCbase.sh xconfImageCheck.sh These are triggered in case if the network is reconnected after there is no active network connection available during an Unsolicited Maintenance and network retry fails with Maintenance Status with MAINTENANCE_ERROR
Prototype	void MaintenanceManager::startCriticalTasks
Input Args	NA
Return	void

checkNetwork

Description	This method checks if Network Plugin is active and subscribes to Network state change; This uses runtime token creation to communicate with Thunder
Prototype	bool MaintenanceManager::checkNetwork()
Input Args	None
Return	false	returns when Plugin is not active or the network plugin connection check fails
Return	true	other cases

isDeviceOnline

Description	This utility method brings in a network retry mechanism with MAX_NETWORK_RETRIES retries (4) for a retry delay NETWORK_RETRY_INTERVAL (30 seconds). This uses checkNetwork() to get the state of network plugin state change.
Prototype	bool MaintenanceManager::isDeviceOnline()
Input Args	None
Return	true	returns if any try or retry of network check gives true
Return	false	returns if the network is not available even after 4 retries

Initialize

Description	This method initializes Maintenance Manager service and instance and also calls subscribeToDeviceInitializationEvent() in WhoAmI to subscribe to onDeviceInitializationContextUpdate Event from SecManager. This also initializes IARM thru InitializeIARM() and returns an empty string
Prototype	const string MaintenanceManager::Initialize(PluginHost::IShell* service)
Input Args	PluginHost::IShell* service
Return	string

Deinitialize

Description	This method Deinitializes the MaintenanceManager services releases it triggers stopMaintenanceTasks() to gracefully exit maintenance and deinitializes IARM via DeinitializeIARM()
Prototype	void MaintenanceManager::Deinitialize(PluginHost::IShell* service)
Input Args	PluginHost::IShell* service
Return	void

InitializeIARM

Description	This method checks if IARM is Initialized and registers event handlers for Maintenance Event Update Event and DCM Start Time Event
Prototype	void MaintenanceManager::InitializeIARM()
Input Args	None
Return	void

maintenanceManagerOnBootup

Description	This is a Maintenance API to trigger Maintenance Task execution thread by setting appropriate preconditions and flags to them
Prototype	void MaintenanceManager::maintenanceManagerOnBootup()
Input Args	None
Return	void

_MaintenanceMgrEventHandler

Description	This is an event handler to trigger IARM Event Handler iarmEventHandler() for events triggered by InitializeIARM() and DeinitializeIARM()
Prototype	void MaintenanceManager::_MaintenanceMgrEventHandler(const char owner, IARM_EventId_t eventId, void data, size_t len)
Input Args	const char owner, IARM_EventId_t eventId, void data, size_t len
Return	void

iarmEventHandler

Description	This is an IARM based event handler for Maintenance Manager, triggered by _MaintenanceMgrEventHandler, this sets appropriate Task Completion or Error Status and completion/ incompletion of all tasks it sets the Maintenance Status and joins the thread accordingly
Prototype	void MaintenanceManager::iarmEventHandler(const char owner, IARM_EventId_t eventId, void data, size_t len)
Input Args	const char owner, IARM_EventId_t eventId, void data, size_t len
Return	void

DeinitializeIARM

Description	This method checks if IARM is Connected and removes event handlers for Maintenance Event Update Event and DCM Start Time Event
Prototype	void MaintenanceManager::DeinitializeIARM()
Input Args	None
Return	void

stopMaintenanceTasks

Description	This Utility method is triggered via stopMaintenance() RPC method to stop a running (MAINTENANCE_STARTED) maintenance cycle gracefully. It aborts all the tasks on a running Maintenance using abortTask(). And once that is done, the threads used are joined and MAINTENANCE_ERROR is set as Maintenance Status on exit
Prototype	bool MaintenanceManager::stopMaintenanceTasks()
Input Args	None
Return	true	returns if all tasks are gracefully exited (aborted)
Return	false	returns in any other cases

readRFC

Description	A Utility method to read the RFC Value set for the RFC Parameter passed (Boolean Data type RFCs only) and returns it
Prototype	bool MaintenanceManager::readRFC(const char *rfc)
Input Args	const char *rfc
Return	true	if the RFC Param stored has value "true"
Return	false	if the RFC Param stored has value "false" or the RFC Param does not exist

abortTasks

Description	A Utility method used by Maintenance Manager. This uses getTaskPID() to get the PID of the process to be aborted and terminates it with an appropriate Signal passed. This is triggered in case of Stop Maintenance
Prototype	int MaintenanceManager::abortTask(const char* taskname, int sig_to_send)
Input Args	const char* taskname, int sig_to_send
Return	int

getTaskPID

Description	A Utility method used in Maintenance Manager to get the process PID. This is used to abort the Task gracefully using its PID in case Stop Maintenance is being triggered
Prototype	pid_t MaintenanceManager::getTaskPID(const char* taskname)
Input Args	const char* taskname
Return	pid_t

Remote Procedure Calls (RPC)

getMaintenanceActivityStatus

Description	JsonRPC Method to get the current Maintenance Status or the Maintenance Status of the recent Maintenance Activity (either of MAINTENANCE_IDLE/ MAINTENANCE_STARTED/ MAINTENANCE_ERROR/ MAINTENANCE_COMPLETE/ MAINTENANCE_INCOMPLETE) along with info like isCriticalMaintenance
Prototype	uint32_t MaintenanceManager::getMaintenanceActivityStatus(const JsonObject& parameters, JsonObject& response)
Parameters	const JsonObject& parameters, JsonObject& response
RPC Call	curl -H "Authorization: Bearer `WPEFrameworkSecurityUtility \| cut -d '"' -f 4`" --header "Content-Type: application/json" --request POST --silent -d '{"jsonrpc":"2.0","id":"3","method":"org.rdk.MaintenanceManager.1.getMaintenanceActivityStatus","params":{}}' http://127.0.0.1:9998/jsonrpc
Response	`{"jsonrpc":"2.0","id":3,"result":{"maintenanceStatus":"MAINTENANCE_ERROR","LastSuccessfulCompletionTime":0,"isCriticalMaintenance":false,"isRebootPending":false,"success":true}}`
Return	uint32_t

getMaintenanceStartTime

Description	returns the future time window - start time in epoch or Unix Time (UTC) - of the critical activities in the maintenance schedule. The time the critical activities start is uniformly distributed by XConf across the entire population of CPEs (based on the PMI). Maintenance start time from XConf comes as local time. This is because in the past the maintenance start time is used to configure Cron directly, which works based on local time. This has limitations at the begin and at the end of the daylight saving period as it can happen that maintenance doesn't run for that day, or run twice (e.g. if it's between 1am and 2am, it won't run the day daylight saving period starts – and clock moves forward - and it will run twice the day daylight saving period ends – and clock moves backward) on the contrary, the start time provided to the upper layers by the maintenance Thunder service needs to make sure that maintenance never fails to run or run twice during the same day. For this reason, the time reported back is always epoch or Unix Time express in number of seconds since 1st Jan 1970 (UTC). Care needs to be put in converting local time to epoch time so that maintenance always run once every day.
Prototype	uint32_t MaintenanceManager::getMaintenanceStartTime (const JsonObject& parameters, JsonObject& response)
Parameters	const JsonObject& parameters, JsonObject& response
RPC Call	curl -H "Authorization: Bearer `WPEFrameworkSecurityUtility \| cut -d '"' -f 4`" --header "Content-Type: application/json" --request POST --silent -d '{"jsonrpc":"2.0","id":"3","method":"org.rdk.MaintenanceManager.1.getMaintenanceStartTime","params":{}}' http://127.0.0.1:9998/jsonrpc
Response	Positive	`{"jsonrpc":"2.0","id":3,"result":{"maintenanceStartTime":1640094912,"success":true}}`
Response	Negative	`{"jsonrpc":"2.0","id":3,"result":{"maintenanceStartTime":-1,"success":true}}`
Return	uint32_t

startMaintenance

Description	JsonRPC Method to start a Maintenance Cycle (SOLICITED MAINTENANCE). This is triggered by App (EPG/AS) to start scheduled maintenance based on the Start Maintenance Time. This RPC can be triggered only when a Maintenance Cycle is not running already. This will give a successful response with positive response when a maintenance is not running already and the trigger is successfule. It returns false response if the trigger fails or if there is already a maintenance is running.
Prototype	uint32_t MaintenanceManager::startMaintenance(const JsonObject& parameters, JsonObject& response)
Parameters	const JsonObject& parameters, JsonObject& response
RPC Call	curl -H "Authorization: Bearer `WPEFrameworkSecurityUtility \| cut -d '"' -f 4`" --header "Content-Type: application/json" --request POST --silent -d '{"jsonrpc":"2.0","id":"3","method":"org.rdk.MaintenanceManager.1.startMaintenance","params":{}}' http://127.0.0.1:9998/jsonrpc
Response	Positive	`{"jsonrpc":"2.0","id":3,"result":{"success":true}}`
Response	Negative	`{"jsonrpc":"2.0","id":3,"result":{"success":false}}`
Return	uint32_t

stopMaintenance

Description	JsonRPC Method to gracefully end all the current running maintenance tasks, joins all threads and exists maintenance with MAINTENANCE_ERROR Status. This RPC Call gives successful response with true if the running maintenance is stopped successfully and it returns false response if there is a failure in the stopping process or if thee is no running Maintenance to stop. This is used by App (EPG/AS) to stop the current running maintenance. Stop Maintenance feature is controlled via an `RFC Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.StopMaintenance.Enable` (this is defaulted to true i.e. Enabled by default)
Prototype	uint32_t MaintenanceManager::stopMaintenance(const JsonObject& parameters, JsonObject& response)
Parameters	const JsonObject& parameters, JsonObject& response
RPC Call	curl -H "Authorization: Bearer `WPEFrameworkSecurityUtility \| cut -d '"' -f 4`" --header "Content-Type: application/json" --request POST --silent -d '{"jsonrpc":"2.0","id":"3","method":"org.rdk.MaintenanceManager.1.stopMaintenance","params":{}}' http://127.0.0.1:9998/jsonrpc
Response	Positive	`{"jsonrpc":"2.0","id":3,"result":{"success":true}}`
Response	Negative	`{"jsonrpc":"2.0","id":3,"result":{"success":false}}`
Return	uint32_t

setMaintenanceMode

Description	JsonRPC Method to set the Maintenance Mode. FOREGROUND This mode is a vanilla mode that lets Maintenance to run all the Maintenance Tasks BACKGROUND Abort activities currently running and if the task can't run in background (if maintenance was already started with FOREGROUND mode). The only exception is a critical software download. Execute tasks not impacting user experience if the maintenance mode is set to BACKGROUND before calling startMaintenance
Prototype	uint32_t MaintenanceManager::setMaintenanceMode(const JsonObject& parameters, JsonObject& response)
Parameters	const JsonObject& parameters, JsonObject& response
RPC Call	FOREGROUND	curl -H "Authorization: Bearer `WPEFrameworkSecurityUtility \| cut -d '"' -f 4`" --header "Content-Type: application/json" --request POST --silent -d '{"jsonrpc":"2.0","id":"3","method":"org.rdk.MaintenanceManager.1.setMaintenanceMode","params":{"maintenanceMode":"FOREGROUND"}}' http://127.0.0.1:9998/jsonrpc
RPC Call	BACKGROUND	curl -H "Authorization: Bearer `WPEFrameworkSecurityUtility \| cut -d '"' -f 4`" --header "Content-Type: application/json" --request POST --silent -d '{"jsonrpc":"2.0","id":"3","method":"org.rdk.MaintenanceManager.1.setMaintenanceMode","params":{"maintenanceMode":"BACKGROUND"}}' http://127.0.0.1:9998/jsonrpc
Response	Positive	`{"jsonrpc":"2.0","id":3,"result":{"success":true}}`
Response	Negative	`{"jsonrpc":"2.0","id":3,"result":{"success":false}}`
Return	uint32_t

getMaintenanceMode

Description	JsonRPC Method to get the current Maintenance Manager Mode (either FOREGROUND/ BACKGROUND). The received response will also have OptOut Value (NONE/ IGNORE_UPDATE/ BYPASS_OUTPUT/ ENFORECE_OPTOUT)
Prototype	uint32_t MaintenanceManager::getMaintenanceMode(const JsonObject& parameters, JsonObject& response)
Parameters	const JsonObject& parameters, JsonObject& response
RPC Call	curl -H "Authorization: Bearer `WPEFrameworkSecurityUtility \| cut -d '"' -f 4`" --header "Content-Type: application/json" --request POST --silent -d '{"jsonrpc":"2.0","id":"3","method":"org.rdk.MaintenanceManager.1.getMaintenanceMode","params":{"maintenanceMode":"BACKGROUND"}}' http://127.0.0.1:9998/jsonrpc
Response	Foreground	`{"jsonrpc":"2.0","id":3,"result":{"maintenanceMode":"FOREGROUND","optOut":"NONE","success":true}}`
Response	Background	`{"jsonrpc":"2.0","id":3,"result":{"maintenanceMode":"BACKGROUND","optOut":"NONE","success":true}}`
Return	uint32_t

Events

onMaintenanceStatusChange

Description	This is a critical utility method which sends IARM Event upon Maintenance Status Event changes during the Maintenance Flow
Prototype	void MaintenanceManager::onMaintenanceStatusChange(Maint_notify_status_t status)
Input Args	Maint_notify_status_t status
Return	void

Functions

notifyStatusToString

Description	Utility function to get the string equivalent of Maintenance Status passed
Prototype	string notifyStatusToString(Maint_notify_status_t &status);
Input Args	Maint_notify_status_t &status
Return	string	Returns the corresponding Maintenance Status String for the respective Maintenance Status. Below are the possible Maintenance statuses that will be returned MAINTENANCE_IDLE MAINTENANCE_STARTED MAINTENANCE_ERROR MAINTENANCE_COMPLETE MAINTENANCE_INCOMPLETE

checkValidOptOutModes

Description	Utility function if the OptOut passed is from one that is honored by Maintenance Manager
Prototype	bool checkValidOptOutModes(string OptoutModes);
Input Args	string OptoutModes
Return	true	Returns when the passed OptOutMode is in either of the one below; ENFORCE_OPTOUT BYPASS_OPTOUT IGNORE_UPDATE NONE
Return	false	Returns when the OptOut is not any of the above

moduleStatusToString

Description	Utility function to get the string equivalent of Maintenance IARM Activity Status passed
Prototype	string moduleStatusToString(IARM_Maint_module_status_t &status);
Input Args	IARM_Maint_module_status_t &status
Return	string	Returns the corresponding Maintenance Status String for the respective Maintenance IARM Status. Below are the possible IARM statuses that will be returned MAINTENANCE_DCM_COMPLETE MAINTENANCE_DCM_ERROR MAINTENANCE_RFC_COMPLETE MAINTENANCE_RFC_ERROR MAINTENANCE_LOGUPLOAD_COMPLETE MAINTENANCE_LOGUPLOAD_ERROR MAINTENANCE_PINGTELEMETRY_COMPLETE MAINTENANCE_PINGTELEMETRY_ERROR MAINTENANCE_FWDOWNLOAD_COMPLETE MAINTENANCE_FWDOWNLOAD_ERROR MAINTENANCE_REBOOT_REQUIRED MAINTENANCE_FWDOWNLOAD_ABORTED MAINTENANCE_CRITICAL_UPDATE MAINTENANCE_EMPTY

Note

Maintenance Manager should log to maintenance.log file
Maintenance Manager should not crash or leak memory at any cost

Unsolicited Maintenance

Verify Unsolicited Maintenance (Maintenance on Bootup)

Feature: MaintenanceManager trigger on bootup

  Background:
    Given the system is Up with WPEFramework running

  Scenario: Verify unsolicited maintenance on every bootup
     Given the system is rebooted
     When the MaintenanceManager starts
     Then the MaintenanceManager should trigger all the Maintenance tasks
     And the Maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR

Solicited Maintenance

Verify Solicited Maintenance (Scheduled Maintenance)

Feature: Verify Solicited maintenance behavior (Maintenance Window)

  Background:
    Given the system has no active Maintenance running

  Scenario: Verify solicited maintenance execution
    Given the Maintenance is scheduled at a specific time
    And the current Maintenance status is not in progress
    When the scheduled maintenance time is reached
    Then the solicited Maintenance should execute all the Maintenance tasks
    And the maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR
    And the default behavior is to set "isRebootPending" flag to true

Verify Solicited Maintenance (via RPC Call)

Feature: Verify Solicited Maintenance behavior (startMaintenance jsonPRC Test on an inactive Maintenance Cycle)

  Background:
    Given the system has no active Maintenance running

  Scenario: Verify solicited maintenance execution
    Given the current Maintenance status is not in progress
    When the startMaintenance jsonRPC is invoked
    Then the solicited maintenance should execute all the Maintenance tasks
    And the maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR
    And the default behavior is to set "isRebootPending" flag to true
    And the return response for RPC call should be true

Verify Solicited Maintenance (via RPC Call on an active Maintenance)

Feature: Verify Solicited Maintenance behavior (startMaintenance jsonPRC Test on an active Maintenance Cycle)

  Background:
    Given the system has an active Maintenance running

  Scenario: Verify solicited maintenance invocation during active maintenance
    Given the current Maintenance status is in progress
    When the startMaintenance jsonRPC is invoked
    Then the solicited maintenance should not start any new tasks
   And the return response for RPC call should be false
   And the current running Maintenance tasks should proceed
   And the final Maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR
    And the "isRebootPending" flag should be true

JsonRPC

Verify stopMaintenance on an active Maintenance Cycle

Feature: Verify stopMaintenance jsonPRC on an active Maintenance Cycle

  Background:
    Given the system has an active Maintenance running

  Scenario: Verify stop maintenance when maintenance is in progress
    Given the current Maintenance status is in progress
    When the stopMaintenance jsonRPC is invoked
    Then the currently running tasks should be aborted immediately
    And the maintenance status should be MAINTENANCE_ERROR
    And the return response should be true

Verify stopMaintenance on an inactive Maintenance

Feature: Verify stopMaintenance jsonPRC on an active Maintenance Cycle

  Background:
    Given the system has no Maintenance running i.e Maintenance is either in MAINTENANCE_ERROR/ MAINTENANCE_COMPLETE

  Scenario: Verify stop maintenance when no maintenance is active
    Given the current Maintenance status is not in progress
    When the stopMaintenance jsonRPC is invoked
    Then no maintenance tasks should be stopped or aborted
    And the maintenance status should remain unchanged
    And the return response should be false

Maintenance on No Internet

Feature: Test Maintenance Manager behavior when there is no internet

  Background:
    Given the system is configured to perform maintenance tasks

  Scenario: Handle no internet connection during maintenance
    Given the device has no internet connection
    When the maintenance manager starts
    Then the maintenance Manager should subscribe to network event
    Then the maintenance manager should check the network status 4 times
    And there should be a 30-second delay between each retry
    And if there is still no internet connection after retries, the maintenance manager subscribes to Network Event
    And the maintenance status should be set to MAINTENANCE_ERROR
    And no maintenance tasks should run

Maintenance on No Internet, then reconnected

Feature: Test Maintenance Manager behavior when there is no internet

  Background:
    Given the system is configured to perform maintenance tasks and Network Plugin is active after some time

  Scenario: Handle internet reconnection
    Given the device has no internet connection
    When the maintenance manager starts
   Then the maintenance Manager should subscribe to network event
    Then the maintenance manager should check the network status 4 times
    And there should be a 30-second delay between each retry
    And if there is still no internet connection after retries, the maintenance manager subscribes to Network Event
    And the maintenance status should be set to MAINTENANCE_ERROR
    And no maintenance tasks should run

   When the internet is reconnected and the maintenance manager receives FULLY_CONNECTED event
    Then the maintenance manager should trigger critical tasks like RFC and xconfImageCheck
    And the maintenance status should remain MAINTENANCE_ERROR
    And no maintenance tasks should run

getMaintenanceActivityStatus jsonRPC

Feature: Test Maintenance Manager getMaintenanceActivityStatus jsonRPC Call

  Background:
    Given the system is configured to perform maintenance tasks

  Scenario: Verify getMaintenanceActivityStatus returns the correct status information
    Given the system has completed a maintenance activity with status MAINTENANCE_ERROR
    When the getMaintenanceActivityStatus jsonRPC is invoked
    Then the response should include "maintenanceStatus" as either of MAINTENANCE_IDLE/ MAINTENANCE_STARTED/ MAINTENANCE_ERROR/ MAINTENANCE_COMPLETE
    And the response should include "LastSuccessfulCompletionTime" an integer value
    And the response should include "isCriticalMaintenance" be either true/ false
    And the response should include "isRebootPending" as true/ false
    And the response should include "success" as true

getMaintenanceStartTime jsonRPC returns Start Time

Feature: Test Maintenance Manager getMaintenanceActivityStatus jsonRPC Call

  Background:
      Given the system is configured to provide maintenance start times based on UTC in Unix epoch time

   Scenario: Verify getMaintenanceStartTime returns a valid future time window
    Given the system is scheduled for a future maintenance activity
    When the getMaintenanceStartTime jsonRPC is invoked
    Then the response should include "maintenanceStartTime" as a positive Unix epoch time
    And the response should include "success" as true

getMaintenanceStartTime jsonRPC does not return Start Time

Feature: Test Maintenance Manager getMaintenanceActivityStatus jsonRPC Call

  Background:
      Given the system is configured to provide maintenance start times based on UTC in Unix epoch time

   Scenario: Verify getMaintenanceStartTime handles the absence of a maintenance schedule
    Given the system has no scheduled maintenance/ no timeStamp files like /opt/persistent/timeZoneDST, etc
    When the getMaintenanceStartTime jsonRPC is invoked
    Then the response should include "maintenanceStartTime" as -1
    And the response should include "success" as true

Suppress Maintenance

Maintenance Manager behavior on Suppress Maintenance

Feature: Test Maintenance Manager behavior when SUPPRESS_MAINTENANCE is enabled

  Background:
      Given SUPPRESS_MAINTENANCE is enabled

   Scenario: Execute all tasks when device activation status does not require skipping firmware
    Given the device activation status is ACTIVATED/ ACTIVATION_DISCONNECT
    And the skipFirmware flag is set to false
    When the maintenance tasks are being executed
   Then the RFC task should be executed
    And the firmware download task should be executedd
    And the log upload task should be executed
    And the maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR

Maintenance Manager skips FWDOWNLOAD Task on Suppress Maintenance

Feature: Skip Firmware Task when SUPRESS_MAINTENANCE is enabled for Maintenance Manager

  Background:
      Given SUPPRESS_MAINTENANCE is enabled, and FWDOWNLOAD Task is skipped

   Scenario: Skip firmware task when device activation status requires it
    Given the device activation status is NOT_ACTIVATED/ ACTIVATION_READY/ ACTIVATION_CONNECT
    And the skipFirmware flag is set to true
    When the maintenance tasks are being executed
    Then the RFC task should be executed
   And the firmware download task should be skipped
    And the log upload task should be executed
    And the maintenance status should be MAINTENANCE_COMPLETE/ MAINTENANCE_ERROR

WhoAmI

Verify Unsolicited Maintenance (Maintenance on Bootup when WhoAmI is enabled)

Feature: MaintenanceManager trigger on bootup when ENABLE_WHOAMI flag is enabled

  Background:
    Given the system is Up with WPEFramework running, Authservice and SecManager Plugins are active and running

  Scenario: Verify unsolicited maintenance on every bootup
     Given the WhoAmI enabled system is rebooted
     When the MaintenanceManager is initialized
     And MaintenanceManager should subscribe to SecManager Plugin for onDeviceInitializationContextUpdate Event
     And MaintenanceManager should get the ActivationStatus of the device
     And MaintenanceManager should call getDeviceInitializationContext API to get the Device Context
         When received the context data
            The context should be set to respective RFCs and Continue with task execution
         When context is not received
           When the SecManager is active, the MaintenanceManager should wait for Context Event
            When SecManager sends the Context Event MaintenanceManager
            The context should be set to respective RFCs and Continue with task execution
     Then the MaintenanceManager should trigger all the Maintenance tasks
     And the Maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR

1.	Reboot	Verify unsolicited maintenance on every bootup	Maintenance manager should trigger all the 4 maintenance tasks on bootup and maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR
2.	Maintenance should not be in progress	Verify solicited maintenance	Maintenance should happened at the scheduled time. It should execute all the above mentioned 3 tasks and maintenance status should be MAINTENANCE_COMPLETE or MAINTENANCE_ERROR. Verify "isRebootPending" flag is set as true.
3.	Maintenance in progress	Verify stop maintenance	Currently running tasks should be aborted immediately and maintenance status should be MAINTENANCE_ERROR
4.	Disconnect internet on bootup	Device should not connect to internet on first 2 mins	Device should exit from maintenance activities and maintenance status should be MAINTENANCE_ERROR
5.	Disconnect internet on bootup	MM should subscribe for network event	If there is no internet on first 2 mins, then device should exit from maintenance and MM should subscribe for network event. MM should trigger StartDCM_maintaince.sh and xconfImageCheck.sh once device got internet to start telemetry and RDM
6.	Reboot	Verify crash	Ensure there are no crashes in wpeframework due to Maintenance Manager. Check corelog.txt once maintenance completed
7.	Reboot	Verify maintenance completed successfully	Maintenance status should not be MAINTENANCE_STARTED for long time. It should moved to MAINTENANCE_COMPLETE or MAINTENANCE_ERROR as soon as maintenance completed
8.	Reboot	Maintenance Manager Api's should return either return true or false	Verify MM Api's returns success or failure response immediately. It should not hang for long time.
9	Reboot	Verify whoAmI feature is not enabled in non xfinity entos platforms	whoAmI feature or code should not reflect in non whoAmI enabled platforms
10	Reboot	whoAmI lookup must happen on every bootup in whoAmI enabled platforms	Device should acquire gdi identity and should store the gdi response in respective RFC parameters

RDK Resources

[*RDK Preferred*]

Code Management Facility

RDK Forums

[RDK Conferences]

RDK Support

Archives

RDK Maintenance Manager

Overview

Source Code

Terminology

Types of Maintenance

Unsolicited Maintenance

Solicited Maintenance

Functionality Observations

WhoAmI

Maintenance Activity/ Task

XConf Settings

Architecture Overview (HLA)

Activity Diagram

High Level Sequence Diagram

Maintenance Status Notifications

Distros and RFCs used in Maintenance Manager

Notable features

Logs

Child Pages

Methods, RPCs. Events and Functions

Methods

subscribeForInternetStatusEvent

task_execution_thread

getThunderPluginHandle

internetStatusChangeEventHandler

startCriticalTasks

checkNetwork

isDeviceOnline

Initialize

Deinitialize

InitializeIARM

maintenanceManagerOnBootup

_MaintenanceMgrEventHandler

iarmEventHandler

DeinitializeIARM

stopMaintenanceTasks

readRFC

abortTasks

getTaskPID

Remote Procedure Calls (RPC)

getMaintenanceActivityStatus

getMaintenanceStartTime

startMaintenance

stopMaintenance

setMaintenanceMode

getMaintenanceMode

Events

onMaintenanceStatusChange

Functions

notifyStatusToString

checkValidOptOutModes

moduleStatusToString

[RDK Preferred]