Introduction

Selfheal is another feature implemented in Test And Diagnostic Component. SelfHeal is a monitoring and recovery module.

Self-heal Periodically monitors the below scenarios:

• CPU usage
• Memory Usage
• Critical RDK-B processes

Self-heal stores Reset Count and Reboot Count.
Self-heal takes required action like: Rebooting the device, Restarting required process based on predefined conditions.
Self-heal does connectivity test.

Design Considerations

Self Heal functionality is handled by a set of scripts. These scripts are available in the RDK-B RPI build by default, and customised to rpi system specificification referring to actual devices.

Please ensure that below Self heal scripts are present on the device at the path "/usr/ccsp/tad".

• resource_monitor.sh

• task_health_monitor.sh

• corrective_action.sh

• self_heal_connectivity_test.sh

Resource Monitoring

       "resource_monitor.sh" script is used for monitoring Memory and CPU usage.Monitors the resources periodically (eg: 60 seconds). If "Average Memory Used" reaches threshold value, reboot action will be executed. 

For Resource Monitor Sequence ,

                          i)  First cycle onwards -  sleep will calculate based on below commands

                                                   Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_UsageComputeWindow *60 .  For example : By default, RMInterval value as 1 . so, sleep will be 60.

Process Monitoring

1.         "task_health_monitor.sh" script is used for monitoring all RDKB processes .Monitors the processes periodically (eg:- 60 seconds) based on it's process id (pid).
2.           Based on the process id availability, required action will be taken such as restarting the process, rebooting the device.

       Important points to remember  :                 

• Ccsp processes: If any of these processes crashed, it will be restarted via Self Heal.
• "CcspCrSsp": If this process is crashed, device will be rebooted.
• "syseventd": If syseventd is crashed, device will be rebooted.

  
  

For  task monitor sequence,

                          i)  First cycle onwards -  sleep will calculate based on below commands

                                                   Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_UsageComputeWindow *60 .  For example : By default, RMInterval value as 1 . so, sleep will be 60.

Connectivity Test

         "self_heal_connectivity_test.sh" script is used for ping test.Ping test will be done through server IP/URI (this needs to be configured). If server IP/URI is not configured, Ping test won't be executed and no action will be taken. If server is configured and ping test fails, reboot action will be executed.

For Connectivity Test  sequence ,

                            i) After boot-up,Very  First cycle  - random sleep functionality call was called.

                           ii) Second cycle onwards - sleep will calculate based on below commands

                                                  Device.SelfHeal.ConnectivityTest.X_RDKCENTRAL-COM_PingInterval - value of this command * 60 . For example : By default, PingInterval value as 60 . so, sleep will be 3600.

SelfHeal Logs

          Self heal logs will be created on below folder,

                             /rdklogs/logs/SelfHeal.txt.0

Architecture

Self Heal DataModel Flow 

Process Monitor Flow 

Data Model

Lists of self heal supported DataModel commands 

Limitations

             RPI doesn't have IPv6 support functionality. So we skip the ipv6 logics  from "self_heal_connectivity_test.sh" and "task_health_monitor.sh".