RDK Resources
[*RDK Preferred*]
Code Management Facility
RDK Forums
[RDK Conferences]
RDK Support
Archives
Papers & Presentations Archive
Self Heal is a monitoring and recovery module.
It continuously monitors the system resources like CPU and Memory and monitors the critical Processes running.
Self heal also performs Connectivity tests.
In case of any problems encountered, Self Heal takes corrective actions like: Rebooting the device, Restarting required process based on predefined conditions.
Self-heal stores Reset Count and Reboot Count.
Self Heal functionality is handled by a set of scripts. These scripts are available in the RDK build by default.
Please ensure that below Self heal scripts are present on the device at the path "/usr/ccsp/tad".
resource_monitor.sh
task_health_monitor.sh
corrective_action.sh
self_heal_connectivity.sh
Please refer the below code snippet to verify self heal module was enabled or not,
root@Filogic-GW:/rdklogs/logs# dmcli eRT getv Device.SelfHeal.X_RDKCENTRAL-COM_Enable CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. Parameter 1 name: Device.SelfHeal.X_RDKCENTRAL-COM_Enable type: bool, value: true root@Filogic-GW:/rdklogs/logs# root@Filogic-GW:/rdklogs/logs# ps -alx | grep reso 4 0 4531 1 20 0 3496 2744 do_wai S ? 0:00 /bin/sh /usr/ccsp/tad/resource_monitor.sh 0 0 16777 10009 20 0 2244 808 pipe_w S+ pts/0 0:00 grep reso root@Filogic-GW:/rdklogs/logs# ps -alx | grep self 4 0 4528 1 20 0 3628 2836 do_wai S ? 0:00 /bin/sh /usr/ccsp/tad/self_heal_connectivity_test.sh 4 0 4539 1 20 0 4288 3468 do_wai S ? 0:00 /bin/sh /usr/ccsp/tad/selfheal_aggressive.sh 0 0 16816 10009 20 0 2244 824 pipe_w S+ pts/0 0:00 grep self
Self Heal is enabled by default and is active at the time of boot up.
It periodically performs below actions.
1. By default, AVG CPU threshold value will be set as 100. This value will be stored in syscfg database. If we want the change the default AVG CPU threshold value, Please refer the below code snippet and do the following steps,
root@Filogic-GW:/# dmcli eRT getv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. Parameter 1 name: Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold type: uint, value: 100 root@Filogic-GW:/rdklogs/logs# dmcli eRT setv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold uint 200 CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. root@Filogic-GW:/# dmcli eRT getv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. Parameter 1 name: Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold type: uint, value: 200
2. By default, AVG Memory threshold value will be set as 100. This value will be stored in syscfg database. If we want the change the default AVG Memory threshold value, Please refer the below code snippet and do the following steps,
root@Filogic-GW:~# dmcli eRT getv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. Parameter 1 name: Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold type: uint, value: 100 root@Filogic-GW:/rdklogs/logs# dmcli eRT setv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold uint 200 CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. root@Filogic-GW:~# dmcli eRT getv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. Parameter 1 name: Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold type: uint, value: 200
3. Once it's reaches the threshold value, device will be rebooted automatically.
observation in /rdklogs/logs/SelfHeal.txt.0
For Memory Threshold
41106-10:22:51.210979 RDKB_SELFHEAL : Used memory in system is 153740 at timestamp 2024:11:06:10:22:51
241106-10:22:51.212450 RDKB_SELFHEAL : Free memory in system is 3754608 at timestamp 2024:11:06:10:22:51
241106-10:22:51.213869 RDKB_SELFHEAL : AvgMemUsed in % is 3
241106-10:22:51.267398 <128>CABLEMODEM[Mediatek]:<99000006><2024:11:06:10:22:51><ea:4f:a0:5d:06:99><BananapiBPI-R4> RM Memory threshold reached
241106-10:23:21.732742 RDKB_SELFHEAL : Today's reboot count is 1
241106-10:23:21.734210 RDKB_SELFHEAL : <128>CABLEMODEM[Mediatek]:<99000000><2024:11:06:10:23:21><ea:4f:a0:5d:06:99><BananapiBPI-R4> RM Rebooting device as part of corrective action
241106-10:23:21.735754 Setting Last reboot reason as MEM_THRESHOLD
241106-10:23:21.737264 Setting rebootReason to MEM_THRESHOLD and rebootCounter to 1
241106-10:23:21.789361 RDKB_REBOOT : Rebooting device due to MEM threshold reached
After reboot,
root@Filogic-GW:~# dmcli eRT getv Device.DeviceInfo.X_RDKCENTRAL-COM_LastRebootReason CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. Parameter 1 name: Device.DeviceInfo.X_RDKCENTRAL-COM_LastRebootReason type: string, value: MEM_THRESHOLD
For CPU Threshold
241106-10:54:09.546821 RDKB_SELFHEAL : Today's reboot count is 3
241106-10:54:09.548312 RDKB_SELFHEAL : <128>CABLEMODEM[Mediatek]:<99000000><2024:11:06:10:54:09><d2:33:17:da:85:e4><BananapiBPI-R4> RM Rebooting device as part of corrective action
241106-10:54:09.549727 Setting Last reboot reason as CPU_THRESHOLD
241106-10:54:09.551169 Setting rebootReason to CPU_THRESHOLD and rebootCounter to 1
241106-10:54:09.603327 RDKB_REBOOT : Rebooting device due to CPU threshold reached
<128>CABLEMODEM[Mediatek]:<99000005><2024:11:06:10:53:39><d2:33:17:da:85:e4><BananapiBPI-R4> RM CPU threshold reached
[2024-11-06:10:53:39:083104] Setting Last reboot reason
After reboot,
root@Filogic-GW:~# dmcli eRT getv Device.DeviceInfo.X_RDKCENTRAL-COM_LastRebootReason CR component name is: eRT.com.cisco.spvtg.ccsp.CR subsystem_prefix eRT. Execution succeed. Parameter 1 name: Device.DeviceInfo.X_RDKCENTRAL-COM_LastRebootReason type: string, value: CPU_THRESHOLD
If it detects that any of the process is not running, it automatically restarts that particular Component.
Let us take the example of CcspLMLite Component :
Run a ps command to verify that CcspLMLite is up and running again with different process id
ps aux | grep Ccsp
2. kill CcspLMLite process by using the below command
kill -9 PID(CcspLMLite PID)
3. Verfiy whether the CcspLMLite Process was killed or not by using the below command
ps aux | grep Ccsp
4. After 60 seconds(default), it will automatically restart the Process. Please check the CcspLMLIte PID.
If Connectivity Test fails, device will go for reboot if corrective action enabled.
Validation : Using the below steps to validate the connectivity Test
unplug the ethernet LAN cable or ifconfig erouter0 down
Note : By default DNS and corrective actions are disabled.
1. Using selfHeal logs to trouble shoot the run-time errors. SelfHeal logs will be created the below path,
/rdklogs/logs/SelfHeal.txt.0
2. Resource Monitor sample Logs,
MEM :
41106-10:22:51.210979 RDKB_SELFHEAL : Used memory in system is 153740 at timestamp 2024:11:06:10:22:51
241106-10:22:51.212450 RDKB_SELFHEAL : Free memory in system is 3754608 at timestamp 2024:11:06:10:22:51
241106-10:22:51.213869 RDKB_SELFHEAL : AvgMemUsed in % is 3
241106-10:22:51.267398 <128>CABLEMODEM[Mediatek]:<99000006><2024:11:06:10:22:51><ea:4f:a0:5d:06:99><BananapiBPI-R4> RM Memory threshold reached
241106-10:23:21.732742 RDKB_SELFHEAL : Today's reboot count is 1
241106-10:23:21.734210 RDKB_SELFHEAL : <128>CABLEMODEM[Mediatek]:<99000000><2024:11:06:10:23:21><ea:4f:a0:5d:06:99><BananapiBPI-R4> RM Rebooting device as part of corrective action
241106-10:23:21.735754 Setting Last reboot reason as MEM_THRESHOLD
241106-10:23:21.737264 Setting rebootReason to MEM_THRESHOLD and rebootCounter to 1
241106-10:23:21.789361 RDKB_REBOOT : Rebooting device due to MEM threshold reached
CPU:
241106-10:54:09.546821 RDKB_SELFHEAL : Today's reboot count is 3
241106-10:54:09.548312 RDKB_SELFHEAL : <128>CABLEMODEM[Mediatek]:<99000000><2024:11:06:10:54:09><d2:33:17:da:85:e4><BananapiBPI-R4> RM Rebooting device as part of corrective action
241106-10:54:09.549727 Setting Last reboot reason as CPU_THRESHOLD
241106-10:54:09.551169 Setting rebootReason to CPU_THRESHOLD and rebootCounter to 1
241106-10:54:09.603327 RDKB_REBOOT : Rebooting device due to CPU threshold reached
<128>CABLEMODEM[Mediatek]:<99000005><2024:11:06:10:53:39><d2:33:17:da:85:e4><BananapiBPI-R4> RM CPU threshold reached
[2024-11-06:10:53:39:083104] Setting Last reboot reason
3. Process Monitor Sample Logs,
LMLite Process :
241106-09:22:55.245084 RDKB_SELFHEAL : <128>Ethwan Gateway[Mediatek]:<99000007><2024:11:06:09:22:53><e6:72:eb:94:4f:2e><BananapiBPI-R4> RM CcspLMLite process not running , restarting it
241106-09:22:55.246875 RDKB_SELFHEAL : Resetting process CcspLMLite
4. Connectivity Test Sample Logs ,
Successful Scenario :
190924-08:56:43.577621 [RDKB_SELFHEAL] : GW IP Connectivity Test Successfull
190924-08:56:43.583217 [RDKB_SELFHEAL] : IPv4 GW Address is:192.168.30.1
190924-08:56:43.588370 [RDKB_SELFHEAL] : IPv6 GW Address is:
190924-08:56:43.622618 RDKB_SELFHEAL : Ping server lists are empty , not taking any corrective actions
190924-08:56:43.730057 DNS Response: Got success response for this URL www.google.com
Failure Scenario :
191007-09:00:13.899713 [RDKB_SELFHEAL] : GW IP Connectivity Test Successfull
191007-09:00:13.909201 [RDKB_SELFHEAL] : IPv4 GW Address is:192.168.60.1
191007-09:00:13.918684 [RDKB_SELFHEAL] : IPv6 GW Address is:
191007-09:00:13.972966 RDKB_SELFHEAL : Ping server lists are empty , not taking any corrective actions
191007-09:00:14.119985 DNS Response: fail to resolve this URL www.google.com
191007-09:00:14.152808 RDKB_SELFHEAL : Taking corrective action
RDKBACCL-303 - Getting issue details... STATUS