Crash & Hang Issue Analysis

Crash can happen in 2 types

1. Hang come Crash
2. Direct Crash

Before proceed to analyze the Crash, try to check the back trace for any know issue which has been observed earlier; also check for any AAMP side symbols are there in the Crash Portal back trace. If nothing is there 99% that is not AAMP side issue, assign to appropriate team and save your time.

You can search below strings in Back Trace to find whether we have any AAMP side symbols.

1. 1. Aamp - we mostly have our symbols associated with libaamp.so, receiver (for aampwebvideo) and libaampjsbindings.so for HLS/DASH playbacks.
   2. AesDec - in case if the playback goes through AesDec DRM for HLS.
   3. libAVEPlayer.so - in case if the playback goes through AVE DRM for HLS.
   4. fogcli- fog side crashes.

How to analyze/find Hang come Crash

First search for the below print in Receiver Log.

got error 'RDKBROWSER_RENDER_PROCESS_CRASHED': WebProcess is killed due to hang

Then check for the time stamp and map the log for that time, then search for the below prints in Receiver log.
(If you are using Notepad++ then search the below regular expression on receiver log)

WebProcess is unresponsive|RDKBROWSER_RENDER_PROCESS_CRASHED|aamp_tune:

Then find the place where the WebProcess is unresponsive comes with reply num=1(max=9), so this is the place where the hang really started. Then check the logs above to this point to understand the hang reason from where the hang is initiated.

Usually hang come crash always comes with the signal reason as SIGFPE (HANG) in Crash Portal.

Most of the time, we can see the crash signature and back trace along with symbol name and line number in Crash Portal, but if you could not find the function symbol name along with the line number on Crash Portal, then try to generate the back trace with the help of mini-dump and rootfs.tar.gz of corresponding build by following the below method. (For this you need Local Linux VM (or) Linux Desktop)

Link: Producing stack traces from google breakpad minidump manually
https://etwiki.sys.comcast.net/display/CPE/Producing+stack+traces+from+google+breakpad+minidump

For core dump back trace generation: Will create a new page and add the link here..

How to analyze/find direct Crash

First search for the below print in Receiver Log.

got error 'RDKBROWSER_RENDER_PROCESS_CRASHED': WebProcess crashed

Then check for the time stamp and map the log for that time, then search for the below prints in Receiver log.
(If you are using Notepad++ then search the below regular expression on receiver log)

WebProcess is unresponsive|RDKBROWSER_RENDER_PROCESS_CRASHED|aamp_tune:

In the case of direct crash, there is no WebProcess is unresponsive will be triggered, instead the process will crash immediately with the string got error 'RDKBROWSER_RENDER_PROCESS_CRASHED': WebProcess crashed.

Usually direct crash comes with signal SIGSEGV in Crash portal.

For the direct crash, the crashed thread of mini dump or core dump will have the details of crashed symbol/function. If that is a memory corruption crash, then we may not able to see valid call stack, so that case is bit difficult analyze.

Thread Analysis

Look for crashed thread and aamp side function symbols available threads, some time aamp can be hold by other modules and that time aamp can met with hung (or) some times aamp itself met hang with its calls. For better understanding analyze the back trace along with the logs.

Example:  DELIA-35682 - [Rack][Arris-BCM-XG1v3][AAMPDASH]: WPEIPVideo crash in handleHostCall OPEN  here aamp is blocked with sec-client call, see for understanding..

> will add few more examples..

Useful Links

1. Crash Portal : https://crashportal.stb.r53.xcal.tv/login/index
2. Producing stack traces from google breakpad minidump manually : Producing stack traces from google breakpad minidump