• No labels

6 Comments

  1. Z-Vijayaragavalu Sundaresan

    On one of the product that will have Cable Gateway with WiFi which runs with RDKB, sometimes the CPU usage reaches to 50% for CcspPandMSsp process.

    can you please let me know if there is any option in RDK to know which thread or which area causing more CPU usage.


    ' top ' result  during when CPU reaches to 45%

    root@Docsis-Gateway:~# top
    Mem: 172636K used, 113332K free, 0K shrd, 131K buff, 44K cached
    CPU:   5% usr  45% sys   0% nic  50% idle   0% io   0% irq   0% sirq
    Load average: 3.15 2.46 1.23 2/228 10258
      PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
     3633     1 root     S    84128  29%  45% /usr/bin/CcspPandMSsp -subsys eRT.
    10258  2658 root     R     2112   1%   5% top
     8655     1 root     S    88120  31%   0% IGD brlan0
     5822     1 root     S    74520  26%   0% /usr/bin/CcspWifiSsp -subsys eRT.
     5826     1 root     S    62288  22%   0% /usr/bin/CcspLMLite -subsys eRT.
     2890     1 root     S    51072  18%   0% /usr/bin/gw_prov_utopia
     3631     1 root     S    47832  17%   0% /usr/bin/CcspCMAgentSsp -subsys eRT.
     5771     1 root     S    39440  14%   0% /usr/bin/CcspTr069PaSsp -subsys eRT.
     5753     1 root     S    37488  13%   0% /usr/bin/CcspMtaAgentSsp -subsys eRT.
     3402  3290 root     S    31472  11%   0% /usr/bin/snmp_subagent -x tcp:127.0.0.
     5952     1 root     S    29140  10%   0% /usr/bin/CcspTandDSsp -subsys eRT.
     3514     1 root     S    29096  10%   0% /usr/bin/PsmSsp -subsys eRT.
     2542     1 root     S    28600  10%   0% /usr/bin/bootassist -N /mnt/cmnonvol/c
     3383     1 root     S    27548  10%   0% /usr/bin/CcspCrSsp -subsys eRT.
     6531     1 root     S    26340   9%   0% /usr/bin/CcspXdnsSsp -subsys eRT.
     8546  8544 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
     8558  8544 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
     8580  8544 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
     8586  8544 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
     8752  8546 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini

    Please let me know if anymore information to be shared here other than version information.

    RDK_VERSION=rdkb-20171127

    Linux version 3.14.28

    gcc version 4.8.4 (GCC)


  2. Z-Harikrishnan P H

    I don't know if there is an option in RDKB for analyzing high CPU usage. But you can use "top -H" to get the per-thread CPU usage of all the processes. Provided that the threads used in your application are named. You should be able to start your debugging with this.

  3. Z-Vijayaragavalu Sundaresan

    Hi Z-Harikrishnan P H,

    Thanks for your suggestion.

    The BusyBox which we are using , does not enable -H option for 'top' , hence unable to do this.

    root@Docsis-Gateway:~# top -H
    top: invalid option -- 'H'
    BusyBox v1.22.1 (2018-10-03 12:03:46 IST) multi-call binary.
    Usage: top [-b] [-nCOUNT] [-dSECONDS] [-m]
    root@Docsis-Gateway:~#

  4. Z-Harikrishnan P H

    Can you try "printf h | top -n 1" . Or executing top command and then entering the h/H key. Usually, busybox supports showing per-thread usage of CPU.

  5. Z-Vijayaragavalu Sundaresan

    I tired "printf h | top -n 1" and also "printf H | top -n 1" , it is not taking h/H options


    root@Docsis-Gateway:~# printf H | top -n 1
    Mem: 177092K used, 108876K free, 0K shrd, 133K buff, 44K cached
    CPU:   5% usr  45% sys   0% nic  50% idle   0% io   0% irq   0% sirq
    Load average: 3.03 3.01 2.26 2/232 12794
      PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
     3679     1 root     S    84128  29%  45% /usr/bin/CcspPandMSsp -subsys eRT.
    12794  2611 root     R     2112   1%   5% top -n 1
     6326     1 root     S    88120  31%   0% IGD brlan0
     7579     1 root     S    82316  29%   0% /usr/bin/CcspWifiSsp -subsys eRT.
     7583     1 root     S    62288  22%   0% /usr/bin/CcspLMLite -subsys eRT.
     2895     1 root     S    51072  18%   0% /usr/bin/gw_prov_utopia
     3674     1 root     S    47712  17%   0% /usr/bin/CcspCMAgentSsp -subsys eRT.
     7486     1 root     S    39440  14%   0% /usr/bin/CcspTr069PaSsp -subsys eRT.
     7464     1 root     S    37488  13%   0% /usr/bin/CcspMtaAgentSsp -subsys eRT.
     3261  3148 root     S    31472  11%   0% /usr/bin/snmp_subagent -x tcp:127.0.0.
     3522     1 root     S    29228  10%   0% /usr/bin/PsmSsp -subsys eRT.
     7698     1 root     S    29140  10%   0% /usr/bin/CcspTandDSsp -subsys eRT.
     2538     1 root     S    28600  10%   0% /usr/bin/bootassist -N /mnt/cmnonvol/c
     7193     1 root     S    27720  10%   0% CcspHomeSecurity 8081
     3252     1 root     S    27548  10%   0% /usr/bin/CcspCrSsp -subsys eRT.
     8280     1 root     S    26340   9%   0% /usr/bin/CcspXdnsSsp -subsys eRT.
     6560  6558 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
     6563  6558 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
     6582  6558 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
     6587  6558 root     S    21444   7%   0% /usr/bin/php-cgi -c /etc/php.ini
    root@Docsis-Gateway:~#

    and also top command is not taking H or h keyboard input . but it takes S,R,N,M key board inputs.

  6. Z-Vijayaragavalu Sundaresan

    Can you please help me how to enable backtrace on CcspPandMSsp and CcspWifiSsp

    I did as below 

    1) Enabled Debug symbol on CcspPandMSsp/CcspWifiSsp in recipes-ccsp

    +CFLAGS_append += "-DRDKLOGGER_SUPPORT_WIFI"   

    +DEBUG_BUILD = "1"

    with this change ,  made sure that  CcspPandMSsp and CcspWifiSsp agent executable enabled with debug sysmbol(confirmed with nm tool)

    2) changed as below in WifiSsp/ssp_main.c (and also similar change inPandMSsp/ssp_main.c ) to print backtrace  for signal USR1

    diff --git a/source/WifiSsp/ssp_main.c b/source/WifiSsp/ssp_main.c

    #include <execinfo.h>

     #include "stdlib.h"

    @@ -158,27 +158,31 @@ int  cmd_dispatch(int  command)

     static void _print_stack_backtrace(void)

     {

             void* tracePtrs[100];

             char** funcNames = NULL;

             int i, count = 0;

             count = backtrace( tracePtrs, 100 );

    +        CcspTraceInfo(("_print_stack_backtrace, backtrace count :  %d\n", count));

             backtrace_symbols_fd( tracePtrs, count, 2 );

             funcNames = backtrace_symbols( tracePtrs, count );

             if ( funcNames ) {

    +            CcspTraceInfo(("_print_stack_backtrace, under funcNames \n"));

                 // Print the stack trace

    -            for( i = 0; i < count; i++ )

    +            for( i = 0; i < count; i++ ) {

    +                CcspTraceInfo(("%s\n", funcNames[i]));

                     printf("%s\n", funcNames[i] );

    +            }

                 // Free the string pointers

                 free( funcNames );

             }

     }

     #if defined(_ANSC_LINUX)

    @@ -237,8 +241,10 @@ void sig_handler(int sig)

             exit(0);

         }

         else if ( sig == SIGUSR1 ) {

    +       //signal(SIGUSR1, sig_handler); /* reset it to this function */

    +        _print_stack_backtrace();

    +        CcspTraceInfo(("SIGUSR1 received!\n"));

            signal(SIGUSR1, sig_handler); /* reset it to this function */

    -       CcspTraceInfo(("SIGUSR1 received!\n"));

         }

         else if ( sig == SIGUSR2 ) {

            CcspTraceInfo(("SIGUSR2 received!\n"));

         }

    3) NFS mount the root filesystem to avoid striping of symbols in CcspPandMSsp and CcspWifiSsp

    4) During when CPU usage reached to 50% , sent the signal  to CcspWifiSsp process using "kill"

                        Eg :           kill -s USR1 7579    

    5) made sure from /rdklogs/logs/WiFilog.txt.0 log file , the USR1 signal has been received and _print_stack_backtrace() has been called.

    181004-15:03:37.209911 [mod=WIFI, lvl=INFO] [tid=5822] _print_stack_backtrace, backtrace count :  0

    181004-15:03:37.209988 [mod=WIFI, lvl=INFO] [tid=5822] _print_stack_backtrace, under funcNames

    181004-15:03:37.210015 [mod=WIFI, lvl=INFO] [tid=5822] SIGUSR1 received!

    6) but there were no backtrace log present.

    what i understood is , backtrace is not returning valid count instead it returns 0

    count = backtrace( tracePtrs, 100 );



    (PS : I also make sure that execinfo.h included to the build and  "-D_NO_EXECINFO_H_" has not been included to my build )


    Could you please let me know if i am missing anything here to enable backtrace.