Recently, we upgraded our target board to rdkb2020q2 where we see meshAgent comes as native. We are in the process of integrating Opensync2.0.5 in it, so it requires meshAgent(where it acts as the middle layer between RDKB components and opensync). The issue is whenever we start the /usr/bin/meshAgent, memory leaks drastically and the Box reboots in a minute.

Board's RAM is 512 MB and available memory was ~120 MB before triggering meshAgent process.


root@Docsis-Gateway:/usr/www# /usr/bin/meshAgent -subsys eRT. &
[1] 10491
root@Docsis-Gateway:/usr/www# Conf file /etc/debug.ini open success
rdk_logger_init /etc/debug.ini Already Stack Level Logging processed... not processing again.
********ENTER breakpad_ExceptionHandler******************
******** breakpad_ExceptionHandler EXIT******************

[1]+ Done /usr/bin/meshAgent -subsys eRT.
root@Docsis-Gateway:/usr/www#
root@Docsis-Gateway:/usr/www# top
top - 03:37:41 up 5:01, 1 user, load average: 1.29, 0.88, 0.89
Tasks: 191 total, 2 running, 189 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.6 us, 12.8 sy, 0.0 ni, 79.6 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st
top - 03:37:44 up 5:01, 1 user, load average: 1.18, 0.87, 0.88
Tasks: 194 total, 2 running, 192 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.4 us, 12.7 sy, 0.0 ni, 83.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 366404 total, 141440 free, 177960 used, 47004 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 173068 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6512 root 20 0 3840 1772 1356 S 63.4 0.5 0:22.37 dbus-daemon
10492 root 20 0 143224 101944 2404 S 55.7 27.8 0:16.46 meshAgent

Tasks: 191 total, 3 running, 188 sleeping, 0 stopped, 0 zombie
%Cpu(s): 42.9 us, 25.6 sy, 0.0 ni, 28.5 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st
KiB Mem : 366404 total, 57912 free, 261436 used, 47056 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 89540 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6512 root 20 0 3840 1772 1356 R 64.5 0.5 0:28.77 dbus-daemon
10492 root 20 0 175992 134736 2404 S 56.8 36.8 0:22.07 meshAgent

===================

em : 366404 total, 26292 free, 293044 used, 47068 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 57920 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6512 root 20 0 3840 1772 1356 R 61.9 0.5 0:34.78 dbus-daemon
10492 root 20 0 207736 165884 2404 S 53.8 45.3 0:27.30 meshAgent

And the Box goes for reboot due to out of memory.


Please let us know if anyone faced similar issue? and any thoughts or suggestions

Appreciate your help!

  • No labels

10 Comments

  1. Narayanaswamy Ramaiyer

    Hello Z-Sivaraman

    Is this a reference device( like Raspberry Pi )?

  2. Z-Sivaraman

    Thank you Narayanaswamy Ramaiyer for the response.

    Its Broadcom Reference board used as Docsis Gateway. Linux version: 4.9 and Docsis version: 3.1

    1. Narayanaswamy Ramaiyer

      Could you please let us know which release version you are using( so that it might give a clue if any known fix is there already )

  3. Z-Sivaraman

    One another finding,

    When I try to run the meshAgent process by below method like go to the directory where meshAgent binary is present and run it from there, memory leak is not observed and process is also running.

    “cd /usr/ccsp/mesh”

    “meshAgent &”


    Not sure, why memory leak is observed when we use the full path “/usr/bin/meshAgent &” or “/usr/ccsp/mesh/meshAgent &”

    Is there any difference?


    1. Narayanaswamy Ramaiyer

      Hi Z-Sivaraman what is the output of

      which meshAgent

      Does it crash even if

      /usr/ccsp/mesh/meshAgent -subsys eRT. &

      and 

      meshAgent -subsys eRT. &

  4. Z-Sivaraman

    Hi Narayanaswamy Ramaiyer


    root@Docsis-Gateway:~# which meshAgent
    /usr/bin/meshAgent
    root@Docsis-Gateway:~#


    Yes. it crashes for both the commands you specified.

    mesh-agent we tried is from rdk-next and rdkb2020q2

  5. Narayanaswamy Ramaiyer

    Hi Z-Sivaraman

    OK , so the executable in /usr/bin/meshAgent is the working one where as the one in ccsp path is causing issues. Let us check more on this

  6. Z-Sivaraman

    Narayanaswamy Ramaiyer

    When I run the meshAgent process, by having current working directory as "/usr/ccsp/mesh", it runs properly. I think, only from this directory, it is able to get proper LIB/Files.

    Any thoughts why memory leak happens, when my pwd is other than "/usr/ccsp/mesh"

    1. Narayanaswamy Ramaiyer

      Z-Sivaraman

      Oh, so when you run it from /usr/ccsp/mesh it is running fine? I mean, do you run

      cd /usr/ccsp/mesh

      mesh # meshAgent -subsys eRT. &


      or 

      cd /usr/ccsp/mesh

      mesh # ./meshAgent -subsys eRT. &

  7. Z-Sivaraman

    Narayanaswamy Ramaiyer

    Yes.. Both works, when we run inside "cd /usr/ccsp/mesh"