Breakpad is a library and tool suite that allows to distribute an application to users with compiler-provided debugging information removed. Breakpad library is linked with application which is executed on platform. When application crashes , it produces compact "minidump" files. These minidumps are send back to server and produce C and C++ stack traces.

Breakpad Capabilities

• Removes debug symbols  from client code.

• (Client) Writes minidump file with thread context.
• (Client) Submits minidump to Crash Collector.
• (Crash Collector) Reconstructs human readable stack trace.

When compiled, the stack produces library (libbreakpad_client.a) which should be linked with the test application. When application crashes, it creates minidump files. There are breakpad utilities like dump_syms and minidump_stackwalker which are used to analyze the minidump.

Setting up Breakpad

Cross-Compile Breakpad library for target board (eg:RPI)

• git clone https://chromium.googlesource.com/breakpad/breakpad
• ./configure --host=arm-rdk-linux-gnueabi --prefix=/usr
• make

After successful compilation, “libbreakpad_client.a” will be created in “src/client/linux/” directory.

Compile Breakpad for PC to get utilities to analyze minidump

• git clone https://chromium.googlesource.com/breakpad/breakpad
• ./configure 
• make

After successful compilation, executables dump_syms and minidump_stackwalk will be created in “src/tools/linux/” directory.

Steps to link google breakpad and create minidump

1. Create a sample app 

   vim breakpad_exercise.c

2. Include header file for exception handler
   #include "client/linux/handler/exception_handler.h"

3. Add breakpad handler in your application. This is the callback function which will be executed when a crash occurs:
   Image Added

4. In main() function, add the handler and register it. Instantiate exception handler for breakpad with the path where minidumps will be created. Here, current directory (“./ ”) where sample app is present & executed is given as path.

 Image Added

5. Create a crash in a function and  call this function in main()

Image Added

6. Link Breakpad library and include path in Makefile

Image Added

7. Run the application (which crashes and minidump gets generated)

Image Added

A minidump file will be generated in the same directory:

Image Added

Analyze minidump

dump_syms

Breakpad tool “dump_syms” run on  binaries to produce the text-format symbols. The minidump should be copied to server pc where dump_syms is present.

breakpad/src/tools/linux/dump_syms/dump_syms breakpad_exercise  >  breakpad_exercise.sym

Run below command on symbol file to get the first line:

head -n1 breakpad_exercise.sym

Output (for example):

MODULE Linux arm 73DC1FFAD46D0ECDC4988DBBD008BBC70 breakpad_exercise

In the ideal scenario, this symbol file will be extracted initially and uploaded to some server. The application/library without symbol will be deployed. Once crashed, the minidump will be generated which will be analyzed along with this symbol file to generate stack trace.

minidump_stackwalk

This utility will give meaningful trace from minidump and symbol file

Create directory of name of this string (code), as shown below:

mkdir  -p  symbols/breakpad_exercise/73DC1FFAD46D0ECDC4988DBBD008BBC70

Copy “breakpad_exercise.sym” file to the above path.

cp breakpad_exercise.sym symbols/breakpad_exercise/73DC1FFAD46D0ECDC4988DBBD008BBC70

Run minidump_stackwalk tool on minidump file as below to produce a symbolized stack trace

breakpad/src/processor/minidump_stackwalk  40e9abf8-19cc-4b55-cd2bb29f-dbd37900.dmp  symbols/  >  tracefile

Image Added