Breakpad is a library and tool suite that allows to distribute an application to users with compiler-provided debugging information removed. Breakpad library is linked with application which is executed on platform. When application crashes , it produces compact "minidump" files. These minidumps are send back to server and produce C and C++ stack traces.

Breakpad Capabilities

• Removes debug symbols  from client code.

• (Client) Writes minidump file with thread context.
• (Client) Submits minidump to Crash Collector.
• (Crash Collector) Reconstructs human readable stack trace.

When compiled, the stack produces library (libbreakpad_client.a) which should be linked with the test application. When application crashes, it creates minidump files. There are breakpad utilities like dump_syms and minidump_stackwalker which are used to analyze the minidump.

Setting up Breakpad

Cross-Compile Breakpad library for target board (eg:RPI)

• git clone https://chromium.googlesource.com/breakpad/breakpad
• ./configure --host=arm-rdk-linux-gnueabi --prefix=/usr
• make

After successful compilation, “libbreakpad_client.a” will be created in “src/client/linux/” directory.

Compile Breakpad for PC to get utilities to analyze minidump

• git clone https://chromium.googlesource.com/breakpad/breakpad
• ./configure 
• make

After successful compilation, executables dump_syms and minidump_stackwalk will be created in “src/tools/linux/” directory.

Steps to link google breakpad and create minidump

1. Create a sample app 

   vim breakpad_exercise.c

2. Include header file for exception handler
   #include "client/linux/handler/exception_handler.h"

3. Add breakpad handler in your application. This is the callback function which will be executed when a crash occurs:
   

4. In main() function, add the handler and register it. Instantiate exception handler for breakpad with the path where minidumps will be created. Here, current directory (“./ ”) where sample app is present & executed is given as path.

5. Create a crash in a function and  call this function in main()

6. Link Breakpad library and include path in Makefile

7. Run the application (which crashes and minidump gets generated)

A minidump file will be generated in the same directory:

Analyze minidump

dump_syms

Breakpad tool “dump_syms” run on  binaries to produce the text-format symbols. The minidump should be copied to server pc where dump_syms is present.

breakpad/src/tools/linux/dump_syms/dump_syms breakpad_exercise  >  breakpad_exercise.sym

Run below command on symbol file to get the first line:

head -n1 breakpad_exercise.sym

Output (for example):

MODULE Linux arm 73DC1FFAD46D0ECDC4988DBBD008BBC70 breakpad_exercise

In the ideal scenario, this symbol file will be extracted initially and uploaded to some server. The application/library without symbol will be deployed. Once crashed, the minidump will be generated which will be analyzed along with this symbol file to generate stack trace.

minidump_stackwalk

This utility will give meaningful trace from minidump and symbol file

Create directory of name of this string (code), as shown below:

mkdir  -p  symbols/breakpad_exercise/73DC1FFAD46D0ECDC4988DBBD008BBC70

Copy “breakpad_exercise.sym” file to the above path.

cp breakpad_exercise.sym symbols/breakpad_exercise/73DC1FFAD46D0ECDC4988DBBD008BBC70

Run minidump_stackwalk tool on minidump file as below to produce a symbolized stack trace

breakpad/src/processor/minidump_stackwalk  40e9abf8-19cc-4b55-cd2bb29f-dbd37900.dmp  symbols/  >  tracefile