coretrace

Introduction

Coretrace is a lightweight tool for debugging of embedded Linux applications. It works by analysing core files from crashed applications and outputs a short plain-text backtrace, suitable for putting into logfiles.

The basic idea is to let the failing unit do self analysis of core files. When your remote application crashes it would of course be nice if one could get the whole core home to do some real debugging with e.g. gdb, but unfortunately this is not an option for many embedded systems. Your upstream bandwidth might be to limited, the central management software may perhaps don't support binary data or the customer might simply not like passing files from his network around the globe. In any of these cases coretrace will probably be a "good enough" tradeoff. It's not a tool used in the early development stage but rather in maintenance. Being a utility in production systems it can be used for long term stability tests or tracking down rare crashes.

Non-interactive small-sized embedded systems is the primary target, thus small footprint is of major concern, it will be stored on flash. Currently the compiled size is approximately 20 kb.

Basic usage

A simple backtrace:

$ coretrace -e myapp -c core
Insert the trace into syslog with a pipe:

$ coretrace -e myapp -c core | logger
How to check for crashes and launch your application in one stroke:

$ [ -f core ] && \
  coretrace -e myapp -c core >crash.log; \
  rm -f core; \
  ./myapp

Your application can then easily read the logfile (/tmp/crash.log) and alert an operator.

Screenshot, read explanation of the output in tips below.

Automatic usage

Kernel v2.6.19 and later can be configured to invoke a core file analyzer automaticaly when a crash occur. There is no need to actually write the core file to disk any more, instead one can have it piped to the standard input of coretrace in real-time and have the resulting backtrace end up in the syslog. Read core(5) and "Piping core dumps to a program" for additional information. Note: you need to execute these commands as the root user.

For kernel v2.6.24 and later:

$ echo "|/usr/bin/coretrace -e /proc/%p/exe" >/proc/sys/kernel/core_pattern
For kernel v2.6.19 to v2.6.23:

$ echo "|/usr/bin/coretrace" >/proc/sys/kernel/core_pattern

Supported platforms

Runs on x86, x86-64 (AMD64), ARM, PowerPC and Etrax (cris) Linux systems.

Requirements

Compile your application with framepointers enabled! These general CFLAGS are recommended in your Makefile:

CFLAGS += -g -fno-omit-frame-pointer -fno-optimize-sibling-calls
CFLAGS += -fno-strict-aliasing -fno-crossjumping
CFLAGS += -falign-functions
The enabled debuging has a price though, performance of you app may (very) slightly decrease.
For the x86 and x86-64 architectures this additional CFLAG is nice too:

CFLAGS += -mno-omit-leaf-frame-pointer
For the ARM architecture these additional CFLAGS are nice too:
CFLAGS += -fno-short-enums -mapcs-frame -mno-sched-prolog
For the PowerPC architecture these additional CFLAGS are nice too:
CFLAGS += -mcpu=powerpc -m32 -meabi
For the CRISv10 architecture these additional CFLAGS are nice too:
CFLAGS += -m32bit -mstack-align -mlinux
Your target system must use ELF binaries as default (most Linux systems do).

Downloads

Get your copy here

Building

For the x86 and x86-64 support, unpack the archive and run:

$ tar -xvzf coretrace.tar.gz
$ make
For the ARM support one must first give some arguments since there are many different names and flavors of cross-gcc's. An example of cross building for recent Debian ARM-ports:

$ tar -xvzf coretrace.tar.gz
$ make CC=arm-none-linux-gnueabi-gcc coretrace-arm
For the PowerPC support one must first give some arguments since there are many different names and flavors of cross-gcc's. An example of building for MPC8xx:

$ tar -xvzf coretrace.tar.gz
$ make CC=powerpc-860-linux-gnu-gcc coretrace-ppc
For the Cris support, unpack the archive and run:

$ tar -xvzf coretrace.tar.gz
$ make coretrace-cris
There is no "make install" target since this is way to dependant upon your particular embedded system. You need to copy coretrace-xxx into /usr/bin yourself.

License

coretrace is GPL.

Authors

Tips

The output from coretrace might look complicated since it's mostly a bunch of hex digits but there's a motive for this. Since we're targeting embedded systems all binarys are (usually) striped and debugsymbols removed. You therefore need two copys of your application binary, one stripped and one non-striped. Whenever you receive a crashlog you can then translate those digits into readable code with the help of the non-striped copy. Use one of these standard tools from binutils:

$ addr2line -fse myapp [addr addr ...]
$ objdump -S myapp | less
Don't forget to enable core files in your system! Many embedded hosts have them turned off by default. Use setrlimit(2) or ulimit -c unlimited for this.
Disable randomized dynamic linking to be able to trace craches in librarys (*.so files). Note that this incur a slight security risk though.
"echo 0 >/proc/sys/kernel/randomize_va_space"
If your app is started from a read-only fs you probably need to symlink "core" into /tmp or a core file might not be generated by the kernel.
Some systems create core files with dynamic names. Issue
"echo 0 >/proc/sys/kernel/core_uses_pid" at boot to prevent this.
Remember that debugging can be more troublesome with compiler optimization enabled. If you think your codeflow looks strange, try reducing optimization with
CFLAGS += -O1.

Bugs / known issues

Help is appreciated

Debug of x86 applications doesn't work when built for x86-64.
Signal trampolines ain't handled yet.
To simple build/makefile. Could easily fail.
Only C source code compiled with gcc has been tested enough.

Todo

Help is appreciated

Thread support. Even though this ain't frequently used in embedded systems it would be nice.
Better C++ support by adding parsing of DWARF2 stack unwind runtime information.