Ranting Jacaré

A 21st century sapien and his random thoughts on binary encoded partial scripts.

© 2021. All rights reserved.

Different Ways to Run GDB

05 Jun 2021
debug
 
gdb
 

A collection of practical ways to use GNU Debugger.

For a comprehensive manual, type gdb --help or visit the official GDB Documentation. This page is my personal collection of practical commands and notes on GDB.

GDB with Scripts

In “Scripting with GDB” we explored how to use scripts to automate some of the debugging process. To use scripts

  • Set up .gdbinit. Global setups can be placed in ~/.gdbinit and project specific setups can be placed in .gdbinit under project directory where GDB is executed from;
  • Place scripts in a source file example.gdb and execute it from GDB with source example.gdb;
  • Run with option gdb --command=example.gdb.

Source and Library Hints (TODO move to practical notes)

A particular set of commands that are useful when the binary is built and run on different hosts (e.g. in production where binaries are built on dedicated build servers and run on prod hosts) are

set solib-search-path /path/to/.libs
set substitute-path /path/on/build/src /path/on/prod/src

solib-search-path instructs GDB to look into directories for shared libraries object files (e.g. *.so). Because the build server and execution server might differ in this aspect, it is not uncommon for the build servers to extract, update and pack its shared libraries object files with the binary like

$ tree package-name-1.0.0
package-name-1.0.0
├── bin
│   ├── hello_world
│   └── helper_tool
├── schemas
│   └── custom_schema.json
└── .libs 
    ├── libc++.so.1
    └── libc++abi.so.1
$ ldd bin/hello_world
    ...
    libc++.so.1 => /path/to/prod/package-name-1.1.0/bin/../.libs/libc++.so.1
    libc++abi.so.1 => /path/to/prod/package-name-1.1.0/bin/../.libs/libc++abi.so.1

This helps GDB find the correct library files.

debug or relwithdebuginfo binaries include extra debug information (e.g. mapping .text virtual addresses to source files and line numbers):

$ diff <(objdump -h release/bin) <(objdump -h relwithdebinfo/bin)
2c2
< release/bin:     file format elf64-x86-64
---
> relwithdebinfo/bin:     file format elf64-x86-64
59a60,73
>  27 .debug_aranges 00000040  0000000000000000  0000000000000000  0000303a  2**0
>                   CONTENTS, READONLY, DEBUGGING, OCTETS
>  28 .debug_info   0000b156  0000000000000000  0000000000000000  0000307a  2**0
>                   CONTENTS, READONLY, DEBUGGING, OCTETS
>  29 .debug_abbrev 000009ca  0000000000000000  0000000000000000  0000e1d0  2**0
>                   CONTENTS, READONLY, DEBUGGING, OCTETS
>  30 .debug_line   00000500  0000000000000000  0000000000000000  0000eb9a  2**0
>                   CONTENTS, READONLY, DEBUGGING, OCTETS
>  31 .debug_str    0000ecc4  0000000000000000  0000000000000000  0000f09a  2**0
>                   CONTENTS, READONLY, DEBUGGING, OCTETS
>  32 .debug_loc    0000002e  0000000000000000  0000000000000000  0001dd5e  2**0
>                   CONTENTS, READONLY, DEBUGGING, OCTETS
>  33 .debug_ranges 000000a0  0000000000000000  0000000000000000  0001dd8c  2**0
>                   CONTENTS, READONLY, DEBUGGING, OCTETS

GDB needs either

  1. have the source files placed in the same path as when built to correctly display the source code (common when built and run on the same server), or
  2. use substitute-path to redirect source files to different path.

These commands can be put into a project’s .gdbinit where gdb is run from so they are automatically executed for any GDB instance run from that directory. For example, for the setup below

$ ls -lah
archive/
package_root/ -> archive/package-name-1.0.0/
config/
$ tree archive
archive
├── package-name-1.0.0
│   ├── .libs
│   ├── bin
│   ├── schemas
│   └── src 
└── package-name-1.2.0
    ├── .libs
    ├── bin
    ├── schemas 
    └── src

we can have

#.gdbinit
set solib-search-path package_root/.libs
set substitute-path /build/path/src package_root/src

and only re-link package_root to different package versions from archive.

Run with Args and Wrappers and Live Instances

To run GDB on program with arguments, do

gdb --args <binary> <arg1> <arg2> <argN>

To add wrappers, prepend them to gdb, e.g.

onload gdb <binary>
capability-wrapper tasket -c 5 <gdb> <binary>

To attach GDB to a running instance, first find the instance’s PID, then

gdb <binary> <pid>
gdb <binary> -p <pid>

This is particularly useful when

  1. debugging library code, e.g. cython compiled code called from a Python program;
  2. debugging binary that requires standard input (i.e. stdin or FD 0), where we can debug the program without needing to redirect its input/output.

For the latter, if stdin is redirected from file,

gdb <binary>
(gdb) run <args> < inpufile.txt

Core Dumps

To analyze a core dump, we can run GDB with

gdb <binary> <core-file>
gdb <binary> -c <core-file>

and the rest is similar to running GDB with a binary and process.

Apart from analyzing core dumps caused by program crashes, we can also generate core dumps while running GDB:

(gdb) generate-core-file <core-file>

which can be used by later analysis.

In rare circumstances where we want to continue execution upon identifying an issue (e.g. in a production trading environment where we identify a condition that points to an issue, but not bad enough to halt trading. This issue is difficult to reproduce in testing environment and current information such as logs and network traffic does not explain its root cause, and we believe that having the full stack can be helpful in debugging.), we can obtain a snapshot of current stacks without pausing execution by forking and crashing a sub-process. The fork provides an exact copy of the state of the execution, and crashing the cloned sub-process keeps the original process intact:

if (strangeCondition && !snapshotObtained) {
  pid_t pID = fork();
  if (pID == 0) { // child
    ::abort();
  }
  else {
    snapshotObtained = true;
  }
}

Now, we can look of clues in the core dump file generated by the crashed sub-process.

GDB Server

When desired to run GDB and the binary from different hosts, we can make use of GDBServer. This can be useful when the host for the binary is limited in terms of GDB capability. This setup requires to run GDBServer on the target host

gdbserver :<port-number>

and from the debugging host, run

(gdb) target remote <host>:<port>

This can also be integrated with external tools. For example, from the debugging host, we can integrate CLion:

Related Posts