Every tool in this module ends with a stack trace, and they all read the same way. Learning to read one quickly is the highest-value skill in this module.

What a stack is

Lesson 8.1 introduced the call stack: each active function has a frame holding its parameters and locals. A stack trace is a snapshot of that stack at the moment something went wrong.

#0  the function where it happened
#1  the function that called #0
#2  the function that called #1
...
#n  main

Frame #0 is the most recent, and reading upward from the bottom gives you the path in.

A real crash

// CRASHES ON PURPOSE — build it and run it to see the segfault for yourself.
#include <iostream>
#include <string>
#include <vector>

struct Record {
    std::string name;
};

const Record* find_record(const std::vector<Record>& records, const std::string& name) {
    for (const Record& r : records) {
        if (r.name == name) {
            return &r;
        }
    }
    return nullptr;                     // not found
}

void print_record(const Record* r) {
    std::cout << "name: " << r->name << "\n";   // BUG: no null check
}

int main() {
    std::vector<Record> records = {{"alice"}, {"bob"}};
    print_record(find_record(records, "carol"));   // returns nullptr
    return 0;
}

Run it and the whole diagnosis is one word:

Segmentation fault

Exit code 139. No line, no variable, nothing. This is why you reach for a tool.

The trace

Built with -g and run under GDB:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7d6bd44 in std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, ...>
    (std::basic_ostream<char, ...>&, std::__cxx11::basic_string<char, ...> const&) ()
   from /lib/x86_64-linux-gnu/libstdc++.so.6

(gdb) backtrace
#0  0x00007ffff7d6bd44 in std::operator<< <char, ...> () from /lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x000055555555643f in print_record (r=0x0) at crash.cpp:19
#2  0x0000555555556599 in main () at crash.cpp:24

That is genuinely intimidating the first time. Here is how to read it in ten seconds.

The technique: find the first frame that is yours

Frame #0 is inside libstdc++. That is almost never where the bug is — the standard library is not broken. Scan down until you reach a frame with your file name on it:

#1  0x000055555555643f in print_record (r=0x0) at crash.cpp:19
                                       ^^^^^         ^^^^^^^^^^^^
                                       the value     your file and line

crash.cpp:19 is the line. And GDB has printed the argument: r=0x0 — the pointer is null.

Frame #2 tells you how you got there: main at line 24, which is the call print_record(find_record(records, "carol")). find_record returned nullptr because "carol" is not in the list, and nothing checked.

Three lines of trace gave you the bad value, the line that used it, and the call that produced it.

The rule

Frame #0 is where it crashed. The first frame in your own code is usually where the bug is.

A crash deep inside std::string, std::vector, or malloc almost always means you handed the library something invalid. The library is the victim, not the culprit.

Template names, and how to skim them

Standard library frames look like this in full:

std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char,
std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char,
std::char_traits<char> >&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)

Mentally delete everything inside <> and you get:

std::operator<<(std::basic_ostream&, std::basic_string const&)

Printing a string to a stream. std::basic_string<char, ...> is std::string; std::basic_ostream<char, ...> is std::ostream. Those two substitutions make most standard library frames readable.

The same trace from other tools

ASan puts the newest frame first, same as GDB:

    #0 0x... in sum_range(int const*, int) buggy.cpp:6
    #1 0x... in main buggy.cpp:17

Valgrind reads top-down with at for the innermost frame and by for its callers:

==1653==    at 0x1091DC: sum_range(int const*, int) (buggy.cpp:6)
==1653==    by 0x109268: main (buggy.cpp:17)

Same information, three layouts. at is Valgrind's #0.

When there are no line numbers

If frames show addresses and ?? instead of names and lines, you built without -g:

#0  0x0000555555555189 in ?? ()
#1  0x00007ffff7829d90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Rebuild with -g and run again. Also prefer -O0: at -O2 the compiler inlines functions, so frames disappear from the trace entirely and the one you want may not be there.

Core dumps

For a crash you cannot reproduce interactively, save the corpse:

ulimit -c unlimited          # allow core files
./program                    # crashes, writes a core file
gdb ./program core           # inspect it after the fact

Inside GDB, backtrace works exactly as before. This is how you debug a crash that happened on a server hours ago.

A checklist

1. Find the first frame in your own code. Skip library frames.
2. Read the arguments GDB printed. r=0x0 is the answer more often than not.
3. Read frame #0's error type: SIGSEGV is a bad pointer; SIGABRT is usually an uncaught exception or a failed assertion.
4. Walk outward to see how the bad value was produced.
5. No line numbers? Rebuild with -g -O0.

Try it

Checkpoint

• A stack trace is the call stack at the moment of failure; #0 is the most recent frame.
• Skip library frames and find the first frame in your own code — that is usually the bug.
• The printed arguments often contain the answer outright, such as r=0x0.
• A crash inside std::string or malloc means you handed the library something invalid.
• Mentally delete everything inside <> to read standard library frames.
• Valgrind's at is #0, and by lines are its callers.
• ?? and no line numbers means rebuild with -g -O0.
• SIGSEGV is a bad pointer; SIGABRT is usually an exception or assertion.