Fun to Program -- Debug: level 3

This was originally written and created around 2013 and may require to be updated. (2021)

Debug: level 3

The GNU debugger (GDB) make you look into binary programs.

If you do not mind reading the code in the assembler, no source code is required.

The GDB works even better if the program is compiled with the -g option and the source code is kept in place on the same machine after the compilation with the -g option. See ELF: Compile hello-gdb.

GDB commands

Please have GDB QUICK REFERENCE with you.

Here are some basic GDB commands.

command	alias	meaning
`help COMMAND`	`h`	Print help for `COMMAND`.
`file FILE`	`fil`	Use `FILE` as the program to be debugged.
`set VAR = EXP`		Evaluate `EXP` and assign its result to `VAR`.
`show VAR`	`sho`	Show `VAR` contents.
`set trace-commands on\|off`		Set/reset tracing of GDB CLI commands
`set substitute-path FROM TO`		Set a substitution rule replacing `FROM` into `TO` in source file paths. (`unset ...`)
`set args ARGS`		Set command arguments `ARGS` to be used when GDB runs a program to be debugged.
`set env VAR VALUE`		Set environment variable `VAR` to `VALUE`. (`unset ...`)
`directory DIR ...`	`dir`	Add `DIR` to the search path for source files (`:` separated). Reset the search path to the default, if invoked without `DIR`.
`run`	`r`	Start debugged program.
`run ARGS`	`r`	Start debugged program with its command arguments `ARGS`.
`start`	`star`	Run the debugged program like `run` but stop at the beginning of `main`.
`cont`	`c`	Resume execution, after signal or breakpoint.
`next`	`n`	Step to the next source code line (proceed through soubroutine call)
`nexti`	`ni`	Step to the next instruction line (proceed through soubroutine call)
`step`	`s`	Step one source code line, step into subroutine call as needed.
`stepi`	`si`	Step one instruction line, step into subroutine call as needed.
`finish`	`fin`	Execute until selected stack frame returns.
`until`	`u`	Execute until the program reaches a source line greater than the current or a specified location within the current frame.
`kill`	`k`	Kill execution of program being debugged.
`quit`	`q`	Exit “`gdb`”.
`break [FILE:]LINE`	`b`	Set breakpoint at `LINE` [of `FUNCTION`].
`break [FILE:]LINE if COND`	`b`	Set breakpoint at `LINE` [of `FUNCTION`] which breaks only if `COND` is true.
`break * ADDRESS`	`b`	Set breakpoint at `ADDRESS`
`info break`	`i`	Status of specified breakpoints (all user-settable breakpoints if no argument).
`condition N COND`	`cond`	Specify breakpoint number `N` to break only if `COND` is true.
`tbreak [FILE:]LINE`	`tb`	Set a temporary breakpoint at `LINE` [of `FUNCTION`].
`rbreak REGEXP`	`rb`	Set a breakpoint for all functions matching `REGEXP`.
`clear`	`cl`	Clear breakpoint at specified line or function.
`delete`	`d`	Delete some breakpoints or auto-display expressions.
`enable`	`en`	Enable some breakpoints.
`disable`	`dis`	Disable some breakpoints.
`define NAME`	`def`	Define a new command NAME. End with a `end` line.
`command N`	`comm`	Set commands to be executed when a breakpoint N is hit. End with a `end` line.
`ignore N COUNT`	`ig`	Set ignore-count of breakpoint number `N` to `COUNT`.
`signal NUM`	`sig`	Resume execution with signal `NUM`. (without signal if “0”)
`catch EVENT`	`cat`	Set catchpoints to catch `EVENT`(assert catch exception exec fork syscall throw vfork).
`info catch`	`i`	Exceptions that can be caught in the current stack frame
`trace [FILE:]LINE`	`tr`	Set a tracepoint at `LINE` [of `FUNCTION`].
`info trace`	`i`	Specified watchpoints (all watchpoints if no argument).
`watch EXP`	`wa`	Stop execution whenever the value of an `EXP` changes. If `-l` is given, the value of a `*(EXP)` changes.
`info watch`	`i`	Specified watchpoints (all watchpoints if no argument).
`list`	`l`	List ten more lines after or around previous listing.
`list -`	`l`	List the ten lines before a previous ten-line listing.
`list FROM[,TO]`	`l`	List specified function or line.
`disassemble`	`disas`	Disassemble a specified section of memory.
`disassemble/m`	`disas`	Disassemble a specified section of memory with source lines (if available). (m: more code)
`disassemble/r`	`disas`	Disassemble a specified section of memory with raw instructions in hex. (r: raw code)
`info`	`i`	Generic command for showing things about the program being debugged.
`info all-reg`	`i`	List of all registers and their contents, for selected stack frame.
`info locales`	`i`	List of local variables of current stack frame.
`info reg`	`i`	List of integer registers and their contents, for selected stack frame.
`print EXP`	`p`	Print value of expression `EXP`.
`print/F EXP`	`p`	Print value of expression `EXP` with `F` format.
`printf "FMT", EXP, ...`		Print value of expression `EXP, ...` with C-style FMT.
`print-object`	`po`	Ask an Objective-C object to print itself.
`call FUNC`	`cal`	Call a function `FUNC` in the program and print its result.
`call/F FUNC`	`cal`	Call a function `FUNC` in the program and print its result with `F` format.
`ptype TYPE`	`pt`	Print definition of type `TYPE`. (unrolls any typedefs)
`whatis EXP`	`wha`	Print data type of expression `EXP`. (Only one level of typedefs is unrolled.)
`x/NFU`	`x`	Examine memory, `N` data with `F` format in `U` unit
`backtrace`	`bt`	Print backtrace of all stack frames, or innermost `COUNT` frames. (`where` == `bt`)
`backtrace full`	`bt f`	Print backtrace of all stack frames and the values of the local variables.
`frame`	`f`	Select and print a stack frame.
`down`	`do`	Select and print stack frame called by this one.
`up`		Select and print stack frame that called this one.
`attach PID`	`at`	Attach to a `PID` process outside of GDB.
`detach`	`det`	Detach a process or file previously attached to GDB.
`target`	`tar`	Connect to a target machine or process.
`overlay`	`ov`	Commands for debugging overlays.
`thread N`	`t`	Switch to the thread number `N`.
`thread apply N COMMAND`	`t a`	Apply a `COMMAND` to the thread number `N`.
`thread apply all COMMAND`	`t a a`	Apply a `COMMAND` to all threads.
`thread apply all bt full`	`t a a bt f`	Apply `bt full` to all threads.
`info thread`	`i`	List of information on all currently known threads.
`advance`	`adv`	Continue the program up to the specified line or function.
`go`	`g`	Go to an earlier-bookmarked point in the program’s execution history.
`jump`	`ju`	Continue program being debugged at specified line or address.
`generate-core-file`	`gcore`	Save a core file with the current state of the debugged process.
`shell`	`!`	Execute the rest of the line as a shell command.
`cd DIR`		Set working directory to `DIR` for debugger and program being debugged.
`make`	`mak`	Run the “`make`” program using the rest of the line as arguments.
`pwd`	`pw`	Print working directory. This is used for your program as well.
`focus`	`fs`	Set focus to the next/prev/src/asm/regs/cmd window. (TUI)
`layout`	`lo`	Change the layout of windows to next/prev/src/asm/split/regs. (TUI)
`info win`	`i`	List of all displayed windows. (TUI)
`winheight W [+\|-] VAL`	`wh`	Set the height of a window, W=src/cmd/asm/regs. (TUI)

Formating option F is used with call/F, print/F and x/NFU sequences:

a address
c char
d decimal
f floating point
i machine instruction (x command only)
o octal
r raw
s string (x command only)
t binary(radix two)
u unsigned decimal
x hexadecimal

Unit is used with x/NFU sequence:

b byte
h half word (2 bytes)
w word (4 bytes)
g giant word (8 bytes)

Initialization files:

System-wide init file: /etc/gdb/gdbinit
User provided init files: ~/.gdbinit -> ./.gdbinit
-x and -nx options disable default init files.

TIP: Use the “Return”-key to repeat the last command.

Detached debugging symbols

The Debian system provided library packages have no debugging symbols in them.

The dh_strip(1) is normally used to strip executables, shared libraries, and some static libraries in those packages.

The dh_strip(1) is also used to generate a separate *-dbg package with “dh_strip --dbg-package=foo-dbg”.

The low level operation to enable detached debugging symbols is described in the objcopy(1) manpage under “--only-keep-debug” and “--add-gnu-debuglink”. (You can strip binary with the strip(1) command, too.)

Examples for the path to the key file of library and its debugging symbols

package type	package name	path to the key file
library	`libfoo`	`/usr/lib/x86_64-linux-gnu/libfoo.so`
debug symbols	`libfoo-dbg`	`/usr/lib/debug/lib/x86_64-linux-gnu/libfoo.so`

When ever GDB executes a function in the stripped library /usr/lib/x86_64-linux-gnu/libfoo.so file, GDB shipped with Debian obtains its debugging symbols from the /usr/lib/debug/lib/x86_64-linux-gnu/libfoo.so file.

TIP: Debian wanted to strip debug symbols from the shipped binary packages while FSF wished to keep them. This was one of the reason for Debian to become independent of FSF. Now we have detached debugging symbol packages which solves this difference. See Debian’s Debugging Debacle: the Debrief.

Source file path

GDB also use the corresponding C source file path and C source line information stored in the ELF file as described in ELF: readelf -wL to display them at the proper address.

GDB searches file, e.g., foo.c as follows without recursion:

Exactly foo.c in the current working directory.
Some file location based on the file location recorded in the ELF file and the substitution rule.
- Exactly /usr/src/foo-1.0/lib/foo.c as recorded in the ELF file. (default)
- Exactly /home/bar/foo-1.0/lib/foo.c with “set substitute-path /usr/src /home/bar”.
Exactly /mnt/cross/foo.c if the source path is set to /mnt/cross.
- GDB command prompt: “directory /mnt/cross” (: separated list possible)
- GDB command argument: “gdb -d /mnt/cross” (multiple -d options possible)

If you are debugging a locally compiled binary, the C source file path stored in the ELF and the actual C source file path should match. So GDB should have no difficulty finding them.

If you are debugging a packaged binary, you should obtain its corresponding source package and expand it into a source tree, first. For this case, the C source file path stored in the compiled ELF and the actual C source file path on your system should be different but most likely should share taling portion of the path.

Use of the substitution rule is usually the wise choice and the most elegant one. But sometimes, it may be required to use a brute force approach:

$ gdb $(find src_path -type d -printf '-d %p ') prog

gdb hello-gdb

Here we practice to walk through a program hello-gdb with GDB.

Let’s start loading hello-gdb ELF binary generated by GCC with the -g option in ELF: Compile hello-gdb to GDB.

$ gdb hello-gdb
GNU gdb (GDB) 7.6 (Debian 7.6-5)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /path/to/c/hello-gdb...done.

Now you are at a interactive GDB prompt with (gdb).

Let’s set up several breakpoints.

(gdb) list
1    #include <stdio.h>
2    #include <stdlib.h>
3    /* my first C program */
4    int main()
5    {
6        printf("Hello, world!\n");
7        return EXIT_SUCCESS;
8    }
(gdb) break 3
Breakpoint 1 at 0x400501: file hello.c, line 3.
(gdb) break 5
Breakpoint 2 at 0x400501: file hello.c, line 5.
(gdb) break 6
Breakpoint 3 at 0x400501: file hello.c, line 6.
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000000000400501 in main at hello.c:3
2       breakpoint     keep y   0x0000000000400501 in main at hello.c:5
3       breakpoint     keep y   0x0000000000400501 in main at hello.c:6

As you see, not all C source lines translate to independent lines. Breakpoint 3 and 5 stops at the same address.

Let’s run program while stopping at line 3, 5 , and 6.

(gdb) run
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x2aaaaaacc000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?

Breakpoint 1, main () at hello.c:6
6        printf("Hello, world!\n");
(gdb) cont
Hello, world!
[Inferior 1 (process 30593) exited normally]
(gdb) quit
$

Here “inferior” is an object generated by GDB to represent the state of each program and typically corresponds to a process but more general.

Let’s start again.

make sure to install the libc6-dbg:amd64 package containing “Embedded GNU C Library: detached debugging symbols”.
start GDB less noisily with the -q option.
list source in the assembly code in various ways.

$ COLUMNS=80 dpkg -l libc6-dbg:amd64
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  libc6-dbg:amd6 2.17-92      amd64        Embedded GNU C Library: detached
$ gdb -q hello-gdb
(gdb) list
1    #include <stdio.h>
2    #include <stdlib.h>
3    /* my first C program */
4    int main()
5    {
6        printf("Hello, world!\n");
7        return EXIT_SUCCESS;
8    }
(gdb) disas main
Dump of assembler code for function main:
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>
   0x000000000040050b <+14>:    mov    $0x0,%eax
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq
End of assembler dump.
(gdb) disas/m main
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
   0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.
(gdb) disas/r main
Dump of assembler code for function main:
   0x00000000004004fd <+0>:    55    push   %rbp
   0x00000000004004fe <+1>:    48 89 e5    mov    %rsp,%rbp
   0x0000000000400501 <+4>:    bf c4 05 40 00    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    e8 d5 fe ff ff    callq  0x4003e0 <puts@plt>
   0x000000000040050b <+14>:    b8 00 00 00 00    mov    $0x0,%eax
   0x0000000000400510 <+19>:    5d    pop    %rbp
   0x0000000000400511 <+20>:    c3    retq
End of assembler dump.

For GDB, just typing disas is as good as typing disassemble.

GDB also supports TAB completion to expand disas+TAB into disassemble.

It is quite ammusing to see some hex codes such as c3 (ASCII character “]”) in amd64 which are the same ones as i386 with the same meaning “return from a function”.

Let’s read the assembler code here. The %rdi register holds the 1st function argument passed to the puts function according to the ABI described in GCC: Assembler code. Its value is set to 0x40067c.

Let’s see what is there using the examine memory command for string.

(gdb) x/s 0x40067c
0x40067c:    "D"
(gdb) x/14x 0x40067c
0x40067c:    0x44    0x00    0x00    0x00    0x7d    0xfe    0xff    0xff
0x400684:    0x15    0x00    0x00    0x00    0x00    0x41

OK. This memory data matches the original C function definition.

Let’s trace program execution a bit slowly this time.

(gdb) start
Temporary breakpoint 1 at 0x400501: file hello.c, line 6.
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?

Temporary breakpoint 1, main () at hello.c:6
6        printf("Hello, world!\n");
(gdb) disas/m
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
=> 0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
   0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.

You are at the start of the main function.

There are few basic stepping commands to walk through program with GDB.

The next command goes to the next line in the C source without stepping into the child function.
The step command goes to the next line in the C source possibly stepping into the child function.
The nexti and stepi commands moves slowly by the line based on the assembler code instead of the C source.

Let’s trace slowly to play with calling of printf.

Verify initial value of the %edi register with the info reg ... command.
Move to the next assembler code line with nexti.
Verify the first function argument value in the %edi register with the info reg ... command.
Tweak the %edi register value to point 1 byte forward to show the power of GDB.
Verify the updated first function argument value.

(gdb) info reg edi
edi            0x1    1
(gdb) nexti
0x0000000000400506    6        printf("Hello, world!\n");
(gdb) disas/m
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
=> 0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
   0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.
(gdb) info reg edi
edi            0x4005c4    4195780
(gdb) print /x $edi
$1 = 0x4005c4
(gdb) x/s $edi
0x4005c4:    "Hello, world!"
(gdb) set $edi=$edi+1
(gdb) x/s $edi
0x4005c5:    "ello, world!"

Let’s step into the printf function with step.

(gdb) step
_IO_puts (str=0x4005c5 "ello, world!") at ioputs.c:34
34         const char *str;
(gdb) disas/m
Dump of assembler code for function _IO_puts:
34         const char *str;
=> 0x00002aaaaad3c2e0 <+0>:    push   %r12
   0x00002aaaaad3c2e2 <+2>:    mov    %rdi,%r12
   0x00002aaaaad3c2e5 <+5>:    push   %rbp
   0x00002aaaaad3c2e6 <+6>:    push   %rbx

35    {
   0x00002aaaaad3c409 <+297>:    mov    $0xffffffff,%ebp
   0x00002aaaaad3c40e <+302>:    jmp    0x2aaaaad3c3bd <_IO_puts+221>

36      int result = EOF;
   0x00002aaaaad3c2e7 <+7>:    callq  0x2aaaaad53080 <__strlen_sse2>
   0x00002aaaaad3c2f3 <+19>:    mov    %rax,%rbp

37      _IO_size_t len = strlen (str);
   0x00002aaaaad3c2ec <+12>:    mov    0x33a43d(%rip),%rbx        # 0x2aaaab0767
30 <stdout>
   0x00002aaaaad3c2f6 <+22>:    mov    (%rbx),%eax
   0x00002aaaaad3c2f8 <+24>:    mov    %rbx,%rdi
   0x00002aaaaad3c2fb <+27>:    and    $0x8000,%eax
   0x00002aaaaad3c300 <+32>:    jne    0x2aaaaad3c358 <_IO_puts+120>
   0x00002aaaaad3c302 <+34>:    mov    0x88(%rbx),%r8
   0x00002aaaaad3c309 <+41>:    mov    %fs:0x10,%rdx
   0x00002aaaaad3c312 <+50>:    cmp    0x8(%r8),%rdx
   0x00002aaaaad3c316 <+54>:    je     0x2aaaaad3c410 <_IO_puts+304>
   0x00002aaaaad3c31c <+60>:    mov    $0x1,%esi
   0x00002aaaaad3c321 <+65>:    cmpl   $0x0,0x33e6cc(%rip)        # 0x2aaaab07a9
f4 <__libc_multiple_threads>
   0x00002aaaaad3c328 <+72>:    je     0x2aaaaad3c337 <_IO_puts+87>
   0x00002aaaaad3c32a <+74>:    lock cmpxchg %esi,(%r8)
   0x00002aaaaad3c32f <+79>:    jne    0x2aaaaad3c468 <_L_lock_50>
   0x00002aaaaad3c335 <+85>:    jmp    0x2aaaaad3c341 <_IO_puts+97>
   0x00002aaaaad3c337 <+87>:    cmpxchg %esi,(%r8)
   0x00002aaaaad3c33b <+91>:    jne    0x2aaaaad3c468 <_L_lock_50>
   0x00002aaaaad3c341 <+97>:    mov    0x88(%rbx),%r8
   0x00002aaaaad3c348 <+104>:    mov    0x33a3e1(%rip),%rdi        # 0x2aaaab076
730 <stdout>
   0x00002aaaaad3c34f <+111>:    mov    %rdx,0x8(%r8)
   0x00002aaaaad3c353 <+115>:    addl   $0x1,0x4(%r8)
   0x00002aaaaad3c410 <+304>:    mov    %rbx,%rdi
   0x00002aaaaad3c413 <+307>:    jmpq   0x2aaaaad3c353 <_IO_puts+115>

38      _IO_acquire_lock (_IO_stdout);
39
   0x00002aaaaad3c400 <+288>:    cmp    $0xffffffff,%eax
   0x00002aaaaad3c403 <+291>:    je     0x2aaaaad3c370 <_IO_puts+144>

40      if ((_IO_vtable_offset (_IO_stdout) != 0
   0x00002aaaaad3c358 <+120>:    mov    0xc0(%rdi),%eax
   0x00002aaaaad3c35e <+126>:    test   %eax,%eax
   0x00002aaaaad3c360 <+128>:    jne    0x2aaaaad3c400 <_IO_puts+288>
   0x00002aaaaad3c366 <+134>:    movl   $0xffffffff,0xc0(%rdi)

41           || _IO_fwide (_IO_stdout, -1) == -1)
   0x00002aaaaad3c370 <+144>:    mov    0xd8(%rdi),%rax
   0x00002aaaaad3c377 <+151>:    mov    %rbp,%rdx
   0x00002aaaaad3c37a <+154>:    mov    %r12,%rsi
   0x00002aaaaad3c37d <+157>:    callq  *0x38(%rax)
   0x00002aaaaad3c380 <+160>:    cmp    %rax,%rbp
   0x00002aaaaad3c383 <+163>:    jne    0x2aaaaad3c409 <_IO_puts+297>

42          && _IO_sputn (_IO_stdout, str, len) == len
   0x00002aaaaad3c389 <+169>:    mov    0x33a3a0(%rip),%rdi        # 0x2aaaab076
730 <stdout>
   0x00002aaaaad3c390 <+176>:    mov    0x28(%rdi),%rax
   0x00002aaaaad3c394 <+180>:    cmp    0x30(%rdi),%rax
   0x00002aaaaad3c398 <+184>:    jae    0x2aaaaad3c418 <_IO_puts+312>
   0x00002aaaaad3c39e <+190>:    movb   $0xa,(%rax)
   0x00002aaaaad3c3a1 <+193>:    add    $0x1,%rax
   0x00002aaaaad3c3a5 <+197>:    mov    %rax,0x28(%rdi)
   0x00002aaaaad3c418 <+312>:    mov    $0xa,%esi
   0x00002aaaaad3c41d <+317>:    callq  0x2aaaaad46ea0 <__GI___overflow>
   0x00002aaaaad3c422 <+322>:    add    $0x1,%eax
   0x00002aaaaad3c425 <+325>:    je     0x2aaaaad3c409 <_IO_puts+297>
   0x00002aaaaad3c427 <+327>:    jmpq   0x2aaaaad3c3a9 <_IO_puts+201>

43          && _IO_putc_unlocked ('\n', _IO_stdout) != EOF)
   0x00002aaaaad3c3a9 <+201>:    add    $0x1,%rbp
   0x00002aaaaad3c3ad <+205>:    mov    $0x7fffffff,%eax
   0x00002aaaaad3c3b2 <+210>:    cmp    $0x7fffffff,%rbp
   0x00002aaaaad3c3b9 <+217>:    cmova  %rax,%rbp

44        result = MIN (INT_MAX, len + 1);
45
46      _IO_release_lock (_IO_stdout);
47      return result;
   0x00002aaaaad3c3f6 <+278>:    pop    %rbx
   0x00002aaaaad3c3f7 <+279>:    mov    %ebp,%eax
   0x00002aaaaad3c3f9 <+281>:    pop    %rbp
   0x00002aaaaad3c3fa <+282>:    pop    %r12
   0x00002aaaaad3c3fc <+284>:    retq
   0x00002aaaaad3c3fd <+285>:    nopl   (%rax)
   0x00002aaaaad3c42c <+332>:    testl  $0x8000,(%rbx)
   0x00002aaaaad3c432 <+338>:    mov    %rax,%rsi
   0x00002aaaaad3c435 <+341>:    jne    0x2aaaaad3c460 <_IO_puts+384>

End of assembler dump.

Since required detached debugging symbols are properly installed by the libc6-dbg:amd64 package into its proper path /usr/lib/debug/lib/x86_64-linux-gnu, you see the symbol _IO_puts is resolved.

I used “apt-get source eglibc” to get the source files for the libc library used on the Debian system and copied the ioputs.c file from the eglibc-*.**/libio/ioputs.c file into the current directory in advance to enable the C source code display. You can blame me going easy but …

Let’s get out of the _IO_puts function using the finish command which finishes child function and returns to the parent function.

(gdb) finish
main () at hello.c:7
7        return EXIT_SUCCESS;
Value returned is $2 = 13
(gdb) disas/m
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
=> 0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.

Now you are back to the main function.

Let’s finish by continuing.

(gdb) cont
ello, world!
[Inferior 1 (process 30633) exited normally]
(gdb) quit
$

Please note that the program output lacks “H” in “Hello” since I tweaked the program execution via GDB.

This is just a simple walk through but you get roughly what GDB can do.

gdb prime8-gdb

The Library: libpthread had a buggy prime8.c code. Let’s check what was wrong under GDB.

Let’s compile prime8.c with -g as prime8-gdb and run it to create the core file as [ELF: Core dump]/en/2013/08/09/fun2prog-elf/#core-dump), then analyze it with GDB.

Compiling prime8.c with -g as prime8-gdb and check core under GDB.

$ gcc -g -Wall -lpthread -o prime8-gdb prime8.c
$ ulimit -c unlimited
$ ./prime8-gdb 1090
Segmentation fault (core dumped)
$ ls -l core
-rw------- 1 osamu osamu 40873984 Aug 17 23:42 core
$ gdb -q prime8-gdb core
[New LWP 29809]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./prime8-gdb 1090'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000400b09 in main (argc=2, argv=0x7fff204861b8) at prime8.c:108
108            tail->next = thd[i].head;
(gdb) print i
$1 = 63
(gdb) print tail
$2 = (primelist *) 0x0
(gdb) print thd[i].head
$3 = (primelist *) 0x0
(gdb) print thd[i-1].head
$4 = (primelist *) 0x0
(gdb) print thd[i-2].head
$5 = (primelist *) 0x2aaab8001d40
(gdb) list
103        for (i=0; i < TMAX; i++) {
104            /* TMAX thread of checkprime loop */
105            if (pthread_join(thd[i].th, (void *) NULL) ) {
106                printf ("E: error joining thread at %li\n", i);
107            }
108            tail->next = thd[i].head;
109            tail = thd[i].tail;
110        }
111
112        p=head;
(gdb) quit

Alternatively, let’s run prime8-gdb under GDB.

Compiling prime8.c with -g as prime8-gdb and running it under GDB.

$ gdb -q
(gdb) file prime8-gdb
(gdb) run 1090
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
... (snip thread dialog)

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400b09 in main (argc=2, argv=0x7fffffffdf88) at prime8.c:108
108            tail->next = thd[i].head;
(gdb) print i
$1 = 63
(gdb) print tail
$2 = (primelist *) 0x0
(gdb) print thd[i].head
$3 = (primelist *) 0x0
(gdb) print thd[i-1].head
$4 = (primelist *) 0x0
(gdb) print thd[i-2].head
$5 = (primelist *) 0x2aaaac001f40
(gdb) list
103        for (i=0; i < TMAX; i++) {
104            /* TMAX thread of checkprime loop */
105            if (pthread_join(thd[i].th, (void *) NULL) ) {
106                printf ("E: error joining thread at %li\n", i);
107            }
108            tail->next = thd[i].head;
109            tail = thd[i].tail;
110        }
111
112        p=head;
(gdb) kill
Kill the program being debugged? (y or n) [answered Y; input not from terminal]
(gdb) quit

I see the cause of the bug clearly in the result of both methods. Assignment at prime8.c:108 should not be done if no prime numbers were found (returning NULL) from the calculation in the thread. So I created prime9.patch to append only when the thread finds some prime numbers.

Creating and compiling prime9.c with -g as prime9-gdb and running it.

$ cat prime9.patch
--- prime8.c    2013-02-23 23:14:45.943466837 +0900
+++ prime9.c    2013-02-23 23:16:41.144054858 +0900
@@ -105,8 +105,10 @@
         if (pthread_join(thd[i].th, (void *) NULL) ) {
             printf ("E: error joining thread at %li\n", i);
         }
-        tail->next = thd[i].head;
-        tail = thd[i].tail;
+        if (thd[i].head != NULL) { /* prime found */
+            tail->next = thd[i].head;
+            tail = thd[i].tail;
+        }
     }

     p=head;
$ cp prime8.c prime9.c
$ patch -p0 prime9.c <prime9.patch
patching file prime9.c
$ gcc -g -Wall -lpthread -o prime9-gdb prime9.c
$ ./prime9-gdb "1090">/dev/null; echo $?
0
$ ./prime9-gdb "4"
2
3
$ ./prime9-gdb "3"
2
3
3
3
3
3
3
3
3
 ... (snip)

Oops, still something is wrong here. (I see I was careless with iteration boundaries.)

Let’s continue debugging …

gdb prime-gdb

Let’s fix the ptime9.c code by:

split this into prime.c, prime.h, checkprime.c and subthread.c files.
keep each *.c file to contain only one function in it.
make TMAX as variable tmax.
fix bugs by carefully going over iteration boundaries.
use “set trace-commands on” under the batch command execution.

prime.c

#include "prime.h"

primelist *head=NULL, *tail=NULL;
thdata       thd[TMAX];

int main(int argc, char **argv) {
    primelist *p = NULL, *q = NULL;
    long n, n_max, i, nd, tmax = TMAX;
    n_max = atol(argv[1]); /* >=3 */
    head = calloc(1, sizeof(primelist));
    tail = head;
    tail->prime = 2;
    n = 2;
    while((n - 1) * (n - 1) < n_max) {
        n++;
        if (checkprime(n)) {
            q= calloc(1, sizeof(primelist));
            tail->next = q;
            tail = q;
            tail->prime = n;
        }
    }
    nd = (n_max - n + tmax - 1) / (long) tmax;
    for (i=0; i < tmax; i++) {
        /* tmax thread of checkprime loop */
        thd[i].n0 = n + 1; /* next unchecked */
        thd[i].n1 = n + nd;
        if (thd[i].n1 >= n_max) {
            thd[i].n1 = n_max;
        }
	n = thd[i].n1;
        if (pthread_create(&thd[i].th,
                NULL, 
                (void *) subthread, 
                (void *) &(thd[i]) ) ) {
            printf ("E: error creating thread at %li\n", i);
        }
    }
    for (i=0; i < tmax; i++) {
        /* tmax thread of checkprime loop */
        if (pthread_join(thd[i].th, (void *) NULL) ) {
            printf ("E: error joining thread at %li\n", i);
        }
        if (thd[i].head != NULL) { /* prime found */
            tail->next = thd[i].head;
            tail = thd[i].tail;
        }
    }

    p=head;
    while(p) {
        printf ("%ld\n", p->prime);
        p = p->next;
    }
    p=head;
    while(p) {
        q = p->next;
	    free(p);
	    p = q;
    }
    return EXIT_SUCCESS;
}

Here, 3 lines in prime.c have been corrected (bugs!).

line 14: “<=” -> “<”
line 23: adjust the loop stepping.
line 26: start from the next integer.

prime.h

#include <stdlib.h>
#include <stdio.h>
#include <semaphore.h>
#include <pthread.h>
#define TRUE 1
#define FALSE 0
#define TMAX 64L

struct _primelist {
    long prime; 
    struct _primelist *next;
    };
typedef struct _primelist primelist;

extern primelist *head, *tail;

struct _thdata {
    pthread_t           th;
    long                n0;
    long                n1;
    primelist           *head;
    primelist           *tail;
};
typedef struct _thdata thdata;

extern thdata       thd[TMAX];

int checkprime(long n);
void subthread(thdata *thd);
int main(int argc, char **argv);

Please note the use of extern in the prime.h.

Then here are supporting sources.

checkprime.c

#include "prime.h"

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
    p = head;
    while(p) {
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
        p = p->next;
    }
    return flag;
}

subthread.c

#include "prime.h"

void subthread(thdata *thd) {
    long i;
    primelist *p=NULL, *q=NULL;
    thd->head = NULL;
    for (i = thd->n0; i <= thd->n1; i++) {
        if (checkprime(i)) {
            q = calloc(1, sizeof(primelist));
            q->prime = i;
            if (!thd->head) {
                thd->head = q;
                p = q;
            } else {
                p->next = q;
                p = q;
            }
            thd->tail = q;
        }
    }
}

Now we should have good program.

Compiling prime.c and related files with -g as prime-gdb and running it without hitting bugs.

$ gcc -g -Wall -lpthread -c prime.c
$ gcc -g -Wall -lpthread -c checkprime.c
$ gcc -g -Wall -lpthread -c subthread.c
$ gcc -g -Wall -lpthread checkprime.o subthread.o prime.o -o prime-gdb
$ ./prime-gdb "1090">/dev/null; echo $?
0
$ ./prime-gdb "3"
2
3
$ ./prime-gdb "4"
2
3
$ ./prime-gdb "5"
2
3
5
$ ./prime-gdb "6"
2
3
5

This is running nicely without hitting bugs for all input range.

Let’s run this program under the batch mode of GDB to verify its internal situation where it used to segfaults.

TIP: Use of “set trace-commands on” enables tracing of commands under the batch mode.

Running prime-gdb under GDB. (session #1)

$ cat prime.1.gdb
set trace-commands on
file prime-gdb
set arg 1090
break prime.c:44 if i >= 61
# define macro pre-definition
define xprint
print i
print tail
printf "thd[i].head=%08X\n", thd[i].head
printf "thd[i-1].head=%08X\n", thd[i-1].head
printf "thd[i-2].head=%08X\n", thd[i-2].head
end
# autorun xprint
command 1
xprint
end
# debug run start
run
cont
cont
cont
quit
$ gdb -batch -x prime.1.gdb
+file prime-gdb
+set arg 1090
+break prime.c:44 if i >= 61
Breakpoint 1 at 0x400aef: file prime.c, line 44.
+define xprint
+command 1
+run
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x2aaaaaacc000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x2aaaab497700 (LWP 30671)]
[Thread 0x2aaaab497700 (LWP 30671) exited]
[New Thread 0x2aaaab698700 (LWP 30672)]
[Thread 0x2aaaab698700 (LWP 30672) exited]
[New Thread 0x2aaaab899700 (LWP 30674)]
[Thread 0x2aaaab899700 (LWP 30674) exited]
[New Thread 0x2aaaaba9a700 (LWP 30675)]
[Thread 0x2aaaaba9a700 (LWP 30675) exited]
[New Thread 0x2aaaabc9b700 (LWP 30676)]
 ... (snip)

Breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+xprint
++print i
$1 = 61
++print tail
$2 = (primelist *) 0x2aaaac001dc0
++printf "thd[i].head=%08X\n", thd[i].head
thd[i].head=AC001DE0
++printf "thd[i-1].head=%08X\n", thd[i-1].head
thd[i-1].head=AC001D80
++printf "thd[i-2].head=%08X\n", thd[i-2].head
thd[i-2].head=AC001D20
+cont

Breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+xprint
++print i
$3 = 62
++print tail
$4 = (primelist *) 0x2aaaac001de0
++printf "thd[i].head=%08X\n", thd[i].head
thd[i].head=00000000
++printf "thd[i-1].head=%08X\n", thd[i-1].head
thd[i-1].head=AC001DE0
++printf "thd[i-2].head=%08X\n", thd[i-2].head
thd[i-2].head=AC001D80
+cont

Breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+xprint
++print i
$5 = 63
++print tail
$6 = (primelist *) 0x2aaaac001de0
++printf "thd[i].head=%08X\n", thd[i].head
thd[i].head=00000000
++printf "thd[i-1].head=%08X\n", thd[i-1].head
thd[i-1].head=00000000
++printf "thd[i-2].head=%08X\n", thd[i-2].head
thd[i-2].head=AC001DE0
+cont
2
3
5
7
11
13
... (snip)
1063
1069
1087
[Inferior 1 (process 30667) exited normally]
+quit

http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library[Native POSIX Thread Library (NTPL)] uses light-weight process (LWP) to enable the http://en.wikipedia.org/wiki/POSIX_Threads[POSIX thread (pthread)] on the modern Linux system.

Please note that the breakpoint defined with condition stops just before the previous segfault location (i = 61, 62, 63).

Let’s run this program again creating a breakpoint at the start of subthread.c. Also, let’s change tmax value from 64 to 4.

Running prime-gdb under GDB. (session #2)

$ cat prime.2.gdb
set trace-commands on
# start program with argument 1090 and stop at the first line
start 1090
# break at the start of  subthread.c
break subthread.c:1
# define macro pre-definition
define xprint
printf "thd->n0=%08X\n", thd->n0
printf "thd->n1=%08X\n", thd->n1
bt
end
# autorun xprint at breakpoint 2
command 2
xprint
end
# conditional breakpoint
break prime.c:44 if i >= 3
# one time break point
tbreak 16
c
print tmax
set tmax = 4
print tmax
bt full
s
bt f
finish
where f
c
c
c
c
info thread
thread apply all bt
c
print i
c
quit
$ gdb -batch -x prime.2.gdb prime-gdb
+start 1090
Temporary breakpoint 1 at 0x400883: file prime.c, line 7.
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x2aaaaaacc000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:7
7        primelist *p = NULL, *q = NULL;
+break subthread.c:1
Breakpoint 2 at 0x4007c0: file subthread.c, line 1.
+define xprint
+command 2
+break prime.c:44 if i >= 3
Breakpoint 3 at 0x400aef: file prime.c, line 44.
+tbreak 16
Temporary breakpoint 4 at 0x4008f3: file prime.c, line 16.
+c

Temporary breakpoint 4, main (argc=2, argv=0x7fffffffdf88) at prime.c:16
16            if (checkprime(n)) {
+print tmax
$1 = 64
+set tmax = 4
+print tmax
$2 = 4
+bt full
#0  main (argc=2, argv=0x7fffffffdf88) at prime.c:16
        p = 0x0
        q = 0x0
        n = 3
        n_max = 1090
        i = 4195920
        nd = 4197437
        tmax = 4
+s
checkprime (n=3) at checkprime.c:7
7        flag = TRUE;
+bt f
#0  checkprime (n=3) at checkprime.c:7
        p = 0x7fffffffdf80
        i = 140737488346784
        n_div_i = 0
        n_mod_i = 46912501095478
        flag = 0
#1  0x00000000004008ff in main (argc=2, argv=0x7fffffffdf88) at prime.c:16
        p = 0x0
        q = 0x0
        n = 3
        n_max = 1090
        i = 4195920
        nd = 4197437
        tmax = 4
+finish
0x00000000004008ff in main (argc=2, argv=0x7fffffffdf88) at prime.c:16
16            if (checkprime(n)) {
Value returned is $3 = 1
+where f
#0  0x00000000004008ff in main (argc=2, argv=0x7fffffffdf88) at prime.c:16
        p = 0x0
        q = 0x0
        n = 3
        n_max = 1090
        i = 4195920
        nd = 4197437
        tmax = 4
+c
[New Thread 0x2aaaab497700 (LWP 30769)]
[Switching to Thread 0x2aaaab497700 (LWP 30769)]

Breakpoint 2, subthread (thd=0x6012a0 <thd>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=00000024
++printf "thd->n1=%08X\n", thd->n1
thd->n1=0000012B
++bt
#0  subthread (thd=0x6012a0 <thd>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaab497700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+c
[New Thread 0x2aaaab698700 (LWP 30770)]
[Thread 0x2aaaab497700 (LWP 30769) exited]
[Switching to Thread 0x2aaaab698700 (LWP 30770)]

Breakpoint 2, subthread (thd=0x6012c8 <thd+40>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=0000012C
++printf "thd->n1=%08X\n", thd->n1
thd->n1=00000233
++bt
#0  subthread (thd=0x6012c8 <thd+40>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaab698700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+c
[New Thread 0x2aaaab899700 (LWP 30772)]
[Thread 0x2aaaab698700 (LWP 30770) exited]
[Switching to Thread 0x2aaaab899700 (LWP 30772)]

Breakpoint 2, subthread (thd=0x6012f0 <thd+80>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=00000234
++printf "thd->n1=%08X\n", thd->n1
thd->n1=0000033B
++bt
#0  subthread (thd=0x6012f0 <thd+80>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaab899700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+c
[Thread 0x2aaaab899700 (LWP 30772) exited]
[New Thread 0x2aaaaba9a700 (LWP 30773)]
[Switching to Thread 0x2aaaaba9a700 (LWP 30773)]

Breakpoint 2, subthread (thd=0x601318 <thd+120>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=0000033C
++printf "thd->n1=%08X\n", thd->n1
thd->n1=00000442
++bt
#0  subthread (thd=0x601318 <thd+120>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaaba9a700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+info thread
  Id   Target Id         Frame
* 5    Thread 0x2aaaaba9a700 (LWP 30773) "prime-gdb" subthread (thd=0x601318 <th
d+120>) at subthread.c:5
  1    Thread 0x2aaaaaafcfc0 (LWP 30765) "prime-gdb" main (argc=2, argv=0x7fffff
ffdf88) at prime.c:44
+thread apply all bt

Thread 5 (Thread 0x2aaaaba9a700 (LWP 30773)):
+bt
#0  subthread (thd=0x601318 <thd+120>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaaba9a700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113

Thread 1 (Thread 0x2aaaaaafcfc0 (LWP 30765)):
+bt
#0  main (argc=2, argv=0x7fffffffdf88) at prime.c:44
+c
[Thread 0x2aaaaba9a700 (LWP 30773) exited]
[Switching to Thread 0x2aaaaaafcfc0 (LWP 30765)]

Breakpoint 3, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+print i
$4 = 3
+c
2
3
5
7
11
13
... (snip)
1063
1069
1087
[Inferior 1 (process 30765) exited normally]
+quit

Please note the backtrace generated by bt:

bt lists all stack frames of the thread on which the breakpoint stopped.
- On the main thread, it reaches main.
- On the sub thread, it does not reach main but reaches NULL pointer function via start_thread() and clone().
bt full lists all stack frames and local variables of the thread.
- Uninitialized variables show bogus valus.
thead apply all bt lists all stack frames of all running threads.

See how a user defined command xprint is defined and a sequence of commands at breakpoint is auto-executed via command.

GDB TUI

Although normal command mode GDB is powerful, it is cumbersome to keep typing “list”, “print ...”, “info break”, and “info reg”, … just to see what is going on.

That is where GDB Text User Interface (TUI) comes in. Wile you are using GDB:

Type “Ctrl-X” and “Ctrl-A” (or “Ctrl-X” and “a”) to get into the TUI mode.
Type “Ctrl-X” and “Ctrl-A” (or “Ctrl-X” and “a”) again to get out of the TUI mode and back to the normal command prompt.
Change the screen layout by “la src”, “la asm”, “la split”, “la regs”, “la next”, or “la prev”. (la is short for layout)
Change the screen focus by “fs cmd”, “fs src”, “fs asm”, “fs split”, “fs regs”, “fs next”, or “fs prev”. (fs is short for focus)
Change the register window by “tui reg general”, “tui reg float”, “tui reg next”, or “tui reg all”.
Refresh the screen by “refresh” or “Ctrl-L”.
Change active window by “Ctrl-X” and “o”.
Check window status by “info win”.

For more, read “info gdb TUI” from the shell prompt.

TIP: Use “gdb -tui” or “gdbtui” in place of “gdb” to start GDB in the TUI mode.

GDB resources

Here are some GDB resources:

GDB QUICK REFERENCE (gdb-doc package, /usr/share/doc/gdb-doc/pdf/refcard.pdf.gz)
HowToGetABacktrace (Debian wiki)
DebugPackage (Debian wiki)
Using GNU’s GDB Debugger (Debian GDB package recommended)
GDB: The GNU Project Debugger (upstream)
GDB Wiki (upstream FAQ wiki)
GNU Project Debugger: More fun with GDB by William B. Zimmerly (IBM DW,Date: 03 Oct 2006)
Fun with strace and the GDB Debugger by William B. Zimmerly (IBM DW, Date: 11 May 2006)
Debugging tools and techniques for Linux on Power by Calvin Sze (IBM DW, Date: 04 Aug 2005)
Mastering Linux debugging techniques by Steve Best (IBM DW, Date: 01 Aug 2002)
Linux software debugging with GDB by David Seager (IBM DW, Date: 01 Feb 2001)

Top

Fun to Program – Debug: level 3

Date: 2013/08/12 (initial publish), 2021/08/02 (last update)

Source: en/fun2-00012.md

TOC