Fun to Program – Debug: level 3

Date: 2013/08/12 (initial publish), 2021/08/02 (last update)

Source: en/fun2-00012.md

Previous Post Top Next Post

TOC

This was originally written and created around 2013 and may require to be updated. (2021)

Debug: level 3

The GNU debugger (GDB) make you look into binary programs.

If you do not mind reading the code in the assembler, no source code is required.

The GDB works even better if the program is compiled with the -g option and the source code is kept in place on the same machine after the compilation with the -g option. See ELF: Compile hello-gdb.

GDB commands

Please have GDB QUICK REFERENCE with you.

Here are some basic GDB commands.

command alias meaning
help COMMAND h Print help for COMMAND.
file FILE fil Use FILE as the program to be debugged.
set VAR = EXP Evaluate EXP and assign its result to VAR.
show VAR sho Show VAR contents.
set trace-commands on|off Set/reset tracing of GDB CLI commands
set substitute-path FROM TO Set a substitution rule replacing FROM into TO in source file paths. (unset ...)
set args ARGS Set command arguments ARGS to be used when GDB runs a program to be debugged.
set env VAR VALUE Set environment variable VAR to VALUE. (unset ...)
directory DIR ... dir Add DIR to the search path for source files (: separated). Reset the search path to the default, if invoked without DIR.
run r Start debugged program.
run ARGS r Start debugged program with its command arguments ARGS.
start star Run the debugged program like run but stop at the beginning of main.
cont c Resume execution, after signal or breakpoint.
next n Step to the next source code line (proceed through soubroutine call)
nexti ni Step to the next instruction line (proceed through soubroutine call)
step s Step one source code line, step into subroutine call as needed.
stepi si Step one instruction line, step into subroutine call as needed.
finish fin Execute until selected stack frame returns.
until u Execute until the program reaches a source line greater than the current or a specified location within the current frame.
kill k Kill execution of program being debugged.
quit q Exit “gdb”.
break [FILE:]LINE b Set breakpoint at LINE [of FUNCTION].
break [FILE:]LINE if COND b Set breakpoint at LINE [of FUNCTION] which breaks only if COND is true.
break * ADDRESS b Set breakpoint at ADDRESS
info break i Status of specified breakpoints (all user-settable breakpoints if no argument).
condition N COND cond Specify breakpoint number N to break only if COND is true.
tbreak [FILE:]LINE tb Set a temporary breakpoint at LINE [of FUNCTION].
rbreak REGEXP rb Set a breakpoint for all functions matching REGEXP.
clear cl Clear breakpoint at specified line or function.
delete d Delete some breakpoints or auto-display expressions.
enable en Enable some breakpoints.
disable dis Disable some breakpoints.
define NAME def Define a new command NAME. End with a end line.
command N comm Set commands to be executed when a breakpoint N is hit. End with a end line.
ignore N COUNT ig Set ignore-count of breakpoint number N to COUNT.
signal NUM sig Resume execution with signal NUM. (without signal if “0”)
catch EVENT cat Set catchpoints to catch EVENT(assert catch exception exec fork syscall throw vfork).
info catch i Exceptions that can be caught in the current stack frame
trace [FILE:]LINE tr Set a tracepoint at LINE [of FUNCTION].
info trace i Specified watchpoints (all watchpoints if no argument).
watch EXP wa Stop execution whenever the value of an EXP changes. If -l is given, the value of a *(EXP) changes.
info watch i Specified watchpoints (all watchpoints if no argument).
list l List ten more lines after or around previous listing.
list - l List the ten lines before a previous ten-line listing.
list FROM[,TO] l List specified function or line.
disassemble disas Disassemble a specified section of memory.
disassemble/m disas Disassemble a specified section of memory with source lines (if available). (m: more code)
disassemble/r disas Disassemble a specified section of memory with raw instructions in hex. (r: raw code)
info i Generic command for showing things about the program being debugged.
info all-reg i List of all registers and their contents, for selected stack frame.
info locales i List of local variables of current stack frame.
info reg i List of integer registers and their contents, for selected stack frame.
print EXP p Print value of expression EXP.
print/F EXP p Print value of expression EXP with F format.
printf "FMT", EXP, ... Print value of expression EXP, ... with C-style FMT.
print-object po Ask an Objective-C object to print itself.
call FUNC cal Call a function FUNC in the program and print its result.
call/F FUNC cal Call a function FUNC in the program and print its result with F format.
ptype TYPE pt Print definition of type TYPE. (unrolls any typedefs)
whatis EXP wha Print data type of expression EXP. (Only one level of typedefs is unrolled.)
x/NFU x Examine memory, N data with F format in U unit
backtrace bt Print backtrace of all stack frames, or innermost COUNT frames. (where == bt)
backtrace full bt f Print backtrace of all stack frames and the values of the local variables.
frame f Select and print a stack frame.
down do Select and print stack frame called by this one.
up Select and print stack frame that called this one.
attach PID at Attach to a PID process outside of GDB.
detach det Detach a process or file previously attached to GDB.
target tar Connect to a target machine or process.
overlay ov Commands for debugging overlays.
thread N t Switch to the thread number N.
thread apply N COMMAND t a Apply a COMMAND to the thread number N.
thread apply all COMMAND t a a Apply a COMMAND to all threads.
thread apply all bt full t a a bt f Apply bt full to all threads.
info thread i List of information on all currently known threads.
advance adv Continue the program up to the specified line or function.
go g Go to an earlier-bookmarked point in the program’s execution history.
jump ju Continue program being debugged at specified line or address.
generate-core-file gcore Save a core file with the current state of the debugged process.
shell ! Execute the rest of the line as a shell command.
cd DIR Set working directory to DIR for debugger and program being debugged.
make mak Run the “make” program using the rest of the line as arguments.
pwd pw Print working directory. This is used for your program as well.
focus fs Set focus to the next/prev/src/asm/regs/cmd window. (TUI)
layout lo Change the layout of windows to next/prev/src/asm/split/regs. (TUI)
info win i List of all displayed windows. (TUI)
winheight W [+|-] VAL wh Set the height of a window, W=src/cmd/asm/regs. (TUI)

Formating option F is used with call/F, print/F and x/NFU sequences:

Unit is used with x/NFU sequence:

Initialization files:

TIP: Use the “Return”-key to repeat the last command.

Detached debugging symbols

The Debian system provided library packages have no debugging symbols in them.

The dh_strip(1) is normally used to strip executables, shared libraries, and some static libraries in those packages.

The dh_strip(1) is also used to generate a separate *-dbg package with “dh_strip --dbg-package=foo-dbg”.

The low level operation to enable detached debugging symbols is described in the objcopy(1) manpage under “--only-keep-debug” and “--add-gnu-debuglink”. (You can strip binary with the strip(1) command, too.)

Examples for the path to the key file of library and its debugging symbols

package type package name path to the key file
library libfoo /usr/lib/x86_64-linux-gnu/libfoo.so
debug symbols libfoo-dbg /usr/lib/debug/lib/x86_64-linux-gnu/libfoo.so

When ever GDB executes a function in the stripped library /usr/lib/x86_64-linux-gnu/libfoo.so file, GDB shipped with Debian obtains its debugging symbols from the /usr/lib/debug/lib/x86_64-linux-gnu/libfoo.so file.

TIP: Debian wanted to strip debug symbols from the shipped binary packages while FSF wished to keep them. This was one of the reason for Debian to become independent of FSF. Now we have detached debugging symbol packages which solves this difference. See Debian’s Debugging Debacle: the Debrief.

Source file path

GDB also use the corresponding C source file path and C source line information stored in the ELF file as described in ELF: readelf -wL to display them at the proper address.

GDB searches file, e.g., foo.c as follows without recursion:

If you are debugging a locally compiled binary, the C source file path stored in the ELF and the actual C source file path should match. So GDB should have no difficulty finding them.

If you are debugging a packaged binary, you should obtain its corresponding source package and expand it into a source tree, first. For this case, the C source file path stored in the compiled ELF and the actual C source file path on your system should be different but most likely should share taling portion of the path.

Use of the substitution rule is usually the wise choice and the most elegant one. But sometimes, it may be required to use a brute force approach:

$ gdb $(find src_path -type d -printf '-d %p ') prog

gdb hello-gdb

Here we practice to walk through a program hello-gdb with GDB.

Let’s start loading hello-gdb ELF binary generated by GCC with the -g option in ELF: Compile hello-gdb to GDB.

$ gdb hello-gdb
GNU gdb (GDB) 7.6 (Debian 7.6-5)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /path/to/c/hello-gdb...done.

Now you are at a interactive GDB prompt with (gdb).

Let’s set up several breakpoints.

(gdb) list
1    #include <stdio.h>
2    #include <stdlib.h>
3    /* my first C program */
4    int main()
5    {
6        printf("Hello, world!\n");
7        return EXIT_SUCCESS;
8    }
(gdb) break 3
Breakpoint 1 at 0x400501: file hello.c, line 3.
(gdb) break 5
Breakpoint 2 at 0x400501: file hello.c, line 5.
(gdb) break 6
Breakpoint 3 at 0x400501: file hello.c, line 6.
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000000000400501 in main at hello.c:3
2       breakpoint     keep y   0x0000000000400501 in main at hello.c:5
3       breakpoint     keep y   0x0000000000400501 in main at hello.c:6

As you see, not all C source lines translate to independent lines. Breakpoint 3 and 5 stops at the same address.

Let’s run program while stopping at line 3, 5 , and 6.

(gdb) run
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x2aaaaaacc000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?

Breakpoint 1, main () at hello.c:6
6        printf("Hello, world!\n");
(gdb) cont
Hello, world!
[Inferior 1 (process 30593) exited normally]
(gdb) quit
$

Here “inferior” is an object generated by GDB to represent the state of each program and typically corresponds to a process but more general.

Let’s start again.

$ COLUMNS=80 dpkg -l libc6-dbg:amd64
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  libc6-dbg:amd6 2.17-92      amd64        Embedded GNU C Library: detached
$ gdb -q hello-gdb
(gdb) list
1    #include <stdio.h>
2    #include <stdlib.h>
3    /* my first C program */
4    int main()
5    {
6        printf("Hello, world!\n");
7        return EXIT_SUCCESS;
8    }
(gdb) disas main
Dump of assembler code for function main:
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>
   0x000000000040050b <+14>:    mov    $0x0,%eax
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq
End of assembler dump.
(gdb) disas/m main
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
   0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.
(gdb) disas/r main
Dump of assembler code for function main:
   0x00000000004004fd <+0>:    55    push   %rbp
   0x00000000004004fe <+1>:    48 89 e5    mov    %rsp,%rbp
   0x0000000000400501 <+4>:    bf c4 05 40 00    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    e8 d5 fe ff ff    callq  0x4003e0 <puts@plt>
   0x000000000040050b <+14>:    b8 00 00 00 00    mov    $0x0,%eax
   0x0000000000400510 <+19>:    5d    pop    %rbp
   0x0000000000400511 <+20>:    c3    retq
End of assembler dump.

For GDB, just typing disas is as good as typing disassemble.

GDB also supports TAB completion to expand disas+TAB into disassemble.

It is quite ammusing to see some hex codes such as c3 (ASCII character “]”) in amd64 which are the same ones as i386 with the same meaning “return from a function”.

Let’s read the assembler code here. The %rdi register holds the 1st function argument passed to the puts function according to the ABI described in GCC: Assembler code. Its value is set to 0x40067c.

Let’s see what is there using the examine memory command for string.

(gdb) x/s 0x40067c
0x40067c:    "D"
(gdb) x/14x 0x40067c
0x40067c:    0x44    0x00    0x00    0x00    0x7d    0xfe    0xff    0xff
0x400684:    0x15    0x00    0x00    0x00    0x00    0x41

OK. This memory data matches the original C function definition.

Let’s trace program execution a bit slowly this time.

(gdb) start
Temporary breakpoint 1 at 0x400501: file hello.c, line 6.
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?

Temporary breakpoint 1, main () at hello.c:6
6        printf("Hello, world!\n");
(gdb) disas/m
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
=> 0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
   0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.

You are at the start of the main function.

There are few basic stepping commands to walk through program with GDB.

Let’s trace slowly to play with calling of printf.

(gdb) info reg edi
edi            0x1    1
(gdb) nexti
0x0000000000400506    6        printf("Hello, world!\n");
(gdb) disas/m
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
=> 0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
   0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.
(gdb) info reg edi
edi            0x4005c4    4195780
(gdb) print /x $edi
$1 = 0x4005c4
(gdb) x/s $edi
0x4005c4:    "Hello, world!"
(gdb) set $edi=$edi+1
(gdb) x/s $edi
0x4005c5:    "ello, world!"

Let’s step into the printf function with step.

(gdb) step
_IO_puts (str=0x4005c5 "ello, world!") at ioputs.c:34
34         const char *str;
(gdb) disas/m
Dump of assembler code for function _IO_puts:
34         const char *str;
=> 0x00002aaaaad3c2e0 <+0>:    push   %r12
   0x00002aaaaad3c2e2 <+2>:    mov    %rdi,%r12
   0x00002aaaaad3c2e5 <+5>:    push   %rbp
   0x00002aaaaad3c2e6 <+6>:    push   %rbx

35    {
   0x00002aaaaad3c409 <+297>:    mov    $0xffffffff,%ebp
   0x00002aaaaad3c40e <+302>:    jmp    0x2aaaaad3c3bd <_IO_puts+221>

36      int result = EOF;
   0x00002aaaaad3c2e7 <+7>:    callq  0x2aaaaad53080 <__strlen_sse2>
   0x00002aaaaad3c2f3 <+19>:    mov    %rax,%rbp

37      _IO_size_t len = strlen (str);
   0x00002aaaaad3c2ec <+12>:    mov    0x33a43d(%rip),%rbx        # 0x2aaaab0767
30 <stdout>
   0x00002aaaaad3c2f6 <+22>:    mov    (%rbx),%eax
   0x00002aaaaad3c2f8 <+24>:    mov    %rbx,%rdi
   0x00002aaaaad3c2fb <+27>:    and    $0x8000,%eax
   0x00002aaaaad3c300 <+32>:    jne    0x2aaaaad3c358 <_IO_puts+120>
   0x00002aaaaad3c302 <+34>:    mov    0x88(%rbx),%r8
   0x00002aaaaad3c309 <+41>:    mov    %fs:0x10,%rdx
   0x00002aaaaad3c312 <+50>:    cmp    0x8(%r8),%rdx
   0x00002aaaaad3c316 <+54>:    je     0x2aaaaad3c410 <_IO_puts+304>
   0x00002aaaaad3c31c <+60>:    mov    $0x1,%esi
   0x00002aaaaad3c321 <+65>:    cmpl   $0x0,0x33e6cc(%rip)        # 0x2aaaab07a9
f4 <__libc_multiple_threads>
   0x00002aaaaad3c328 <+72>:    je     0x2aaaaad3c337 <_IO_puts+87>
   0x00002aaaaad3c32a <+74>:    lock cmpxchg %esi,(%r8)
   0x00002aaaaad3c32f <+79>:    jne    0x2aaaaad3c468 <_L_lock_50>
   0x00002aaaaad3c335 <+85>:    jmp    0x2aaaaad3c341 <_IO_puts+97>
   0x00002aaaaad3c337 <+87>:    cmpxchg %esi,(%r8)
   0x00002aaaaad3c33b <+91>:    jne    0x2aaaaad3c468 <_L_lock_50>
   0x00002aaaaad3c341 <+97>:    mov    0x88(%rbx),%r8
   0x00002aaaaad3c348 <+104>:    mov    0x33a3e1(%rip),%rdi        # 0x2aaaab076
730 <stdout>
   0x00002aaaaad3c34f <+111>:    mov    %rdx,0x8(%r8)
   0x00002aaaaad3c353 <+115>:    addl   $0x1,0x4(%r8)
   0x00002aaaaad3c410 <+304>:    mov    %rbx,%rdi
   0x00002aaaaad3c413 <+307>:    jmpq   0x2aaaaad3c353 <_IO_puts+115>

38      _IO_acquire_lock (_IO_stdout);
39
   0x00002aaaaad3c400 <+288>:    cmp    $0xffffffff,%eax
   0x00002aaaaad3c403 <+291>:    je     0x2aaaaad3c370 <_IO_puts+144>

40      if ((_IO_vtable_offset (_IO_stdout) != 0
   0x00002aaaaad3c358 <+120>:    mov    0xc0(%rdi),%eax
   0x00002aaaaad3c35e <+126>:    test   %eax,%eax
   0x00002aaaaad3c360 <+128>:    jne    0x2aaaaad3c400 <_IO_puts+288>
   0x00002aaaaad3c366 <+134>:    movl   $0xffffffff,0xc0(%rdi)

41           || _IO_fwide (_IO_stdout, -1) == -1)
   0x00002aaaaad3c370 <+144>:    mov    0xd8(%rdi),%rax
   0x00002aaaaad3c377 <+151>:    mov    %rbp,%rdx
   0x00002aaaaad3c37a <+154>:    mov    %r12,%rsi
   0x00002aaaaad3c37d <+157>:    callq  *0x38(%rax)
   0x00002aaaaad3c380 <+160>:    cmp    %rax,%rbp
   0x00002aaaaad3c383 <+163>:    jne    0x2aaaaad3c409 <_IO_puts+297>

42          && _IO_sputn (_IO_stdout, str, len) == len
   0x00002aaaaad3c389 <+169>:    mov    0x33a3a0(%rip),%rdi        # 0x2aaaab076
730 <stdout>
   0x00002aaaaad3c390 <+176>:    mov    0x28(%rdi),%rax
   0x00002aaaaad3c394 <+180>:    cmp    0x30(%rdi),%rax
   0x00002aaaaad3c398 <+184>:    jae    0x2aaaaad3c418 <_IO_puts+312>
   0x00002aaaaad3c39e <+190>:    movb   $0xa,(%rax)
   0x00002aaaaad3c3a1 <+193>:    add    $0x1,%rax
   0x00002aaaaad3c3a5 <+197>:    mov    %rax,0x28(%rdi)
   0x00002aaaaad3c418 <+312>:    mov    $0xa,%esi
   0x00002aaaaad3c41d <+317>:    callq  0x2aaaaad46ea0 <__GI___overflow>
   0x00002aaaaad3c422 <+322>:    add    $0x1,%eax
   0x00002aaaaad3c425 <+325>:    je     0x2aaaaad3c409 <_IO_puts+297>
   0x00002aaaaad3c427 <+327>:    jmpq   0x2aaaaad3c3a9 <_IO_puts+201>

43          && _IO_putc_unlocked ('\n', _IO_stdout) != EOF)
   0x00002aaaaad3c3a9 <+201>:    add    $0x1,%rbp
   0x00002aaaaad3c3ad <+205>:    mov    $0x7fffffff,%eax
   0x00002aaaaad3c3b2 <+210>:    cmp    $0x7fffffff,%rbp
   0x00002aaaaad3c3b9 <+217>:    cmova  %rax,%rbp

44        result = MIN (INT_MAX, len + 1);
45
46      _IO_release_lock (_IO_stdout);
47      return result;
   0x00002aaaaad3c3f6 <+278>:    pop    %rbx
   0x00002aaaaad3c3f7 <+279>:    mov    %ebp,%eax
   0x00002aaaaad3c3f9 <+281>:    pop    %rbp
   0x00002aaaaad3c3fa <+282>:    pop    %r12
   0x00002aaaaad3c3fc <+284>:    retq
   0x00002aaaaad3c3fd <+285>:    nopl   (%rax)
   0x00002aaaaad3c42c <+332>:    testl  $0x8000,(%rbx)
   0x00002aaaaad3c432 <+338>:    mov    %rax,%rsi
   0x00002aaaaad3c435 <+341>:    jne    0x2aaaaad3c460 <_IO_puts+384>

End of assembler dump.

Since required detached debugging symbols are properly installed by the libc6-dbg:amd64 package into its proper path /usr/lib/debug/lib/x86_64-linux-gnu, you see the symbol _IO_puts is resolved.

I used “apt-get source eglibc” to get the source files for the libc library used on the Debian system and copied the ioputs.c file from the eglibc-*.**/libio/ioputs.c file into the current directory in advance to enable the C source code display. You can blame me going easy but …

Let’s get out of the _IO_puts function using the finish command which finishes child function and returns to the parent function.

(gdb) finish
main () at hello.c:7
7        return EXIT_SUCCESS;
Value returned is $2 = 13
(gdb) disas/m
Dump of assembler code for function main:
5    {
   0x00000000004004fd <+0>:    push   %rbp
   0x00000000004004fe <+1>:    mov    %rsp,%rbp

6        printf("Hello, world!\n");
   0x0000000000400501 <+4>:    mov    $0x4005c4,%edi
   0x0000000000400506 <+9>:    callq  0x4003e0 <puts@plt>

7        return EXIT_SUCCESS;
=> 0x000000000040050b <+14>:    mov    $0x0,%eax

8    }
   0x0000000000400510 <+19>:    pop    %rbp
   0x0000000000400511 <+20>:    retq

End of assembler dump.

Now you are back to the main function.

Let’s finish by continuing.

(gdb) cont
ello, world!
[Inferior 1 (process 30633) exited normally]
(gdb) quit
$

Please note that the program output lacks “H” in “Hello” since I tweaked the program execution via GDB.

This is just a simple walk through but you get roughly what GDB can do.

gdb prime8-gdb

The Library: libpthread had a buggy prime8.c code. Let’s check what was wrong under GDB.

Let’s compile prime8.c with -g as prime8-gdb and run it to create the core file as [ELF: Core dump]/en/2013/08/09/fun2prog-elf/#core-dump), then analyze it with GDB.

Compiling prime8.c with -g as prime8-gdb and check core under GDB.

$ gcc -g -Wall -lpthread -o prime8-gdb prime8.c
$ ulimit -c unlimited
$ ./prime8-gdb 1090
Segmentation fault (core dumped)
$ ls -l core
-rw------- 1 osamu osamu 40873984 Aug 17 23:42 core
$ gdb -q prime8-gdb core
[New LWP 29809]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./prime8-gdb 1090'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000400b09 in main (argc=2, argv=0x7fff204861b8) at prime8.c:108
108            tail->next = thd[i].head;
(gdb) print i
$1 = 63
(gdb) print tail
$2 = (primelist *) 0x0
(gdb) print thd[i].head
$3 = (primelist *) 0x0
(gdb) print thd[i-1].head
$4 = (primelist *) 0x0
(gdb) print thd[i-2].head
$5 = (primelist *) 0x2aaab8001d40
(gdb) list
103        for (i=0; i < TMAX; i++) {
104            /* TMAX thread of checkprime loop */
105            if (pthread_join(thd[i].th, (void *) NULL) ) {
106                printf ("E: error joining thread at %li\n", i);
107            }
108            tail->next = thd[i].head;
109            tail = thd[i].tail;
110        }
111
112        p=head;
(gdb) quit

Alternatively, let’s run prime8-gdb under GDB.

Compiling prime8.c with -g as prime8-gdb and running it under GDB.

$ gdb -q
(gdb) file prime8-gdb
(gdb) run 1090
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
... (snip thread dialog)

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400b09 in main (argc=2, argv=0x7fffffffdf88) at prime8.c:108
108            tail->next = thd[i].head;
(gdb) print i
$1 = 63
(gdb) print tail
$2 = (primelist *) 0x0
(gdb) print thd[i].head
$3 = (primelist *) 0x0
(gdb) print thd[i-1].head
$4 = (primelist *) 0x0
(gdb) print thd[i-2].head
$5 = (primelist *) 0x2aaaac001f40
(gdb) list
103        for (i=0; i < TMAX; i++) {
104            /* TMAX thread of checkprime loop */
105            if (pthread_join(thd[i].th, (void *) NULL) ) {
106                printf ("E: error joining thread at %li\n", i);
107            }
108            tail->next = thd[i].head;
109            tail = thd[i].tail;
110        }
111
112        p=head;
(gdb) kill
Kill the program being debugged? (y or n) [answered Y; input not from terminal]
(gdb) quit

I see the cause of the bug clearly in the result of both methods. Assignment at prime8.c:108 should not be done if no prime numbers were found (returning NULL) from the calculation in the thread. So I created prime9.patch to append only when the thread finds some prime numbers.

Creating and compiling prime9.c with -g as prime9-gdb and running it.

$ cat prime9.patch
--- prime8.c    2013-02-23 23:14:45.943466837 +0900
+++ prime9.c    2013-02-23 23:16:41.144054858 +0900
@@ -105,8 +105,10 @@
         if (pthread_join(thd[i].th, (void *) NULL) ) {
             printf ("E: error joining thread at %li\n", i);
         }
-        tail->next = thd[i].head;
-        tail = thd[i].tail;
+        if (thd[i].head != NULL) { /* prime found */
+            tail->next = thd[i].head;
+            tail = thd[i].tail;
+        }
     }

     p=head;
$ cp prime8.c prime9.c
$ patch -p0 prime9.c <prime9.patch
patching file prime9.c
$ gcc -g -Wall -lpthread -o prime9-gdb prime9.c
$ ./prime9-gdb "1090">/dev/null; echo $?
0
$ ./prime9-gdb "4"
2
3
$ ./prime9-gdb "3"
2
3
3
3
3
3
3
3
3
 ... (snip)

Oops, still something is wrong here. (I see I was careless with iteration boundaries.)

Let’s continue debugging …

gdb prime-gdb

Let’s fix the ptime9.c code by:

prime.c

#include "prime.h"

primelist *head=NULL, *tail=NULL;
thdata       thd[TMAX];

int main(int argc, char **argv) {
    primelist *p = NULL, *q = NULL;
    long n, n_max, i, nd, tmax = TMAX;
    n_max = atol(argv[1]); /* >=3 */
    head = calloc(1, sizeof(primelist));
    tail = head;
    tail->prime = 2;
    n = 2;
    while((n - 1) * (n - 1) < n_max) {
        n++;
        if (checkprime(n)) {
            q= calloc(1, sizeof(primelist));
            tail->next = q;
            tail = q;
            tail->prime = n;
        }
    }
    nd = (n_max - n + tmax - 1) / (long) tmax;
    for (i=0; i < tmax; i++) {
        /* tmax thread of checkprime loop */
        thd[i].n0 = n + 1; /* next unchecked */
        thd[i].n1 = n + nd;
        if (thd[i].n1 >= n_max) {
            thd[i].n1 = n_max;
        }
	n = thd[i].n1;
        if (pthread_create(&thd[i].th,
                NULL, 
                (void *) subthread, 
                (void *) &(thd[i]) ) ) {
            printf ("E: error creating thread at %li\n", i);
        }
    }
    for (i=0; i < tmax; i++) {
        /* tmax thread of checkprime loop */
        if (pthread_join(thd[i].th, (void *) NULL) ) {
            printf ("E: error joining thread at %li\n", i);
        }
        if (thd[i].head != NULL) { /* prime found */
            tail->next = thd[i].head;
            tail = thd[i].tail;
        }
    }

    p=head;
    while(p) {
        printf ("%ld\n", p->prime);
        p = p->next;
    }
    p=head;
    while(p) {
        q = p->next;
	    free(p);
	    p = q;
    }
    return EXIT_SUCCESS;
}

Here, 3 lines in prime.c have been corrected (bugs!).

prime.h

#include <stdlib.h>
#include <stdio.h>
#include <semaphore.h>
#include <pthread.h>
#define TRUE 1
#define FALSE 0
#define TMAX 64L

struct _primelist {
    long prime; 
    struct _primelist *next;
    };
typedef struct _primelist primelist;

extern primelist *head, *tail;

struct _thdata {
    pthread_t           th;
    long                n0;
    long                n1;
    primelist           *head;
    primelist           *tail;
};
typedef struct _thdata thdata;

extern thdata       thd[TMAX];

int checkprime(long n);
void subthread(thdata *thd);
int main(int argc, char **argv);

Please note the use of extern in the prime.h.

Then here are supporting sources.

checkprime.c

#include "prime.h"

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
    p = head;
    while(p) {
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
        p = p->next;
    }
    return flag;
}

subthread.c

#include "prime.h"

void subthread(thdata *thd) {
    long i;
    primelist *p=NULL, *q=NULL;
    thd->head = NULL;
    for (i = thd->n0; i <= thd->n1; i++) {
        if (checkprime(i)) {
            q = calloc(1, sizeof(primelist));
            q->prime = i;
            if (!thd->head) {
                thd->head = q;
                p = q;
            } else {
                p->next = q;
                p = q;
            }
            thd->tail = q;
        }
    }
}

Now we should have good program.

Compiling prime.c and related files with -g as prime-gdb and running it without hitting bugs.

$ gcc -g -Wall -lpthread -c prime.c
$ gcc -g -Wall -lpthread -c checkprime.c
$ gcc -g -Wall -lpthread -c subthread.c
$ gcc -g -Wall -lpthread checkprime.o subthread.o prime.o -o prime-gdb
$ ./prime-gdb "1090">/dev/null; echo $?
0
$ ./prime-gdb "3"
2
3
$ ./prime-gdb "4"
2
3
$ ./prime-gdb "5"
2
3
5
$ ./prime-gdb "6"
2
3
5

This is running nicely without hitting bugs for all input range.

Let’s run this program under the batch mode of GDB to verify its internal situation where it used to segfaults.

TIP: Use of “set trace-commands on” enables tracing of commands under the batch mode.

Running prime-gdb under GDB. (session #1)

$ cat prime.1.gdb
set trace-commands on
file prime-gdb
set arg 1090
break prime.c:44 if i >= 61
# define macro pre-definition
define xprint
print i
print tail
printf "thd[i].head=%08X\n", thd[i].head
printf "thd[i-1].head=%08X\n", thd[i-1].head
printf "thd[i-2].head=%08X\n", thd[i-2].head
end
# autorun xprint
command 1
xprint
end
# debug run start
run
cont
cont
cont
quit
$ gdb -batch -x prime.1.gdb
+file prime-gdb
+set arg 1090
+break prime.c:44 if i >= 61
Breakpoint 1 at 0x400aef: file prime.c, line 44.
+define xprint
+command 1
+run
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x2aaaaaacc000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x2aaaab497700 (LWP 30671)]
[Thread 0x2aaaab497700 (LWP 30671) exited]
[New Thread 0x2aaaab698700 (LWP 30672)]
[Thread 0x2aaaab698700 (LWP 30672) exited]
[New Thread 0x2aaaab899700 (LWP 30674)]
[Thread 0x2aaaab899700 (LWP 30674) exited]
[New Thread 0x2aaaaba9a700 (LWP 30675)]
[Thread 0x2aaaaba9a700 (LWP 30675) exited]
[New Thread 0x2aaaabc9b700 (LWP 30676)]
 ... (snip)

Breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+xprint
++print i
$1 = 61
++print tail
$2 = (primelist *) 0x2aaaac001dc0
++printf "thd[i].head=%08X\n", thd[i].head
thd[i].head=AC001DE0
++printf "thd[i-1].head=%08X\n", thd[i-1].head
thd[i-1].head=AC001D80
++printf "thd[i-2].head=%08X\n", thd[i-2].head
thd[i-2].head=AC001D20
+cont

Breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+xprint
++print i
$3 = 62
++print tail
$4 = (primelist *) 0x2aaaac001de0
++printf "thd[i].head=%08X\n", thd[i].head
thd[i].head=00000000
++printf "thd[i-1].head=%08X\n", thd[i-1].head
thd[i-1].head=AC001DE0
++printf "thd[i-2].head=%08X\n", thd[i-2].head
thd[i-2].head=AC001D80
+cont

Breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+xprint
++print i
$5 = 63
++print tail
$6 = (primelist *) 0x2aaaac001de0
++printf "thd[i].head=%08X\n", thd[i].head
thd[i].head=00000000
++printf "thd[i-1].head=%08X\n", thd[i-1].head
thd[i-1].head=00000000
++printf "thd[i-2].head=%08X\n", thd[i-2].head
thd[i-2].head=AC001DE0
+cont
2
3
5
7
11
13
... (snip)
1063
1069
1087
[Inferior 1 (process 30667) exited normally]
+quit

http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library[Native POSIX Thread Library (NTPL)] uses light-weight process (LWP) to enable the http://en.wikipedia.org/wiki/POSIX_Threads[POSIX thread (pthread)] on the modern Linux system.

Please note that the breakpoint defined with condition stops just before the previous segfault location (i = 61, 62, 63).

Let’s run this program again creating a breakpoint at the start of subthread.c. Also, let’s change tmax value from 64 to 4.

Running prime-gdb under GDB. (session #2)

$ cat prime.2.gdb
set trace-commands on
# start program with argument 1090 and stop at the first line
start 1090
# break at the start of  subthread.c
break subthread.c:1
# define macro pre-definition
define xprint
printf "thd->n0=%08X\n", thd->n0
printf "thd->n1=%08X\n", thd->n1
bt
end
# autorun xprint at breakpoint 2
command 2
xprint
end
# conditional breakpoint
break prime.c:44 if i >= 3
# one time break point
tbreak 16
c
print tmax
set tmax = 4
print tmax
bt full
s
bt f
finish
where f
c
c
c
c
info thread
thread apply all bt
c
print i
c
quit
$ gdb -batch -x prime.2.gdb prime-gdb
+start 1090
Temporary breakpoint 1 at 0x400883: file prime.c, line 7.
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x2aaaaaacc000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, main (argc=2, argv=0x7fffffffdf88) at prime.c:7
7        primelist *p = NULL, *q = NULL;
+break subthread.c:1
Breakpoint 2 at 0x4007c0: file subthread.c, line 1.
+define xprint
+command 2
+break prime.c:44 if i >= 3
Breakpoint 3 at 0x400aef: file prime.c, line 44.
+tbreak 16
Temporary breakpoint 4 at 0x4008f3: file prime.c, line 16.
+c

Temporary breakpoint 4, main (argc=2, argv=0x7fffffffdf88) at prime.c:16
16            if (checkprime(n)) {
+print tmax
$1 = 64
+set tmax = 4
+print tmax
$2 = 4
+bt full
#0  main (argc=2, argv=0x7fffffffdf88) at prime.c:16
        p = 0x0
        q = 0x0
        n = 3
        n_max = 1090
        i = 4195920
        nd = 4197437
        tmax = 4
+s
checkprime (n=3) at checkprime.c:7
7        flag = TRUE;
+bt f
#0  checkprime (n=3) at checkprime.c:7
        p = 0x7fffffffdf80
        i = 140737488346784
        n_div_i = 0
        n_mod_i = 46912501095478
        flag = 0
#1  0x00000000004008ff in main (argc=2, argv=0x7fffffffdf88) at prime.c:16
        p = 0x0
        q = 0x0
        n = 3
        n_max = 1090
        i = 4195920
        nd = 4197437
        tmax = 4
+finish
0x00000000004008ff in main (argc=2, argv=0x7fffffffdf88) at prime.c:16
16            if (checkprime(n)) {
Value returned is $3 = 1
+where f
#0  0x00000000004008ff in main (argc=2, argv=0x7fffffffdf88) at prime.c:16
        p = 0x0
        q = 0x0
        n = 3
        n_max = 1090
        i = 4195920
        nd = 4197437
        tmax = 4
+c
[New Thread 0x2aaaab497700 (LWP 30769)]
[Switching to Thread 0x2aaaab497700 (LWP 30769)]

Breakpoint 2, subthread (thd=0x6012a0 <thd>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=00000024
++printf "thd->n1=%08X\n", thd->n1
thd->n1=0000012B
++bt
#0  subthread (thd=0x6012a0 <thd>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaab497700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+c
[New Thread 0x2aaaab698700 (LWP 30770)]
[Thread 0x2aaaab497700 (LWP 30769) exited]
[Switching to Thread 0x2aaaab698700 (LWP 30770)]

Breakpoint 2, subthread (thd=0x6012c8 <thd+40>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=0000012C
++printf "thd->n1=%08X\n", thd->n1
thd->n1=00000233
++bt
#0  subthread (thd=0x6012c8 <thd+40>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaab698700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+c
[New Thread 0x2aaaab899700 (LWP 30772)]
[Thread 0x2aaaab698700 (LWP 30770) exited]
[Switching to Thread 0x2aaaab899700 (LWP 30772)]

Breakpoint 2, subthread (thd=0x6012f0 <thd+80>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=00000234
++printf "thd->n1=%08X\n", thd->n1
thd->n1=0000033B
++bt
#0  subthread (thd=0x6012f0 <thd+80>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaab899700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+c
[Thread 0x2aaaab899700 (LWP 30772) exited]
[New Thread 0x2aaaaba9a700 (LWP 30773)]
[Switching to Thread 0x2aaaaba9a700 (LWP 30773)]

Breakpoint 2, subthread (thd=0x601318 <thd+120>) at subthread.c:5
5        primelist *p=NULL, *q=NULL;
+xprint
++printf "thd->n0=%08X\n", thd->n0
thd->n0=0000033C
++printf "thd->n1=%08X\n", thd->n1
thd->n1=00000442
++bt
#0  subthread (thd=0x601318 <thd+120>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaaba9a700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113
+info thread
  Id   Target Id         Frame
* 5    Thread 0x2aaaaba9a700 (LWP 30773) "prime-gdb" subthread (thd=0x601318 <th
d+120>) at subthread.c:5
  1    Thread 0x2aaaaaafcfc0 (LWP 30765) "prime-gdb" main (argc=2, argv=0x7fffff
ffdf88) at prime.c:44
+thread apply all bt

Thread 5 (Thread 0x2aaaaba9a700 (LWP 30773)):
+bt
#0  subthread (thd=0x601318 <thd+120>) at subthread.c:5
#1  0x00002aaaaacd6e0e in start_thread (arg=0x2aaaaba9a700) at pthread_create.c:
311
#2  0x00002aaaaafd393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:
113

Thread 1 (Thread 0x2aaaaaafcfc0 (LWP 30765)):
+bt
#0  main (argc=2, argv=0x7fffffffdf88) at prime.c:44
+c
[Thread 0x2aaaaba9a700 (LWP 30773) exited]
[Switching to Thread 0x2aaaaaafcfc0 (LWP 30765)]

Breakpoint 3, main (argc=2, argv=0x7fffffffdf88) at prime.c:44
44            if (thd[i].head != NULL) { /* prime found */
+print i
$4 = 3
+c
2
3
5
7
11
13
... (snip)
1063
1069
1087
[Inferior 1 (process 30765) exited normally]
+quit

Please note the backtrace generated by bt:

See how a user defined command xprint is defined and a sequence of commands at breakpoint is auto-executed via command.

GDB TUI

Although normal command mode GDB is powerful, it is cumbersome to keep typing “list”, “print ...”, “info break”, and “info reg”, … just to see what is going on.

That is where GDB Text User Interface (TUI) comes in. Wile you are using GDB:

For more, read “info gdb TUI” from the shell prompt.

TIP: Use “gdb -tui” or “gdbtui” in place of “gdb” to start GDB in the TUI mode.

GDB resources

Here are some GDB resources:

Previous Post Top Next Post