Troubleshooting an application can be fairly complex, especially when dealing with highly concurrent languages like Go. It can be fairly simple to add print statements to determine subjective application state at specific intervals, however it’s much more difficult to respond dynamically to conditions developing in your code with this method.
Debuggers provide an incredibly powerful troubleshooting mechanism. Adding code for troubleshooting can subtly affect how an application runs. Debuggers can give a much more accurate view of your code in the wild.
A number of debuggers exist for Go, some inject code at compile time to support an interactive terminal which negates some of the benefit of using a debugger. The gdb debugger allows you to inspect your compiled binaries, provided they were linked with debug information, without altering the source code. This is quite a powerful feature, since you can pull a build artifact from your deployment pipeline and debug it interactively. You can read more about this in the official Golang docs, so this guide will give a quick overview into the basic usage of the gdb debugger for Go applications.
There seem to have been a number of changes in gdb since this was announced, most notably the replacement of -> operator with . for accessing object attributes. Keep in mind that there may be subtle changes like this between versions of gdb and Go. This guide was written using gdb version 7.7.1 and go version 1.5beta2.
Getting Started with gdb Debugging
To experiment with gdb I’m using a test application, the complete source code for which can be found on in gdb_sandbox on Github. Let’s start with a really simple application:
package main
import (
"fmt"
)
func main() {
for i := 0; i < 5; i++ {
fmt.Println("looping")
}
fmt.Println("Done")
}We can run this code and see some very predictable output:
$ go run main.go looping looping looping looping looping Done
Let’s debug the application. First, build the Go binary and then execute gdb with the binary path as an argument. Depending on your setup, you’ll also need to load Go runtime support via a source command. At this point we’ll be in the gdb shell, and we can set up breakpoints before executing our binary.
$ go build -gcflags "-N -l" -o gdb_sandbox main.go $ ls gdb_sandbox main.go README.md $ gdb gdb_sandbox .... (gdb) source /usr/local/src/go/src/runtime/runtime-gdb.py Loading Go Runtime support.
First off, let’s put a breakpoint (b) inside the for loop and take a look at what state our code has in each loop execution. We can then use the print (p) command to inspect a variable from the current context and the list (l) and backtrace (bt) commands to take a look at the code around the current step. The application execution can be stepped using next (n) or we can just continue to the next breakpoint (c).
(gdb) b main.go:9
Breakpoint 1 at 0x400d35: file /home/bfosberry/workspace/gdb_sandbox/main.go, line 9.
(gdb) run
Starting program: /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/gdb_sandbox Breakpoint 1, main.main () at
/home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:9
9 fmt.Println("looping")
(gdb) l
4 "fmt"
5 )
6
7 func main() {
8 for i := 0; i < 5; i++ {
9 fmt.Println("looping")
10 }`
11 fmt.Println("Done")
12 }
(gdb) p i
$1 = 0
(gdb) n
looping
Breakpoint 1, main.main () at
/home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:9
9 fmt.Println("looping")
(gdb) p i
$2 = 1
(gdb) bt
# 0 main.main () at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:9You can set breakpoints using a relative file and line number reference, a GOPATH file and line number reference, or a package and function reference. The following are also valid breakpoints:
(gdb) b github.com/bfosberry/gdb_sandbox/main.go:9 (gdb) b 'main.main'
Structs
We can make the code a little more complex to show the benefits of live debugging. We will use the function f to generate a simple pair, x and y, where y = f(x) when x is even, otherwise = x.
type pair struct {
x int
y int
}
func handleNumber(i int) *pair {
val := i
if i%2 == 0 {
val = f(i)
}
return &pair{
x: i,
y: val,
}
}
func f(int x) int {
return x*x + x
}Also we can change the looping code to call these new functions.
p := handleNumber(i)
fmt.Printf("%+v\n", p)
fmt.Println("looping")Let’s say we need to debug the value of y. We can start by setting a breakpoint where y is being set and then step through the code. Using info args we can check function parameters, and as before bt gives us the current backtrace.
(gdb) b 'main.f'
(gdb) run
Starting program: /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/gdb_sandbox
Breakpoint 1, main.f (x=0, ~anon1=833492132160)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:33
33 return x*x + x
(gdb) info args
x = 0
(gdb) continue
Breakpoint 1, main.f (x=0, ~anon1=833492132160)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:33
33 return x*x + x
(gdb) info args
x = 2
(gdb) bt
#0 main.f (x=2, ~anon1=1)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:33
#1 0x0000000000400f0e in main.handleNumber (i=2, ~anon1=0x1)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:24
#2 0x0000000000400c47 in main.main ()
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:14Since we’re in a condition where the value of y is being set based on the function f, we can set out of this function context and examine code farther up the stack. While the application is running we can set another breakpoint at a higher level and examine the state there.
(gdb) b main.go:26
Breakpoint 2 at 0x400f22: file
/home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go, line 26.
(gdb) continue
Continuing.
Breakpoint 2, main.handleNumber (i=2, ~anon1=0x1)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:28
28 y: val,
(gdb) l
23 if i%2 == 0 {
24 val = f(i)
25 }
26 return &pair{
27 x: i,
28 y: val,
29 }
30 }
31
32 func f(x int) int {
(gdb) p val
$1 = 6
(gdb) p i
$2 = 2If we continue at this point we will bypass breakpoint 1 in function f, and we will trigger the breakpoint in the handleNumber function immediately since the function f is only executed for every second value of i. We can avoid this by disabling breakpoint 2 temporarily.
(gdb) disable breakpoint 2
(gdb) continue
Continuing.
&{x:2 y:6}
looping
&{x:3 y:3}
looping
[New LWP 15200]
[Switching to LWP 15200]
Breakpoint 1, main.f (x=4, ~anon1=1)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:33
33 return x*x + x
(gdb)We can also clear and delete breakpoints using clear and delete breakpoint NUMBER respectively. By dynamically creating and toggling breakpoints we can efficiently traverse application flow.
Slices and Pointers
Few applications will be as simple as pure numerical or string values, so let’s make the code a little more complex. By adding a slice of pointers to the main function and storing generated pairs, we could potentially plot them later.
var pairs []*pair
for i := 0; i < 10; i++ {
p := handleNumber(i)
fmt.Printf("%+v\n", p)
pairs = append(pairs, p)
fmt.Println("looping")
}This time around let’s examine the slice or pairs as it gets built. First of all we’ll need to examine the slice by converting it to an array. Since handleNumber returns a *pair type, we’ll need to dereference the pointers and access the struct attributes.
(gdb) b main.go:18
Breakpoint 1 at 0x400e14: file /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go, line 18.
(gdb) run
Starting program: /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/gdb_sandbox &{x:0 y:0}
Breakpoint 1, main.main () at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:18
18 fmt.Println("looping")
(gdb) p pairs
$1 = []*main.pair = {0xc82000a3a0}
(gdb) p pairs[0]
Structure has no component named operator[].
(gdb) p pairs.array
$2 = (struct main.pair **) 0xc820030028
(gdb) p pairs.array[0]
$3 = (struct main.pair *) 0xc82000a3a0
(gdb) p *pairs.array[0]
$4 = {x = 0, y = 0}
(gdb) p (*pairs.array[0]).x
$5 = 0
(gdb) p (*pairs.array[0]).y
$6 = 0
(gdb) continue
Continuing.
looping
&{x:1 y:1}
Breakpoint 1, main.main () at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:18
18 fmt.Println("looping")
(gdb) p (pairs.array[1][5]).y
$7 = 1
(gdb) continue
Continuing.
looping
&{x:2 y:6}
Breakpoint 1, main.main () at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:18
18 fmt.Println("looping")
(gdb) p (pairs.array[2][6]).y
$8 = 6
(gdb)You’ll notice that while gdb does identify the fact that pairs is a slice, we can’t directly access attributes. In order to access the members of the slice we need to convert it to an array via pairs.array. We can check the length and capacity of the slice though:
(gdb) p $len(pairs) $12 = 3 (gdb) p $cap(pairs) $13 = 4
At this point we can run the loop a few times and monitor the increasing value of x and y across different members of the slice. Something to note here is that struct attributes can be accessed via the pointer, so p pairs.array[2].y works just as well.
Goroutines
Now that we can access structs and slices, let’s make the application even more complicated. Let’s add some goroutines into the mix by updating our main function to process each number in parallel and pass the results back through a channel:
pairs := []*pair{}
pairChan := make(chan *pair)
wg := sync.WaitGroup{}
for i := 0; i < 10; i++ {
wg.Add(1)
go func(val int) {
p := handleNumber(val)
fmt.Printf("%+v\n", p)
pairChan <- p
wg.Done()
}(i)
}
go func() {
for p := range pairChan {
pairs = append(pairs, p)
}
}()
wg.Wait()
close(pairChan)If we wait for the WaitGroup to complete and inspect the resulting pairs slice, we can expect the contents to be exactly the same, although perhaps in a different order. The real power of gdb here comes from inspecting goroutines in flight:
(gdb) b main.go:43
Breakpoint 1 at 0x400f7f: file /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go, line 43.
(gdb) run
Starting program: /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/gdb_sandbox
Breakpoint 1, main.handleNumber (i=0, ~r1=0x0)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:43
43 y: val,
(gdb) l
38 if i%2 == 0 {
39 val = f(i)
40 }
41 return &pair{
42 x: i,
43 y: val,
44 }
45 }
46
47 func f(x int) int {
(gdb) info args
i = 0
~r1 = 0x0
(gdb) p val
$1 = 0You’ll notice that we placed a breakpoint in a section of code which is executed within a goroutine. From here we can inspect local variables as well as look at other goroutines in progress:
(gdb) info goroutines
1 waiting runtime.gopark
2 waiting runtime.gopark
3 waiting runtime.gopark
4 waiting runtime.gopark
* 5 running main.main.func1
6 runnable main.main.func1
7 runnable main.main.func1
8 runnable main.main.func1
9 runnable main.main.func1
* 10 running main.main.func1
11 runnable main.main.func1
12 runnable main.main.func1
13 runnable main.main.func1
14 runnable main.main.func1
15 waiting runtime.gopark
(gdb) goroutine 11 bt
#0 main.main.func1 (val=6, pairChan=0xc82001a180, &wg=0xc82000a3a0)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:19
#1 0x0000000000454991 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1696
#2 0x0000000000000006 in ?? ()
#3 0x000000c82001a180 in ?? ()
#4 0x000000c82000a3a0 in ?? ()
#5 0x0000000000000000 in ?? ()
(gdb) goroutine 11 l
48 return x*x + x
49 }
(gdb) goroutine 11 info args
val = 6
pairChan = 0xc82001a180
&wg = 0xc82000a3a0
(gdb) goroutine 11 p val
$2 = 6The first thing we do here is list all running goroutines and identify one of our handlers. We can then view a backtrace and essentially send any debug commands to that goroutine. The backtrace and listing clearly do not match, how backtrace does seem to be accurate. info args on that goroutine shows us local variables, as well as variables available in the main function, outside the scope of the goroutine function which are prepended with an &.
Conclusion
When it comes to debugging applications, gdb can be incredibly powerful. This is still a fairly fresh integration, and not everything works perfectly. Using the latest stable gdb, with go1.5beta2, many things are still broken:
Interfaces
According to the go blog post, go interfaces should be supported, allowing you to dynamically cast then to their base types in gdb. This seems to be broken.
Interface{} types
There is no current way to convert an interface{} to its type.
Listing a different goroutine
Listing surrounding code from within another goroutine causes the line number to drift, eventually resulting in gdb thinking the current line is beyond the bounds of the file and throwing an error:
(gdb) info goroutines
1 waiting runtime.gopark
2 waiting runtime.gopark
3 waiting runtime.gopark
4 waiting runtime.gopark
* 5 running main.main.func1
6 runnable main.main.func1
7 runnable main.main.func1
8 runnable main.main.func1
9 runnable main.main.func1
* 10 running main.main.func1
11 runnable main.main.func1
12 runnable main.main.func1
13 runnable main.main.func1
14 runnable main.main.func1
15 waiting runtime.gopark
(gdb) goroutine 11 bt
#0 main.main.func1 (val=6, pairChan=0xc82001a180, &wg=0xc82000a3a0)
at /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go:19
#1 0x0000000000454991 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1696
#2 0x0000000000000006 in ?? ()
#3 0x000000c82001a180 in ?? ()
#4 0x000000c82000a3a0 in ?? ()
#5 0x0000000000000000 in ?? ()
(gdb) goroutine 11 l
48 return x*x + x
49 }
(gdb) goroutine 11 l
Python Exception <class 'gdb.error'> Line number 50 out of range; /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go has 49 lines.:
Error occurred in Python command: Line number 50 out of range; /home/bfosberry/.go/src/github.com/bfosberry/gdb_sandbox/main.go has 49 lines.Goroutine debugging is unstable
Handling goroutines general tends to be unstable; I managed to cause a number of segfaults executing simple commands. At this stage you should be prepared to deal with some issues.
Configuring gdb with Go support can be troublesome
Running gdb with Go support can be troublesome, getting the right combination of paths and build flags, and gdb auto-loading functionality doesn’t seem to work correctly. First of all, loading Go runtime support via a gdb init file initializes incorrectly. This may need to be loaded manually via a source command once the debugging shell has been initialized as described in this guide.
When should I use a debugger?
So when is it useful to use gdb? Using print statement and debugging code is a much more targeted approach.
- When changing the code is not an option.
- When debugging a problem where the source is not known, and dynamic breakpoints may be beneficial.
- When working with many goroutines where the ability to pause and inspect program state would be beneficial.
| Reference: | Using the gdb Debugger with Go from our WCG partner Florian Motlik at the Codeship Blog blog. |
