Troubleshooting an application can be fairly complex, especially when dealing with highly concurrent languages like Go. It can be fairly simple to add print statements to determine subjective application state at specific intervals, however it\u2019s much more difficult to respond dynamically to conditions developing in your code with this method.<\/p>\n
Debuggers provide an incredibly powerful troubleshooting mechanism. Adding code for troubleshooting can subtly affect how an application runs. Debuggers can give a much more accurate view of your code in the wild.<\/p>\n
A number of debuggers exist for Go, some inject code at compile time<\/a> to support an interactive terminal which negates some of the benefit of using a debugger. The gdb debugger allows you to inspect your compiled binaries, provided they were linked with debug information, without altering the source code. This is quite a powerful feature, since you can pull a build artifact from your deployment pipeline and debug it interactively. You can read more about this in the official Golang docs<\/a>, so this guide will give a quick overview into the basic usage of the gdb debugger for Go applications.<\/p>\n There seem to have been a number of changes in gdb<\/strong> since this was announced<\/a>, most notably the replacement of -><\/strong> operator with .<\/strong> for accessing object attributes. Keep in mind that there may be subtle changes like this between versions of gdb<\/strong> and Go. This guide was written using gdb<\/strong> version 7.7.1 and go version 1.5beta2.<\/p>\n To experiment with gdb<\/strong> I\u2019m using a test application, the complete source code for which can be found on in gdb_sandbox on Github<\/a>. Let\u2019s start with a really simple application:<\/p>\n We can run this code and see some very predictable output:<\/p>\n Let\u2019s debug the application. First, build the Go binary and then execute gdb<\/strong> with the binary path as an argument. Depending on your setup, you\u2019ll also need to load Go runtime support via a source<\/strong> command. At this point we\u2019ll be in the gdb<\/strong> shell, and we can set up breakpoints before executing our binary.<\/p>\n First off, let\u2019s put a breakpoint (b) inside the for loop and take a look at what state our code has in each loop execution. We can then use the print (p) command to inspect a variable from the current context and the list (l) and backtrace (bt) commands to take a look at the code around the current step. The application execution can be stepped using next (n) or we can just continue to the next breakpoint (c).<\/p>\n You can set breakpoints using a relative file and line number reference, a GOPATH<\/strong> file and line number reference, or a package and function reference. The following are also valid breakpoints:<\/p>\n We can make the code a little more complex to show the benefits of live debugging. We will use the function f<\/strong> to generate a simple pair, x and y, where y = f(x) when x is even, otherwise = x.<\/p>\n Also we can change the looping code to call these new functions.<\/p>\n Let\u2019s say we need to debug the value of y<\/strong>. We can start by setting a breakpoint where y<\/strong> is being set and then step through the code. Using info args<\/strong> we can check function parameters, and as before bt<\/strong> gives us the current backtrace.<\/p>\n Since we\u2019re in a condition where the value of y<\/strong> is being set based on the function f<\/strong>, we can set out of this function context and examine code farther up the stack. While the application is running we can set another breakpoint at a higher level and examine the state there.<\/p>\n If we continue at this point we will bypass breakpoint 1 in function f<\/strong>, and we will trigger the breakpoint in the handleNumber<\/strong> function immediately since the function f<\/strong> is only executed for every second value of i<\/strong>. We can avoid this by disabling breakpoint 2 temporarily.<\/p>\n We can also clear and delete breakpoints using clear<\/strong> and delete breakpoint NUMBER<\/strong> respectively. By dynamically creating and toggling breakpoints we can efficiently traverse application flow.<\/p>\n Few applications will be as simple as pure numerical or string values, so let\u2019s make the code a little more complex. By adding a slice of pointers to the main<\/strong> function and storing generated pairs, we could potentially plot them later.<\/p>\n This time around let\u2019s examine the slice or pairs as it gets built. First of all we\u2019ll need to examine the slice by converting it to an array. Since handleNumber<\/strong> returns a *pair<\/strong> type, we\u2019ll need to dereference the pointers and access the struct attributes.<\/p>\n You\u2019ll notice that while gdb<\/strong> does identify the fact that pairs<\/strong> is a slice, we can\u2019t directly access attributes. In order to access the members of the slice we need to convert it to an array via pairs.array<\/strong>. We can check the length and capacity of the slice though:<\/p>\n At this point we can run the loop a few times and monitor the increasing value of x<\/strong> and y<\/strong> across different members of the slice. Something to note here is that struct attributes can be accessed via the pointer, so Now that we can access structs and slices, let\u2019s make the application even more complicated. Let\u2019s add some goroutines into the mix by updating our main function to process each number in parallel and pass the results back through a channel:<\/p>\n If we wait for the WaitGroup to complete and inspect the resulting pairs<\/strong> slice, we can expect the contents to be exactly the same, although perhaps in a different order. The real power of gdb<\/strong> here comes from inspecting goroutines in flight:<\/p>\n You\u2019ll notice that we placed a breakpoint in a section of code which is executed within a goroutine. From here we can inspect local variables as well as look at other goroutines in progress:<\/p>\n The first thing we do here is list all running goroutines and identify one of our handlers. We can then view a backtrace and essentially send any debug commands to that goroutine. The backtrace and listing clearly do not match, how backtrace does seem to be accurate. info args<\/strong> on that goroutine shows us local variables, as well as variables available in the main function, outside the scope of the goroutine function which are prepended with an &<\/strong>.<\/p>\n When it comes to debugging applications, gdb<\/strong> can be incredibly powerful. This is still a fairly fresh integration, and not everything works perfectly. Using the latest stable gdb<\/strong>, with go1.5beta2, many things are still broken:<\/p>\n According to the go blog post<\/a>, go interfaces should be supported, allowing you to dynamically cast then to their base types in gdb<\/strong>. This seems to be broken.<\/p>\n There is no current way to convert an interface{} to its type.<\/p>\n Listing surrounding code from within another goroutine causes the line number to drift, eventually resulting in gdb<\/strong> thinking the current line is beyond the bounds of the file and throwing an error:<\/p>\n Handling goroutines general tends to be unstable; I managed to cause a number of segfaults executing simple commands. At this stage you should be prepared to deal with some issues.<\/p>\n Running gdb<\/strong> with Go support can be troublesome, getting the right combination of paths and build flags, and gdb<\/strong> auto-loading functionality doesn\u2019t seem to work correctly. First of all, loading Go runtime support via a gdb<\/strong> init file initializes incorrectly. This may need to be loaded manually via a source command once the debugging shell has been initialized as described in this guide.<\/p>\n So when is it useful to use gdb<\/strong>? Using print statement and debugging code is a much more targeted approach.<\/p>\nGetting Started with gdb Debugging<\/h2>\n
package main\r\n\r\nimport ( \r\n \"fmt\" \r\n)\r\n\r\nfunc main() { \r\n for i := 0; i < 5; i++ {\r\n fmt.Println(\"looping\") \r\n } \r\n fmt.Println(\"Done\") \r\n}<\/pre>\n$ go run main.go\r\nlooping\r\nlooping\r\nlooping\r\nlooping\r\nlooping\r\nDone<\/pre>\n
$ go build -gcflags \"-N -l\" -o gdb_sandbox main.go \r\n$ ls\r\ngdb_sandbox main.go README.md\r\n$ gdb gdb_sandbox\r\n....\r\n(gdb) source \/usr\/local\/src\/go\/src\/runtime\/runtime-gdb.py\r\nLoading Go Runtime support.<\/pre>\n
(gdb) b main.go:9 \r\nBreakpoint 1 at 0x400d35: file \/home\/bfosberry\/workspace\/gdb_sandbox\/main.go, line 9. \r\n(gdb) run \r\nStarting program: \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/gdb_sandbox Breakpoint 1, main.main () at \r\n\/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:9 \r\n9 fmt.Println(\"looping\") \r\n(gdb) l \r\n4 \"fmt\" \r\n5 ) \r\n6 \r\n7 func main() {\r\n8 for i := 0; i < 5; i++ { \r\n9 fmt.Println(\"looping\") \r\n10 }` \r\n11 fmt.Println(\"Done\") \r\n12 } \r\n(gdb) p i \r\n$1 = 0 \r\n(gdb) n \r\nlooping \r\nBreakpoint 1, main.main () at \r\n\/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:9 \r\n9 fmt.Println(\"looping\") \r\n(gdb) p i \r\n$2 = 1 \r\n(gdb) bt\r\n# 0 main.main () at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:9<\/pre>\n(gdb) b github.com\/bfosberry\/gdb_sandbox\/main.go:9\r\n(gdb) b 'main.main'<\/pre>\n
Structs<\/h2>\n
type pair struct { \r\n x int \r\n y int \r\n}\r\n\r\nfunc handleNumber(i int) *pair { \r\n val := i \r\n if i%2 == 0 { \r\n val = f(i) \r\n } \r\n return &pair{ \r\n x: i, \r\n y: val, \r\n } \r\n}\r\n\r\nfunc f(int x) int { \r\n return x*x + x \r\n}<\/pre>\np := handleNumber(i)\r\n fmt.Printf(\"%+v\\n\", p)\r\n fmt.Println(\"looping\")<\/pre>\n
(gdb) b 'main.f' \r\n(gdb) run \r\nStarting program: \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/gdb_sandbox\r\n\r\nBreakpoint 1, main.f (x=0, ~anon1=833492132160) \r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:33 \r\n33 return x*x + x \r\n(gdb) info args \r\nx = 0 \r\n(gdb) continue \r\nBreakpoint 1, main.f (x=0, ~anon1=833492132160) \r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:33 \r\n33 return x*x + x \r\n(gdb) info args \r\nx = 2 \r\n(gdb) bt\r\n#0 main.f (x=2, ~anon1=1) \r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:33\r\n#1 0x0000000000400f0e in main.handleNumber (i=2, ~anon1=0x1)\r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:24\r\n#2 0x0000000000400c47 in main.main ()\r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:14<\/pre>\n
(gdb) b main.go:26 \r\nBreakpoint 2 at 0x400f22: file \r\n\/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go, line 26. \r\n(gdb) continue \r\nContinuing.\r\n\r\nBreakpoint 2, main.handleNumber (i=2, ~anon1=0x1) \r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:28 \r\n28 y: val, \r\n(gdb) l \r\n23 if i%2 == 0 { \r\n24 val = f(i) \r\n25 } \r\n26 return &pair{ \r\n27 x: i, \r\n28 y: val, \r\n29 } \r\n30 } \r\n31 \r\n32 func f(x int) int { \r\n(gdb) p val \r\n$1 = 6 \r\n(gdb) p i \r\n$2 = 2<\/pre>\n(gdb) disable breakpoint 2 \r\n(gdb) continue \r\nContinuing. \r\n&{x:2 y:6} \r\nlooping \r\n&{x:3 y:3} \r\nlooping \r\n[New LWP 15200] \r\n[Switching to LWP 15200]\r\n\r\nBreakpoint 1, main.f (x=4, ~anon1=1) \r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:33 \r\n33 return x*x + x \r\n(gdb)<\/pre>\nSlices and Pointers<\/h2>\n
var pairs []*pair\r\n for i := 0; i < 10; i++ {\r\n p := handleNumber(i)\r\n fmt.Printf(\"%+v\\n\", p)\r\n pairs = append(pairs, p)\r\n fmt.Println(\"looping\")\r\n }<\/pre>\n(gdb) b main.go:18 \r\nBreakpoint 1 at 0x400e14: file \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go, line 18. \r\n(gdb) run \r\nStarting program: \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/gdb_sandbox &{x:0 y:0}\r\n\r\nBreakpoint 1, main.main () at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:18 \r\n18 fmt.Println(\"looping\") \r\n(gdb) p pairs \r\n$1 = []*main.pair = {0xc82000a3a0} \r\n(gdb) p pairs[0] \r\nStructure has no component named operator[]. \r\n(gdb) p pairs.array \r\n$2 = (struct main.pair **) 0xc820030028 \r\n(gdb) p pairs.array[0] \r\n$3 = (struct main.pair *) 0xc82000a3a0 \r\n(gdb) p *pairs.array[0] \r\n$4 = {x = 0, y = 0} \r\n(gdb) p (*pairs.array[0]).x \r\n$5 = 0 \r\n(gdb) p (*pairs.array[0]).y \r\n$6 = 0 \r\n(gdb) continue \r\nContinuing. \r\nlooping \r\n&{x:1 y:1}\r\n\r\nBreakpoint 1, main.main () at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:18 \r\n18 fmt.Println(\"looping\") \r\n(gdb) p (pairs.array[1][5]).y \r\n$7 = 1 \r\n(gdb) continue \r\nContinuing. \r\nlooping \r\n&{x:2 y:6}\r\n\r\nBreakpoint 1, main.main () at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:18 \r\n18 fmt.Println(\"looping\") \r\n(gdb) p (pairs.array[2][6]).y \r\n$8 = 6 \r\n(gdb)<\/pre>\n(gdb) p $len(pairs)\r\n$12 = 3\r\n(gdb) p $cap(pairs)\r\n$13 = 4<\/pre>\n
p pairs.array[2].y<\/code> works just as well.<\/p>\nGoroutines<\/h2>\n
pairs := []*pair{}\r\n pairChan := make(chan *pair)\r\n wg := sync.WaitGroup{}\r\n for i := 0; i < 10; i++ {\r\n wg.Add(1)\r\n go func(val int) {\r\n p := handleNumber(val)\r\n fmt.Printf(\"%+v\\n\", p)\r\n pairChan <- p\r\n wg.Done()\r\n }(i)\r\n }\r\n go func() {\r\n for p := range pairChan {\r\n pairs = append(pairs, p)\r\n }\r\n }()\r\n wg.Wait()\r\n close(pairChan)<\/pre>\n(gdb) b main.go:43 \r\nBreakpoint 1 at 0x400f7f: file \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go, line 43. \r\n(gdb) run \r\nStarting program: \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/gdb_sandbox\r\n\r\nBreakpoint 1, main.handleNumber (i=0, ~r1=0x0) \r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:43 \r\n43 y: val, \r\n(gdb) l \r\n38 if i%2 == 0 { \r\n39 val = f(i) \r\n40 } \r\n41 return &pair{ \r\n42 x: i, \r\n43 y: val, \r\n44 } \r\n45 } \r\n46 \r\n47 func f(x int) int { \r\n(gdb) info args \r\ni = 0 \r\n~r1 = 0x0 \r\n(gdb) p val \r\n$1 = 0<\/pre>\n(gdb) info goroutines \r\n 1 waiting runtime.gopark \r\n 2 waiting runtime.gopark \r\n 3 waiting runtime.gopark \r\n 4 waiting runtime.gopark \r\n* 5 running main.main.func1 \r\n 6 runnable main.main.func1 \r\n 7 runnable main.main.func1 \r\n 8 runnable main.main.func1 \r\n 9 runnable main.main.func1 \r\n* 10 running main.main.func1 \r\n 11 runnable main.main.func1 \r\n 12 runnable main.main.func1 \r\n 13 runnable main.main.func1 \r\n 14 runnable main.main.func1 \r\n 15 waiting runtime.gopark \r\n(gdb) goroutine 11 bt\r\n#0 main.main.func1 (val=6, pairChan=0xc82001a180, &wg=0xc82000a3a0)\r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:19\r\n#1 0x0000000000454991 in runtime.goexit () at \/usr\/local\/go\/src\/runtime\/asm_amd64.s:1696\r\n#2 0x0000000000000006 in ?? ()\r\n#3 0x000000c82001a180 in ?? ()\r\n#4 0x000000c82000a3a0 in ?? ()\r\n#5 0x0000000000000000 in ?? ()\r\n(gdb) goroutine 11 l \r\n48 return x*x + x \r\n49 } \r\n(gdb) goroutine 11 info args \r\nval = 6 \r\npairChan = 0xc82001a180 \r\n&wg = 0xc82000a3a0 \r\n(gdb) goroutine 11 p val \r\n$2 = 6<\/pre>\n
Conclusion<\/h2>\n
Interfaces<\/h3>\n
Interface{} types<\/h3>\n
Listing a different goroutine<\/h3>\n
(gdb) info goroutines \r\n 1 waiting runtime.gopark \r\n 2 waiting runtime.gopark \r\n 3 waiting runtime.gopark \r\n 4 waiting runtime.gopark \r\n* 5 running main.main.func1 \r\n 6 runnable main.main.func1 \r\n 7 runnable main.main.func1 \r\n 8 runnable main.main.func1 \r\n 9 runnable main.main.func1 \r\n* 10 running main.main.func1 \r\n 11 runnable main.main.func1 \r\n 12 runnable main.main.func1 \r\n 13 runnable main.main.func1 \r\n 14 runnable main.main.func1 \r\n 15 waiting runtime.gopark \r\n(gdb) goroutine 11 bt\r\n#0 main.main.func1 (val=6, pairChan=0xc82001a180, &wg=0xc82000a3a0)\r\n at \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go:19\r\n#1 0x0000000000454991 in runtime.goexit () at \/usr\/local\/go\/src\/runtime\/asm_amd64.s:1696\r\n#2 0x0000000000000006 in ?? ()\r\n#3 0x000000c82001a180 in ?? ()\r\n#4 0x000000c82000a3a0 in ?? ()\r\n#5 0x0000000000000000 in ?? ()\r\n(gdb) goroutine 11 l \r\n48 return x*x + x \r\n49 } \r\n(gdb) goroutine 11 l \r\nPython Exception <class 'gdb.error'> Line number 50 out of range; \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go has 49 lines.: \r\nError occurred in Python command: Line number 50 out of range; \/home\/bfosberry\/.go\/src\/github.com\/bfosberry\/gdb_sandbox\/main.go has 49 lines.<\/pre>\n
Goroutine debugging is unstable<\/h3>\n
Configuring gdb<\/strong> with Go support can be troublesome<\/h3>\n
When should I use a debugger?<\/h2>\n
\n