Debugging assembly

A shorter class today: we pretended to be a processor executing some assembly by hand, and saw a quick demo of how to use gdb to debug with computer assistance.

The general procedure I follow for debugging with gdb is:

• Create the executable file that we wish to debug. Often, on homework assignments, you'll find yourself confused by some particular test case (for example, add.lisp) and want to debug the assembly produced in that specific case. To do this, in the root of your assignment directory, you can run:

  dune build
  dune exec _build/default/bin/compile.exe -- examples/add.lisp out
  

  This will create an executable out/add.lisp.exe that we can use in gdb. (Note: you may have a slightly different directory structure.)

• Run gdb on this executable:

  gdb out/add.lisp.exe
  

• In gdb, set the assembly flavor and layouts.

  (gdb) set disassembly-flavor intel
  (gdb) layout asm 
  (gdb) layout regs
  

• Add a breakpoint at the label that denotes the start of your code. (In homework assignments this is typically lisp_entry; in the class compiler it is entry.) Then let gdb run your program until it reaches this breakpoint.

  (gdb) b lisp_entry
  (gdb) run
  

• At this point you can see the state of all register values and flags, and which line of assembly code will be executed next. (Ignore things in the assembly like QWORD PTR; read eax as rax for now; remember that numbers are written in hexadecimal. This all comes from the disassembler.)

• To step to the next line of your assembly:

  (gdb) si
  

  Hitting enter with no message will run the previous command again, useful if you want to run many lines in a row. You can also enter focus cmd to be able to scroll through your previous commands with the arrow keys.

• To view locations in memory:

  (gdb) x/d $rsp-8 
  (gdb) x/d $rdi
  

  $rsp refers to "the memory address stored in rsp." /d formats the output as a decimal number; you can also use /h for hex.