Debugging with GDB¶
Avocado has two different types of GDB support that complement each other:
- Transparent execution of executables inside the GNU Debugger. This
takes standard and possibly unmodified tests that uses the
avocado.utils.processAPIs for running processes. By using a command line option, the executable is run on GDB. This allows the user to interact with GDB, but to the test itself, things are pretty much transparent.
avocado.utils.gdbAPIs that allows a test to interact with GDB, including setting a executable to be run, setting breakpoints or any other types of commands. This requires a test written with that approach and API in mind.
Even though this section describes the use of the Avocado GDB
features, which allow live debugging of binaries inside Avocado
tests, it’s also possible to debug some application offline by
using tools such as rr. Avocado ships
with an example wrapper script (to be used with
Transparent Execution of Executables¶
This feature adds a few command line options to the Avocado
$ avocado run --help
GNU Debugger support:
Run a given executable inside the GNU debugger,
pausing at a given breakpoint (defaults to "main")
After loading an executable in GDB, but before
actually running it, execute the GDB commands in the
given file. EXECUTABLE is optional, if omitted
COMMANDS will apply to all executables
Automatically generate a core dump when the inferior
process received a fatal signal such as SIGSEGV or
To get started you want to use
--gdb-run-bin, as shown in the example bellow.
The simplest way is to just run
avocado run --gdb-run-bin=doublefree examples/tests/doublefree.py, which
wraps each executed executable with name
doublefree inside GDB server and
stops at the executable entry point.
Optionally you can specify single breakpoint using
doublefree:1) or just
doublefree: to stop only when an interruption happens (eg: SIGABRT).
It’s worth mentioning that when breakpoint is not reached, the test finishes
without any interruption. This is helpful when you identify regions where you
should never get in your code, or places which interests you and you can run
your code in production and GDB variants. If after a long time you get to this
place, the test notifies you and you can investigate the problem. This is
examples/tests/doublefree_nasty.py test. To unveil the
power of Avocado, run this test using:
avocado run --gdb-run-bin=doublefree: examples/tests/doublefree_nasty.py --gdb-prerun-commands examples/tests/doublefree_nasty.py.data/gdb_pre --mux-yaml examples/tests/doublefree_nasty.py.data/iterations.yaml
which executes 100 iterations of this test while setting all breakpoints from
examples/tests/doublefree_nasty.py.data/gdb_pre file (you can specify
whatever GDB supports, not only breakpoints).
As you can see this test usually passes, but once in a while it gets into the problematic area. Imagine this is very hard to spot (dependent on HW registers, …) and this is one way to combine regular testing and the possibility of debugging hard-to-get parts of your code.
Currently, when using the Avocado GDB plugin, that is, when using the –gdb-run-bin option, there are some caveats you should be aware of:
- It is not currently compatible with Avocado’s –output-check-record feature
- There’s no way to perform proper input to the process, that is, manipulate its STDIN
- The process STDERR content is mixed with the content generated by gdbserver on its own STDERR (because they are in fact, the same thing)
But, you can still depend on the process STDOUT, as exemplified by this fictional test:
from avocado import Test
from avocado.utils import process
result = process.run("/path/to/hello", ignore_status=True)
If run under GDB or not, result.stdout behavior and content is expected to be the same.
Reasons for the caveats¶
There are a two basic reasons for the mentioned caveats:
- The architecture of Avocado’s GDB feature
- GDB’s own behavior and limitations
When using the Avocado GDB plugin, that is, –gdb-run-bin, Avocado runs a gdbserver instance transparently and controls it by means of a gdb process. When a given event happens, say a breakpoint is reached, it disconnects its own gdb from the server, and allows the user to use a standard gdb to connect to the gdbserver. This provides a natural and seamless user experience.
But, gdbserver has some limitations at this point, including:
- Not being able to set a controlling tty
- Not separating its own STDERR content from the application being run
These limitations are being addressed both on Avocado and GDB, and will be resolved in future Avocado versions.
If the application you’re running as part of your test can read input from alternative sources (including devices, files or the network) and generate output likewise, then you should not be further limited.
GDB support and avocado-virt¶
Another current limitation is the use of avocado-virt and avocado GDB support.
The supported API for transparent debugging is currently limited to
avocado.utils.process.run(), and does not cover advanced uses of the
avocado.utils.process.SubProcess class. The avocado-virt
extension, though, uses
avocado.utils.process.SubProcess class to
execute qemu in the background.
This limitation will be addressed in future versions of avocado and avocado-virt.
Avocado’s GDB module, provides three main classes that lets a test writer interact with a gdb process, a gdbserver process and also use the GDB remote protocol for interaction with a remote target.
Please refer to
avocado.utils.gdb for more information.
Take a look at
path = os.path.join(self.srcdir, 'print_variable')
app = gdb.GDB()
app.cmd("set variable a = 0xff")
out = "\n".join(app.read_until_break())
self.assertIn("MY VARIABLE 'A' IS: ff", out)
You can see that instead of running the executable using
process.run we invoke
avocado.utils.gdb.GDB. This allows
us to automate the interaction with the GDB in means of setting
breakpoints, executing commands and querying for output.
When you check the output (
--show-job-log) you can see that despite
declaring the variable as 0, ff is injected and printed instead.