Debugging with GDB

Avocado has two different types of GDB support that complement each other:

  • Transparent execution of executables inside the GNU Debugger. This takes standard and possibly unmodified tests that uses the avocado.utils.process APIs for running processes. By using a command line option, the executable is run on GDB. This allows the user to interact with GDB, but to the test itself, things are pretty much transparent.
  • The avocado.utils.gdb APIs that allows a test to interact with GDB, including setting a executable to be run, setting breakpoints or any other types of commands. This requires a test written with that approach and API in mind.

Tip

Even though this section describes the use of the Avocado GDB features, which allow live debugging of binaries inside Avocado tests, it’s also possible to debug some application offline by using tools such as rr. Avocado ships with an example wrapper script (to be used with --wrapper) for that purpose.

Transparent Execution of Executables

This feature adds a few command line options to the Avocado run command:

$ avocado run --help
...
GNU Debugger support:

  --gdb-run-bin EXECUTABLE[:BREAKPOINT]
                        Run a given executable inside the GNU debugger,
                        pausing at a given breakpoint (defaults to "main")
  --gdb-prerun-commands EXECUTABLE:COMMANDS
                        After loading an executable in GDB, but before
                        actually running it, execute the GDB commands in the
                        given file. EXECUTABLE is optional, if omitted
                        COMMANDS will apply to all executables
  --gdb-coredump {on,off}
                        Automatically generate a core dump when the inferior
                        process received a fatal signal such as SIGSEGV or
                        SIGABRT
...

To get started you want to use --gdb-run-bin, as shown in the example bellow.

Example

The simplest way is to just run avocado run --gdb-run-bin=doublefree examples/tests/doublefree.py, which wraps each executed executable with name doublefree inside GDB server and stops at the executable entry point.

Optionally you can specify single breakpoint using --gdb-run-bin=doublefree:$breakpoint (eg: doublefree:1) or just doublefree: to stop only when an interruption happens (eg: SIGABRT).

It’s worth mentioning that when breakpoint is not reached, the test finishes without any interruption. This is helpful when you identify regions where you should never get in your code, or places which interests you and you can run your code in production and GDB variants. If after a long time you get to this place, the test notifies you and you can investigate the problem. This is demonstrated in examples/tests/doublefree_nasty.py test. To unveil the power of Avocado, run this test using:

avocado run --gdb-run-bin=doublefree: examples/tests/doublefree_nasty.py --gdb-prerun-commands examples/tests/doublefree_nasty.py.data/gdb_pre --mux-yaml examples/tests/doublefree_nasty.py.data/iterations.yaml

which executes 100 iterations of this test while setting all breakpoints from the examples/tests/doublefree_nasty.py.data/gdb_pre file (you can specify whatever GDB supports, not only breakpoints).

As you can see this test usually passes, but once in a while it gets into the problematic area. Imagine this is very hard to spot (dependent on HW registers, …) and this is one way to combine regular testing and the possibility of debugging hard-to-get parts of your code.

Caveats

Currently, when using the Avocado GDB plugin, that is, when using the –gdb-run-bin option, there are some caveats you should be aware of:

  • It is not currently compatible with Avocado’s –output-check-record feature
  • There’s no way to perform proper input to the process, that is, manipulate its STDIN
  • The process STDERR content is mixed with the content generated by gdbserver on its own STDERR (because they are in fact, the same thing)

But, you can still depend on the process STDOUT, as exemplified by this fictional test:

from avocado import Test
from avocado.utils import process

class HelloOutputTest(Test):

    def test(self):
        result = process.run("/path/to/hello", ignore_status=True)
        self.assertIn("hello\n", result.stdout)

If run under GDB or not, result.stdout behavior and content is expected to be the same.

Reasons for the caveats

There are a two basic reasons for the mentioned caveats:

  • The architecture of Avocado’s GDB feature
  • GDB’s own behavior and limitations

When using the Avocado GDB plugin, that is, –gdb-run-bin, Avocado runs a gdbserver instance transparently and controls it by means of a gdb process. When a given event happens, say a breakpoint is reached, it disconnects its own gdb from the server, and allows the user to use a standard gdb to connect to the gdbserver. This provides a natural and seamless user experience.

But, gdbserver has some limitations at this point, including:

  • Not being able to set a controlling tty
  • Not separating its own STDERR content from the application being run

These limitations are being addressed both on Avocado and GDB, and will be resolved in future Avocado versions.

Workaround

If the application you’re running as part of your test can read input from alternative sources (including devices, files or the network) and generate output likewise, then you should not be further limited.

GDB support and avocado-virt

Another current limitation is the use of avocado-virt and avocado GDB support.

The supported API for transparent debugging is currently limited to avocado.utils.process.run(), and does not cover advanced uses of the avocado.utils.process.SubProcess class. The avocado-virt extension, though, uses avocado.utils.process.SubProcess class to execute qemu in the background.

This limitation will be addressed in future versions of avocado and avocado-virt.

avocado.utils.gdb APIs

Avocado’s GDB module, provides three main classes that lets a test writer interact with a gdb process, a gdbserver process and also use the GDB remote protocol for interaction with a remote target.

Please refer to avocado.utils.gdb for more information.

Example

Take a look at examples/tests/modify_variable.py test:

def test(self):
    """
    Execute 'print_variable'.
    """
    path = os.path.join(self.workdir, 'print_variable')
    app = gdb.GDB()
    app.set_file(path)
    app.set_break(6)
    app.run()
    self.log.info("\n".join(app.read_until_break()))
    app.cmd("set variable a = 0xff")
    app.cmd("c")
    out = "\n".join(app.read_until_break())
    self.log.info(out)
    app.exit()
    self.assertIn("MY VARIABLE 'A' IS: ff", out)

You can see that instead of running the executable using process.run we invoke avocado.utils.gdb.GDB. This allows us to automate the interaction with the GDB in means of setting breakpoints, executing commands and querying for output.

When you check the output (--show-job-log) you can see that despite declaring the variable as 0, ff is injected and printed instead.