103.0 LTS

The Avocado team is proud to present another LTS (Long Term Stability) release: Avocado 103.0, AKA “Sound of Freedom”, is now available!

LTS Release

For more information on what a LTS release means, please read RFC: Long Term Stability.

Important Announcement

Since the previous LTS version (92.0), Avocado has removed the legacy runner implementation (internally simply called runner). While the runner implementation is not the default on version 92.x (nrunner is the default) it was still available as an option. Now nrunner is the only available runner implementation shipped by default.

Users migrating from Avocado 92.x and using the legacy runner will be impacted by this change and should act accordingly.

If your Avocado configuration file has a [run] section with test_runner set to runner, please keep in mind that:

  • The runner implementation no longer exists

  • The test_runner was renamed to suite_runner (along with the rename of the command line option --test-runner to --suite-runner). This is only relevant if you have a custom runner and want to use it.

While most of the functionality of the legacy runner is available under the new runner implementation (nrunner) there are some differences and even missing features. Known nrunner issues are being tracked on our GitHub project page, with the nrunner tag, and new issue reports are appreciated.

Upgrading from 92.x to 103.0

Upgrading Installations

Avocado is available on a number of different repositories and installation methods. You can find the complete details here.

After looking at your installation options, please consider the following when planning an in-place upgrade or a deployment version bump:

  • When using Python’s own package management, that is, pip, simply choose a version lower to the next Avocado release to benefit from minor releases (bugfixes) in this LTS series. In short, avocado-framework<104.0 will get you the latest release of this LTS series.

  • When using RPM packages on Linux distributions, check if there’s a particular repository that provides LTS packages only. On distributions that provide “modules”, you may find a 103lts stream. If no modules are available, use your package manager tools (such as DNF’s versionlock plugin) to pin the installation to the 92.x versions.

Porting Tests

Depending on the test type (say, avocado-instrumented or exec-test), test writers will have to observe different aspects of the changes since 92.0 LTS. This section deals primarily with avocado-instrumented tests.

To be considered an avocado-instrumented test, the test is based on the avocado.test.Test class. The API available in that class is called the “Test API”. It’s also common, but not mandatory, for such tests to leverage the modules under the avocado.utils namespace, collectively known as the Utility APIs.

With that in mind, the two sections below describe issues you should consider when moving your tests from Avocado 92.X LTS to 103.0 LTS.

Test API compatibility

Fortunately, there have been no changes in the public Test API since 92.0 LTS that should impact tests.

If you’re interested in knowing about all the internal implementation changes, though, feel free to inspect them with a command such as:

$ git diff 92.0..103.0 -- avocado/core/test.py
$ git log 92.0..103.0 -- avocado/core/test.py

Utility API compatibility

The changes in the utility APIs (those that live under the avocado.utils namespace) are too many to present porting suggestions. Please refer to the Utility APIs section for a comprehensive list of changes, including new features your test may be able to leverage.

Changes from previous LTS

Note

This is not a collection of all changes encompassing all releases from 92.0 to 103.0. This list contains changes that are relevant to users of 92.0, when evaluating an upgrade to 103.0.

When compared to the last LTS (version 92.0), the main changes introduced by this versions are:

Users / Test Writers

  • Python 3.10, 3.11 and 3.12 are formally supported (along with 3.7, 3.8 and 3.9)

  • Command line options prefixed with --nrunner- had this prefix removed. A command line option such as --nrunner-spawner is now simply --spawner. The matching configuration options such as nrunner.something became run.something. This is due to the fact that nrunner is now the only runner implementation offered by default, so the differentiation and extra typing seems unnecessary. If other runners are added in the future (or by custom, out of tree, plugins) they can choose to respect the existing options if they apply.

  • A contrib script that provides the features of the legacy --external-runner feature has been added. It’s built on the Job API and nrunner architecture.

  • There were major changes to the Avocado logging behavior, most of them to address feedback from users since the previous logging changes:

  1. The root logger handler was restored. This enables all loggers out of avocado.* namespace by default. If a test, either directly or indirectly through 3rd party libraries, logs into any namespace (say logging.getLogger('my-library')) it will show up in the Avocado’s test logs.

  2. The job.log file continues to contain logs for the avocado.job namespace, but a new file called full.log contains all generated logs for a job, including logs from all tests.

  • The runner.output.utf8 and core.input_encoding were settings were removed, and now default to the system’s setting (by using sys.getdefaultencoding()).

  • Test results files (in a job’s result directory) can now be accessed by their result status. For instance, assuming the latest job results are stored in ~/avocado/job-results/latest (the default location), users can find the results for tests that ended in FAIL at ~/avocado/job-results/latest/test-results/by-status/FAIL.

  • Avocado will now print at the end of the job a more descriptive list of tests that finished with "not OK" statuses.

  • The avocado jobs get-output-files command was removed. Its original intended purpose has not been relevant for some time, and the preservation of output files are already handled directly by all spawner implementations.

  • The Avocado Resolver now allows tests that are implemented in files (by far the most common scenario) and that may also access test data files (see Accessing test data files) to pass that information along to spawners. The effect of that is that when running such tests on “remote” spawner environments (such as “podman”) the test file and the data files can be made available in the remote environment. This is currently enabled in avocado-instrumented, python-unittest, exec-test and tap tests.

  • User of macOS will have a better experience when using Avocado. The full set of Avocado’s selftests are now run under macOS on CI. Please be advised that macOS is not currently supported at the same level of Linux-based operating systems due to the lack of contributors/maintainers with access to the needed hardware. If you are a user/developer and are willing to contribute to this, please let the Avocado team know.

  • Sysinfo collection is now fully supported in nrunner, including per-test collection of system information.

  • New plugin interfaces, PreTest and PostTest, allow actions to be executed right before or after the execution of a test, in the same spawner environment as the test itself.

  • Plugins can now have a builtin priority in relation to other plugins of the same type that will affect its execution order. This is in addition to the configurable plugins.$type.order settings.

  • The dict_variants plugin now allows the configuration of the keys that will determine the variant ID.

  • Environment variables such as AVOCADO_VERSION , AVOCADO_TEST_WORKDIR, AVOCADO_TEST_BASEDIR, AVOCADO_TEST_LOGDIR and AVOCADO_TEST_LOGFILE are now made available to exec-test.

  • Result plugins such as json and xunit now contain more accurate values reflecting Avocado’s concepts of a test’s ID and name.

  • The xunit (AKA junit) result file now contains the class, test’s file names and better error information for test cases.

  • The xunit and json result files now contain the test variant information.

  • Avocado’s requirement management has been extended in a number of ways detailed below. In short, Avocado is now able to install requirements declared for a test in an isolated container if the --nrunner-spawner=podman is chosen (with a base image defined by --spawner-podman-image). That container image with the requirements fulfilled will only be built once, but a fresh environment will be used for every single test execution.

    • All requirements fulfilled will be recorded in a “cache database” so they will only need to be fulfilled once

    • Transparent support for the all spawners. This means that users of the podman spawner will have their requirements checked and fulfilled on the container image of their choice. As stated previously, that information will be kept in a requirement cache database. The container images will be available for super fast reuse after the first execution.

  • A new type of plugin, Cache, has been introduced and is responsible for manipulating with cache, and it’s representation through the Avocado interface. It is currently implemented for the vmimage and requirements cache.

  • Asset cache checksum can now use multiple hash algorithms. The ability to store multiple hashes, created by different algorithm to the cache CHECKSUM file was added. This is useful when different tests refer to the same asset, but use different hash algorithms.

  • Ansible modules can now be used as dependencies. For instance, instead of asking test writers to write code to, say, create user accounts that will be used during a test, a developer can simply use ansible’s user module. Likewise, if a test needs a service installed or up and running: instead of writing all that code, the test writer can describe that dependency using the service ansible module.

  • By setting the spawner.podman.avocado_spawner_egg configuration users can now control the exact Avocado package that will be automatically deployed within Podman containers.

  • When using the podman spawner, if the test URI looks like a filesystem path, it will be exposed read only to the container. This makes running Avocado tests inside containers transparent in many cases.

  • A Podman image can now be treated as a dependency. This is currently mostly useful to tests that will manually create containers, but it’s expected to also be leveraged by the podman spawner and suite and job level dependencies in the near future.

  • avocado.core.nrunner.runnable.Runnable instances will now, under some circumstances, automatically include the current environment’s relevant and required configuration. This makes standalone executions of avocado-runner-* commands easier, while previously it would require manually entering all required configuration.

  • The --failfast feature now acts on any "not OK" result. Previously, it would only act (that is, abort the job) upon a FAIL test result.

  • Test writers can now access a test’s status while the avocado.core.test.Test.tearDown() method is being run.

  • Support was added for Coverage.py when running avocado-instrumented tests (currently limited to the process spawner).

  • A results plugin for Beaker is now available and works out of the box without any manual configuration needed.

  • The task identifiers can now be configured as a format string that utilizes the attributes of the runnable. This allows users to define how test results will be named and presented.

  • When using the Job API, test suites can be enabled or disabled. Having a disabled test suite means it won’t be executed. This eases the creation of custom jobs where the user can choose to run a subset of suites.

  • The remote spawner is a new optional plugin that makes use of remote aexpect sessions (to remote hosts or equivalently remote containers behind remote hosts forwarded via specific ports) as slots to schedule test runs on.

  • The avocado-instrumented runner used to have an internal timeout of 24 hours that users might not have intended to use or respect. This internal timeout has been removed and configuration such as task.timeout.running or --job-timeout are the timeout users should worry about.

  • The json result file now contains the actual start and end time of tests in a format based on time.time().

  • The avocado jobs list command now presents a sorted list of jobs based on the modification of the results file.

  • If the status server is manually configured (that is, --status-server-disable-auto is set), and either the URI (--status-server-uri or the listen address is given (--status-server-listen) the other will automatically be set to the same value for convenience.

  • Resolvers can now choose to report a resolution result of type avocado.core.resolver.ReferenceResolutionResult.CORRUPT, which can be used when the reference seems to be accurate to the resolver in question, but something is broken in the resolved entity (such as a corrupt test file.

  • When running avocado-instrumented tests with a timeout defined, it’s possible to set a timeout_factor parameter that will either extend or shorten the actual timeout for that execution.

Bug Fixes

  • Avocado now presents the correct message when it’s interrupted with a CTRL+C (AKA SIGNINT, AKA KeyboardInterrupt).

  • The fetchasset plugin would fail when parsing some Python test files (in search of assets) and would produce a hard to follow error message (AttributeError: 'Subscript' object has no attribute 'id').

  • A runnable’s variant (and thus parameters) information was not being respected when using the Job API.

  • Test parameters given with the command line argument -p are now internally converted into variants values. This fixes the issue with those parameters not being displayed in the results.html files.

  • The variants.json file, saved at the jobdata directory inside a job’s result directory, now takes into consideration the possible multiple suites in a job. The files are now named after named with a numeric suffix and, if a name was given to the suite, a name suffix as well.

  • The serialization of the job configuration file, also saved in the jobdata directory, has been updated to support set data types.

  • Fixed a limit for exec-test and tap tests, where the STDOUT or STDERR buffers could be exhausted, and the test would get stuck forever on further writes. The exec-test and tap runners can now make use of the (optional) output_dir parameter, resulting in a much more efficient I/O handling. When output_dir is used, the only limitation on STDOUT and STDERR is the file-system itself.

  • Tests of type exec-test with a URI pointing to a file on the current working directory (that is, without an absolute directory prefix or a directory component to it) were not properly executed (“No such file or directory” was given). This is now fixed.

  • The automatic status server was very prone to failures due to AF_UNIX’s limitation on the length of paths, because it was created inside a job’s result directory. Now it’s created on the base system temporary directory, which makes it very hard to exceed the path length limit.

  • Avocado now supports actual file system limitation regarding the maximum file name length, instead of making assumptions about the limits. This allows Avocado to properly behave in environments such as containers backed by overlayfs.

  • A condition that would favor the configuration given to the job, instead of the more specific configuration given to the suite, was fixed.

  • A condition that could crash Avocado when a suite name wasn’t a filesystem safe name has been fixed.

  • Tasks that are “skipped” due to their timeouts being hit now include the appropriate reason.

  • The HTML report now contains an “ALL” selection for test types, instead of an empty value.

  • The independence of --status-server-uri and --status-server-listen was not being respected. Because Tasks are being created with whatever is the current status server URI (either determined automatically, or with the “listen” config), the “uri” config went unused. These issues are now fixed.

  • Running tests’ statuses are now properly marked as INTERRUPTED instead of CANCEL when they reach their own (or the job) timeout.

  • The avocado jobs show command used to show a simplified and possibly incorrect information about the spawner used. This information is no longer displayed, given that it’s a test suite attribute, and not really a job level information.

  • The loader for avocado-instrumented tests could end up using the wrong Python module if a module of the same name existed elsewhere in the import path. Now, the actual path of the Python file containing the test (given in the modulePath parameter) is used explicitly by the Python module importer.

  • When dependencies are not fulfilled, test results would be missing, instead of being clearly shown as CANCEL.

  • avocado.utils.path.init_dir() would raise FileExistsError if the directory already existed, usually causing crashes on its users.

  • The whiteboard file was being created with duplicate content because of the legacy runner implementation, which is now removed.

  • The avocado jobs show command now presents the correct time tests have ended.

  • The avocado.utils.download.url_open() function used to misleadingly says that a URL had been retrieved at a stage where only a response was obtained. It now presents an accurate message.

  • The Podman Spawner had a race condition where the state of the container (and thus the task) would not take into account the transition from “created” to “running”.

  • Avocado has re-enabled stdout and stderr files for avocado-instrumented files.

  • The Spawner interface and implementations now properly checks if the termination of a task was successful or not. The statemachine uses that information to let users know of situations where a task could not be terminated.

  • The tearDown() of avocado-instrumented now gets called properly if a test times out.

  • The Process Spawner now properly handles a situation where, during the termination of a task, the process itself finishes before the spawner has the chance to do so.

  • When interrupting avocado-instrumented tests, the tearDown() will be called to allow for cleanups. If an error occurred during the execution of tearDown() the status of the test would change to ERROR instead of keeping its original INTERRUPT status.

  • The HTML result plugin was not properly filtering tests based on their status.

  • The testlogs plugin was not showing tests with all “not ok” statuses and was showing test names instead of test IDs which are unique in a suite.

Utility APIs

Complete list of changes

For a complete list of changes between the last LTS release (92.0) and this release, please check out the Avocado commit changelog.