103.0 LTS
The Avocado team is proud to present another LTS (Long Term Stability) release: Avocado 103.0, AKA “Sound of Freedom”, is now available!
LTS Release
For more information on what a LTS release means, please read RFC: Long Term Stability.
Important Announcement
Since the previous LTS version (92.0), Avocado has removed the
legacy runner implementation (internally simply called runner
).
While the runner
implementation is not the default on version 92.x
(nrunner
is the default) it was still available as an option. Now
nrunner
is the only available runner implementation shipped by
default.
Users migrating from Avocado 92.x and using the legacy runner will be impacted by this change and should act accordingly.
If your Avocado configuration file has a [run]
section with
test_runner
set to runner
, please keep in mind that:
The
runner
implementation no longer existsThe
test_runner
was renamed tosuite_runner
(along with the rename of the command line option--test-runner
to--suite-runner
). This is only relevant if you have a custom runner and want to use it.
While most of the functionality of the legacy runner is available
under the new runner implementation (nrunner
) there are some
differences and even missing features. Known nrunner issues
are being tracked on our GitHub project page, with the nrunner
tag, and new issue reports are appreciated.
Upgrading from 92.x to 103.0
Upgrading Installations
Avocado is available on a number of different repositories and installation methods. You can find the complete details here.
After looking at your installation options, please consider the following when planning an in-place upgrade or a deployment version bump:
When using Python’s own package management, that is,
pip
, simply choose a version lower to the next Avocado release to benefit from minor releases (bugfixes) in this LTS series. In short,avocado-framework<104.0
will get you the latest release of this LTS series.When using RPM packages on Linux distributions, check if there’s a particular repository that provides LTS packages only. On distributions that provide “modules”, you may find a
103lts
stream. If no modules are available, use your package manager tools (such as DNF’s versionlock plugin) to pin the installation to the 92.x versions.
Porting Tests
Depending on the test type (say, avocado-instrumented
or
exec-test
), test writers will have to observe different aspects of
the changes since 92.0 LTS. This section deals primarily with
avocado-instrumented
tests.
To be considered an avocado-instrumented
test, the test is based
on the avocado.test.Test
class. The
API available in that class is called the “Test API”. It’s also
common, but not mandatory, for such tests to leverage the modules
under the avocado.utils
namespace, collectively known as the
Utility APIs.
With that in mind, the two sections below describe issues you should consider when moving your tests from Avocado 92.X LTS to 103.0 LTS.
Test API compatibility
Fortunately, there have been no changes in the public Test API since 92.0 LTS that should impact tests.
If you’re interested in knowing about all the internal implementation changes, though, feel free to inspect them with a command such as:
$ git diff 92.0..103.0 -- avocado/core/test.py
$ git log 92.0..103.0 -- avocado/core/test.py
Utility API compatibility
The changes in the utility APIs (those that live under the
avocado.utils
namespace) are too many to present porting
suggestions. Please refer to the Utility APIs section
for a comprehensive list of changes, including new features your test
may be able to leverage.
Changes from previous LTS
Note
This is not a collection of all changes encompassing all releases from 92.0 to 103.0. This list contains changes that are relevant to users of 92.0, when evaluating an upgrade to 103.0.
When compared to the last LTS (version 92.0), the main changes introduced by this versions are:
Users / Test Writers
Python 3.10, 3.11 and 3.12 are formally supported (along with 3.7, 3.8 and 3.9)
Command line options prefixed with
--nrunner-
had this prefix removed. A command line option such as--nrunner-spawner
is now simply--spawner
. The matching configuration options such asnrunner.something
becamerun.something
. This is due to the fact thatnrunner
is now the only runner implementation offered by default, so the differentiation and extra typing seems unnecessary. If other runners are added in the future (or by custom, out of tree, plugins) they can choose to respect the existing options if they apply.A contrib script that provides the features of the legacy
--external-runner
feature has been added. It’s built on the Job API andnrunner
architecture.There were major changes to the Avocado logging behavior, most of them to address feedback from users since the previous logging changes:
The root logger handler was restored. This enables all loggers out of
avocado.*
namespace by default. If a test, either directly or indirectly through 3rd party libraries, logs into any namespace (saylogging.getLogger('my-library')
) it will show up in the Avocado’s test logs.The job.log file continues to contain logs for the avocado.job namespace, but a new file called
full.log
contains all generated logs for a job, including logs from all tests.
The
runner.output.utf8
andcore.input_encoding
were settings were removed, and now default to the system’s setting (by usingsys.getdefaultencoding()
).Test results files (in a job’s result directory) can now be accessed by their result status. For instance, assuming the latest job results are stored in
~/avocado/job-results/latest
(the default location), users can find the results for tests that ended inFAIL
at~/avocado/job-results/latest/test-results/by-status/FAIL.
Avocado will now print at the end of the job a more descriptive list of tests that finished with
"not OK"
statuses.The
avocado jobs get-output-files
command was removed. Its original intended purpose has not been relevant for some time, and the preservation of output files are already handled directly by all spawner implementations.The Avocado Resolver now allows tests that are implemented in files (by far the most common scenario) and that may also access test data files (see Accessing test data files) to pass that information along to spawners. The effect of that is that when running such tests on “remote” spawner environments (such as “podman”) the test file and the data files can be made available in the remote environment. This is currently enabled in
avocado-instrumented
,python-unittest
,exec-test
andtap
tests.User of macOS will have a better experience when using Avocado. The full set of Avocado’s selftests are now run under macOS on CI. Please be advised that macOS is not currently supported at the same level of Linux-based operating systems due to the lack of contributors/maintainers with access to the needed hardware. If you are a user/developer and are willing to contribute to this, please let the Avocado team know.
Sysinfo collection is now fully supported in
nrunner
, including per-test collection of system information.New plugin interfaces,
PreTest
andPostTest
, allow actions to be executed right before or after the execution of a test, in the same spawner environment as the test itself.Plugins can now have a builtin priority in relation to other plugins of the same type that will affect its execution order. This is in addition to the configurable
plugins.$type.order
settings.The
dict_variants
plugin now allows the configuration of the keys that will determine the variant ID.Environment variables such as
AVOCADO_VERSION
,AVOCADO_TEST_WORKDIR
,AVOCADO_TEST_BASEDIR
,AVOCADO_TEST_LOGDIR
andAVOCADO_TEST_LOGFILE
are now made available toexec-test
.Result plugins such as
json
andxunit
now contain more accurate values reflecting Avocado’s concepts of a test’s ID and name.The
xunit
(AKAjunit
) result file now contains the class, test’s file names and better error information for test cases.The
xunit
andjson
result files now contain the test variant information.Avocado’s requirement management has been extended in a number of ways detailed below. In short, Avocado is now able to install requirements declared for a test in an isolated container if the
--nrunner-spawner=podman
is chosen (with a base image defined by--spawner-podman-image
). That container image with the requirements fulfilled will only be built once, but a fresh environment will be used for every single test execution.All requirements fulfilled will be recorded in a “cache database” so they will only need to be fulfilled once
Transparent support for the all spawners. This means that users of the podman spawner will have their requirements checked and fulfilled on the container image of their choice. As stated previously, that information will be kept in a requirement cache database. The container images will be available for super fast reuse after the first execution.
A new type of plugin,
Cache
, has been introduced and is responsible for manipulating with cache, and it’s representation through the Avocado interface. It is currently implemented for thevmimage
andrequirements
cache.Asset cache checksum can now use multiple hash algorithms. The ability to store multiple hashes, created by different algorithm to the cache
CHECKSUM
file was added. This is useful when different tests refer to the same asset, but use different hash algorithms.Ansible modules can now be used as dependencies. For instance, instead of asking test writers to write code to, say, create user accounts that will be used during a test, a developer can simply use ansible’s user module. Likewise, if a test needs a service installed or up and running: instead of writing all that code, the test writer can describe that dependency using the service ansible module.
By setting the
spawner.podman.avocado_spawner_egg
configuration users can now control the exact Avocado package that will be automatically deployed within Podman containers.When using the podman spawner, if the test URI looks like a filesystem path, it will be exposed read only to the container. This makes running Avocado tests inside containers transparent in many cases.
A Podman image can now be treated as a dependency. This is currently mostly useful to tests that will manually create containers, but it’s expected to also be leveraged by the podman spawner and suite and job level dependencies in the near future.
avocado.core.nrunner.runnable.Runnable
instances will now, under some circumstances, automatically include the current environment’s relevant and required configuration. This makes standalone executions ofavocado-runner-*
commands easier, while previously it would require manually entering all required configuration.The
--failfast
feature now acts on any"not OK"
result. Previously, it would only act (that is, abort the job) upon aFAIL
test result.Test writers can now access a test’s status while the
avocado.core.test.Test.tearDown()
method is being run.Support was added for
Coverage.py
when runningavocado-instrumented
tests (currently limited to the process spawner).A results plugin for Beaker is now available and works out of the box without any manual configuration needed.
The task identifiers can now be configured as a format string that utilizes the attributes of the runnable. This allows users to define how test results will be named and presented.
When using the Job API, test suites can be enabled or disabled. Having a disabled test suite means it won’t be executed. This eases the creation of custom jobs where the user can choose to run a subset of suites.
The remote spawner is a new optional plugin that makes use of remote aexpect sessions (to remote hosts or equivalently remote containers behind remote hosts forwarded via specific ports) as slots to schedule test runs on.
The
avocado-instrumented
runner used to have an internal timeout of 24 hours that users might not have intended to use or respect. This internal timeout has been removed and configuration such astask.timeout.running
or--job-timeout
are the timeout users should worry about.The
json
result file now contains the actual start and end time of tests in a format based ontime.time()
.The
avocado jobs list
command now presents a sorted list of jobs based on the modification of the results file.If the status server is manually configured (that is,
--status-server-disable-auto
is set), and either the URI (--status-server-uri
or the listen address is given (--status-server-listen
) the other will automatically be set to the same value for convenience.Resolvers can now choose to report a resolution result of type
avocado.core.resolver.ReferenceResolutionResult.CORRUPT
, which can be used when the reference seems to be accurate to the resolver in question, but something is broken in the resolved entity (such as a corrupt test file.When running
avocado-instrumented
tests with atimeout
defined, it’s possible to set atimeout_factor
parameter that will either extend or shorten the actual timeout for that execution.
Bug Fixes
Avocado now presents the correct message when it’s interrupted with a
CTRL+C
(AKASIGNINT
, AKAKeyboardInterrupt
).The
fetchasset
plugin would fail when parsing some Python test files (in search of assets) and would produce a hard to follow error message (AttributeError: 'Subscript' object has no attribute 'id'
).A runnable’s variant (and thus parameters) information was not being respected when using the Job API.
Test parameters given with the command line argument
-p
are now internally converted into variants values. This fixes the issue with those parameters not being displayed in theresults.html
files.The
variants.json
file, saved at the jobdata directory inside a job’s result directory, now takes into consideration the possible multiple suites in a job. The files are now named after named with a numeric suffix and, if a name was given to the suite, a name suffix as well.The serialization of the job configuration file, also saved in the
jobdata
directory, has been updated to support set data types.Fixed a limit for
exec-test
andtap
tests, where theSTDOUT
orSTDERR
buffers could be exhausted, and the test would get stuck forever on further writes. The exec-test and tap runners can now make use of the (optional) output_dir parameter, resulting in a much more efficient I/O handling. When output_dir is used, the only limitation onSTDOUT
andSTDERR
is the file-system itself.Tests of type
exec-test
with a URI pointing to a file on the current working directory (that is, without an absolute directory prefix or a directory component to it) were not properly executed (“No such file or directory” was given). This is now fixed.The automatic status server was very prone to failures due to
AF_UNIX
’s limitation on the length of paths, because it was created inside a job’s result directory. Now it’s created on the base system temporary directory, which makes it very hard to exceed the path length limit.Avocado now supports actual file system limitation regarding the maximum file name length, instead of making assumptions about the limits. This allows Avocado to properly behave in environments such as containers backed by overlayfs.
A condition that would favor the configuration given to the job, instead of the more specific configuration given to the suite, was fixed.
A condition that could crash Avocado when a suite name wasn’t a filesystem safe name has been fixed.
Tasks that are “skipped” due to their timeouts being hit now include the appropriate reason.
The HTML report now contains an “ALL” selection for test types, instead of an empty value.
The independence of
--status-server-uri
and--status-server-listen
was not being respected. Because Tasks are being created with whatever is the current status server URI (either determined automatically, or with the “listen” config), the “uri” config went unused. These issues are now fixed.Running tests’ statuses are now properly marked as
INTERRUPTED
instead ofCANCEL
when they reach their own (or the job) timeout.The
avocado jobs show
command used to show a simplified and possibly incorrect information about the spawner used. This information is no longer displayed, given that it’s a test suite attribute, and not really a job level information.The loader for
avocado-instrumented
tests could end up using the wrong Python module if a module of the same name existed elsewhere in the import path. Now, the actual path of the Python file containing the test (given in themodulePath
parameter) is used explicitly by the Python module importer.When dependencies are not fulfilled, test results would be missing, instead of being clearly shown as
CANCEL
.avocado.utils.path.init_dir()
would raiseFileExistsError
if the directory already existed, usually causing crashes on its users.The whiteboard file was being created with duplicate content because of the legacy runner implementation, which is now removed.
The
avocado jobs show
command now presents the correct time tests have ended.The
avocado.utils.download.url_open()
function used to misleadingly says that a URL had been retrieved at a stage where only a response was obtained. It now presents an accurate message.The Podman Spawner had a race condition where the state of the container (and thus the task) would not take into account the transition from “created” to “running”.
Avocado has re-enabled
stdout
andstderr
files foravocado-instrumented
files.The Spawner interface and implementations now properly checks if the termination of a task was successful or not. The statemachine uses that information to let users know of situations where a task could not be terminated.
The
tearDown()
ofavocado-instrumented
now gets called properly if a test times out.The Process Spawner now properly handles a situation where, during the termination of a task, the process itself finishes before the spawner has the chance to do so.
When interrupting
avocado-instrumented
tests, thetearDown()
will be called to allow for cleanups. If an error occurred during the execution oftearDown()
the status of the test would change toERROR
instead of keeping its originalINTERRUPT
status.The HTML result plugin was not properly filtering tests based on their status.
The
testlogs
plugin was not showing tests with all “not ok” statuses and was showing test names instead of test IDs which are unique in a suite.
Utility APIs
avocado.utils.nvme
added new functions, such as:avocado.utils.multipath
added a new function:avocado.utils.multipath.get_mpath_paths_status()
that returns the status of all paths of a mpath device was introduced.
-
Received updates to support more recent versions of the UnionTech OS.
Added specific detection for CentOS Stream.
Improved detection for Amazon Linux.
-
Introduced a utility to check if a given CPU is hotpluggable
-
The
avocado.utils.ssh.Session.cmd()
now supports setting atimeout
for the command execution.
-
Removed deprecated modules and methods.
Adds a missing network mask prefix when creating static configurations.
Added
avocado.utils.network.hosts.Host.validate_mac_addr()
which checks if a given MAC address is valid.A function to check packet loss was added to
avocado.utils.network.interfaces
avocado.utils.network.interfaces.NetworkInterface.is_bond()
that allows users to check if a given interface is a bonding device.The
avocado.utils.network.ports
fixed some wrong premises regarding the availability of open ports for different protocols (such as a free TCP versus a free UDP port).
-
Introduced
avocado.utils.disk.get_disks_by_id()
which returns all disks by device ids.
-
Added support for Zstandard uncompression.
-
Received the changes necessary to cope with changes in
signal.default_int_handler()
. It now passes all the given arguments along.
avocado.utils.software_manager
:Allows DNF/YUM repository options to be customized.
-
A new method was added that checks the existence of a software raid device.
-
A new
avocado.utils.podman.Podman.get_container_info()
method introduced.
-
Now allows for a finer grained usage of the functionality in
avocado.utils.cloudinit.PhoneHomeServer
.
-
Ubuntu’s provider now properly handles the version number when it compares versions with trailing zeroes.
Ubuntu and OpenSUSE providers can now fetch the best (latest) version available when no version is given.
OpenSUSE provider will now use OpenStack images starting from version 15.3, due to the other images having been discontinued.
Uses https://cloud.debian.org for obtaining Debian Cloud images.
Complete list of changes
For a complete list of changes between the last LTS release (92.0) and this release, please check out the Avocado commit changelog.