52.0 LTS
The Avocado team is proud to present another release: Avocado version 52.0, the second Avocado LTS version.
What’s new?
When compared to the last LTS (v36), the main changes introduced by this versions are:
Support for TAP[2] version 12 results, which are generated by default in test results directory (
results.tap
file).The download of assets in tests now allow for an expiration time.
Environment variables can be propagated into tests running on remote systems.
The plugin interfaces have been moved into the
avocado.core.plugin_interfaces
module.Support for running tests in a Docker container.
Introduction of the “Fail Fast” feature (
--failfast on
option) to therun
command, which interrupts the Job on a first test failure.Special keyword
latest
for replaying previous jobs.Support to replay a Job by path (in addition to the Job ID method and the
latest
keyword).Diff-like categorized report of jobs (
avocado diff <JOB_1> <JOB_2>
).The introduction of a
rr
based wrapper.The automatic VM IP detection that kicks in when one uses
--vm-domain
without a matching--vm-hostname
, now uses a more reliable method (libvirt/qemu-gust-agent query).Set
LC_ALL=C
by default on sysinfo collection to simplify avocado diff comparison between different machines.Result plugins system is now pluggable and the results plugins (JSON, XUnit, HTML) were turned into stevedore plugins. They are now listed in the
avocado plugins
command.Multiplexer was replaced with Varianter plugging system with defined API to register plugins that generate test variants.
Old
--multiplex
argument, which used to turn yaml files into variants, is now handled by an optional plugin calledyaml_to_mux
and the--multiplex
option is being deprecated in favour of the--mux-yaml
option, which behaves the same way.It’s now possible to disable plugins by using the configuration file.
Better error handling of the virtual machine plugin (
--vm-domain
and related options).When discovering tests on a directory, the result now is a properly alphabetically ordered list of tests.
Plugins can now be setup in Avocado configuration file to run at a specific order.
Support for filtering tests by user supplied “tags”.
Users can now see the test tags when listing tests with the
-V
(verbose) option.Users can now choose to keep the complete set of files, including temporary ones, created during an Avocado job run by using the
--keep-tmp
option (e.g. to keep those files forrr
).Tests running with the external runner (
--external-runner
) feature will now have access to the extended behavior for SIMPLE tests, such as being able to exit a test with the WARNING status.Encoding support was improved and now Avocado should safely treat localized test-names.
Test writers now have access to a test temporary directory that will last not only for the duration of the test, but for the duration of the whole job execution to allow sharing state/exchanging data between tests. The path for that directory is available via Test API (
self.teststmpdir
) and via environment variable (AVOCADO_TESTS_COMMON_TMPDIR
).Avocado is now available on Fedora standard repository. The package name is
python2-avocado
. The optional plugins and examples packages are also available. Rundnf search avocado
to list them all.Optional plugins and examples packages are also available on PyPI under
avocado-framework
name.Avocado test writers can now use a family of decorators, namely
avocado.skip()
,avocado.skipIf()
andavocado.skipUnless()
to skip the execution of tests.Sysinfo collection based on command execution now allows a timeout to be set in the Avocado configuration file.
The non-local runner plugins, the html plugin and the yaml_to_mux plugin are now distributed in separate packages.
The Avocado main process will now try to kill all test processes before terminating itself when it receives a SIGTERM.
Support for new type of test status,
CANCEL
, and of course the mechanisms to set a test with this status (e.g. viaself.cancel()
).avocado.TestFail
,avocado.TestError
andavocado.TestCancel
are now public Avocado Test APIs, available in the main avocado namespace.Introduction of the robot plugin, which allows Robot Framework tests to be listed and executed natively within Avocado.
A brand new ResultsDB optional plugin.
Listing of supported loaders (
--loaders \?
) was refined.Variant-IDs generated by yaml_to_mux plugin now include leaf node names to make them more meaningful, making easier to skim through the results.
yaml_to_mux now supports internal filters defined inside the YAML file expanding the filtering capabilities even further.
Avocado now supports resuming jobs that were interrupted.
The HTML report now presents the test ID and variant ID in separate columns, allowing users to also sort and filter results based on those specific fields.
The HTML report will now show the test parameters used in a test when the user hovers the cursor over the test name.
Avocado now reports the total job execution time on the UI, instead of just the tests execution time.
New
avocado variants
has been added which supersedes theavocado multiplex
.Loaders were tweaked to provide more info on
avocado list -V
especially when they don’t recognize the reference.Users can use
--ignore-missing-references on
to run a job with undiscovered test referencesUsers can now choose in which order the job will execute tests (from its suite) and variants. The two available options are
--execution-order=variants-per-test
(default) or--execution-order=tests-per-variant
.Test methods can be recursively discovered from parent classes by upon the
:avocado: recursive
docstring directive.
Besides the list above, we had several improvements in our utils
libraries that are important for test writers, some of them are listed
below:
time_to_seconds
,geometric_mean
andcompare_matrices
were added inavocado.utils.data_structures
.avocado.utils.distro
was refined.Many
avocado.utils
new modules were introduced, likefilelock
,lv_utils
,multipath
,partition
andpci
.avocado.utils.memory
contains several new methods.New
avocado.utils.process.SubProcess.get_pid()
method.sudo
support inavocado.utils.process
was improvedThe
avocado.utils.process
library makes it possible to ignore spawned background processes.New
avocado.utils.linux_modules.check_kernel_config()
.Users of the
avocado.utils.process
module will now be able to access the process ID in theavocado.utils.process.CmdResult
.Improved
avocado.utils.iso9660
with a more complete standard API across all back-end implementations.Improved
avocado.utils.build.make()
, which will now return the make process exit status code.The
avocado.Test
class now better exports (and protects) the core class attributes members (such asparams
andrunner_queue
).avocado.utils.linux_modules
functions now returns module name, size, submodules if present, filename, version, number of modules using it, list of modules it is dependent on and finally a list ofparams
.
It is also worth mentioning:
Improved documentation, with new sections to Release Notes and Optional Plugins, very improved Contribution and Community Guide. New content and new examples everywhere.
The avocado-framework-tests GitHub organization was founded to encourage companies to share Avocado tests.
Bugs were always handled as high priority and every single version was delivered with all the reported bugs properly fixed.
When compared to the last LTS, we had:
1187 commits (and counting).
15 new versions.
4811 more lines of Python code (+27,42%).
1800 more lines of code comment (+24,67%).
31 more Python files (+16,48%).
69 closed GitHub issues.
34 contributors from at least 12 different companies, 26 of them contributing for the fist time to the project.
Switching from 36.4 to 52.0
You already know what new features you might expect, but let’s emphasize the main changes required to your workflows/tests when switching from 36.4 to 52.0
Installation
All the previously supported ways to install Avocado are still valid and few new ones were added, but beware that Avocado was split into several optional plugins so you might want to adjust your scripts/workflows.
Multiplexer (the YAML parser which used to generate variants) was turned into an optional plugin
yaml_to_mux
also known asavocado_framework_plugin_varianter_yaml_to_mux
. Without it Avocado does not require PyYAML, but you need it to support the parsing of YAML files to variants (unless you use a different plugin with similar functionality, which is now also possible).The HTML result plugin is now also an optional plugin so one has to install it separately.
The remote execution features (
--remote-hostname
,--vm-domain
,--docker
) were also turned into optional plugins so if you need those you need to install them separately.Support for virtual environment (
venv
) was greatly improved and we do encourage people who want to usepip
to do that via this method.
As for the available ways:
Fedora/RHEL can use our custom repositories, either LTS-only or all releases. Note that latest versions (non-lts) are also available directly in Fedora and also in EPEL.
OpenSUSE - Ships the 36 LTS versions, hopefully they’ll start shipping the 52 ones as well (but we are not in charge of that process)
Debian - The
contrib/packages/debian
script is still available, although un-maintained for a long timePyPI/pip - Avocado as well as all optional plugins are available in PyPI and can be installed via
pip install avocado-framework\*
, or selectively one by one.From source - Makefile target
install
is still available but it does not install the optional plugins. You have to install them one by one by going to their directory (eg.cd optional_plugins/html
and runningsudo python setup.py install
)
As before you can find the details in Installing Avocado.
Usage
Note
As mentioned in previous section some previously core features were turned into optional plugins. Do check your install script if some command described here are missing on your system.
Most workflows should work the same, although there are few little changes and a few obsoleted constructs which are still valid, but you should start using the new ones.
The hard changes which does not provide backward compatibility:
- Human result was tweaked a bit:
The
TESTS
entry (displaying number of tests) was removed as one can easily get this information fromRESULTS
.Instead of tests time (sum of test times) you get job time (duration of the job execution) in the human result
- Json results also contain some changes:
They are pretty-printed
As
cancel
status was introduced, json result contain an entry of number of canceled tests (cancel
)url was renamed to id (url entry is to be removed in
53.0
so this is actually a soft change with a backward compatibility support)
The
avocado multilex|variants
does not expect multiplex YAML files as positional arguments, one has to use-m|--mux-yaml
followed by one or more paths.Test variants are not serialized numbers anymore in the default
yaml_to_mux
(multiplexer
), but ordered list of leaf-node names of the variant followed by hash of the variant content (paths+environment). Therefore instead ofmy_test:1
you can get something likemy_test:arm64-virtio_scsi-RHEL7-4a3c
.results.tap
is now generated by default in job results along theresults.json
andresults.xml
(unless disabled)The
avocado run --replay
andavocado diff
are unable to parse results generated by36.4
to this date. We should be able to introduce such feature with not insignificant effort, but no one was interested yet.
And the still working but to be removed in 53.0
constructs:
The long version of the
-m|--multiplex
argument available inavocado run|multiplex|variants
was renamed to-m|--mux-yaml
which corresponds better to the rest of--mux-*
arguments.The
avocado multiplex
was renamed toavocado variants
The
avocado multiplex|variants
arguments were reworked to better suite the possible multiple varianter plugins:Instead of picking between
tree
representation of list of variants one can use--summary
, resp--variants
followed by verbosity, which supersedes-c|contents
,-t|--tree
,-i|--inherit
Instead of
--filter-only|--filter-out
the--mux-filter-only|--mux-filter-out
are availableThe
--mux-path
is now also available inavocado multiplex|variants
Test API
Main features stayed the same, there are few new ones so do check our documentation for details. Anyway while porting tests you should pay attention to following changes:
If you were overriding
avocado.Test
attributes (eg.name
,params
,runner_queue
, …) you’ll get anAttributeError: can't set attribute
error as most of them were turned into properties to avoid accidental override of the important attributes.The
tearDown
method is now executed almost always (always when thesetUp
is entered), including when the test is interrupted while runningsetUp
. This might require some changes to yoursetUp
andtearDown
methods but generally it should make them simpler. (See Setup and cleanup methods and following chapters for details)Test exceptions are publicly available directly in
avocado
(TestError
,TestFail
,TestCancel
) and when raised inside test they behave the same way asself.error
,self.fail
orself.cancel
. (Seeavocado
)New status is available called
CANCEL
. It means the test (or even justsetUp
) started but the test does not match prerequisites. It’s similar toSKIP
in other frameworks, but theSKIP
result is reserved for tests that were not executed (nor thesetUp
was entered). TheCANCEL
status can be signaled byself.cancel
or by raisingavocado.TestCancel
exception and theSKIP
should be set only byavocado.skip
,avocado.skipIf
oravocado.skipUnless
decorators. Theself.skip
method is still supported but will be removed after in53.0
so you should replace it byself.cancel
which has similar meaning but it additionally executes thetearDown
. (See Test statusesThe
tag
argument ofavocado.Test
was removed as it is part ofname
, which can only beavocado.core.test.TestName
instance. (Seeavocado.core.test.Test()
)The
self.job.logdir
which used to be abused to share state/data between tests inside one job can now be dropped towards theself.teststmpdir
, which is a shared temporary directory which sustains throughout job execution and even between job executions if set viaAVOCADO_TESTS_COMMON_TMPDIR
environmental value. (Seeavocado.core.test.Test.teststmpdir()
)Those who write inherited test classes will be pleasantly surprised as it is now possible to mark a class as avocado test including all
test*
methods coming from all parent classes (similarly to how dynamic discovery works inside Python unittest, see docstring-directive-recursive for details)The
self.text_output
is not published after the test execution. If you were using it simply open theself.logfile
and read the content yourself.
Utils API
Focusing only on the changes you might need to adjust the usage of:
avocado.utils.build.make
calls as it now reports onlyexit_status
. To get the full result object you need to executeavocado.utils.build.run_make
.avocado.utils.distro
reportsRed Hat Enterprise Linux
/rhel
instead ofRed Hat
/redhat
.avocado.process
where the check for availability ofsudo
was improved, which might actually start executing some code which used to fail in 36.4.
Also check out the avocado.utils
for complete list of available
utils as there were many additions between 36.4 and 52.0.
Complete list of changes
For a complete list of changes between the last LTS release (36.4) and this release, please check out the Avocado commit changelog.