autopkgtest - Debian Perl Group autopkgtest implementation and team practices
The autopkgtest package,
which provides as-installed testing of Debian packages,
is integrated with pkg-perl packages as of version 3.5.1 (September 2014).
The integration is intended to allow for continuous integration testing at https://ci.debian.net/.
This document describes the implementation and best practices for the team.
Declaring
Testsuite: autopkgtest-pkg-perl
in the source package section of debian/control will activate an implicit test control file that's currently embedded in the autopkgtest package. This test control file has an indirection layer that pushes all the test logic into the pkg-perl-autopkgtest package, maintained inside the team. The entry point for the logic is the runner (https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/blob/master/autopkgtest/scripts/runner) script, which runs tests in /usr/share/pkg-perl-autopkgtest/*.d directories.
Thanks to the uniformity of CPAN distributions, this implicit control file works with most (currently about 75%) of the pkg-perl packages. The rest will need package specific tweaks, as detailed below.
An explicit test control file that corresponds to the implicit one described above is included in the pkg-perl-autopkgtest package in /usr/share/doc/pkg-perl-autopkgtest/examples and can be viewed at https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/blob/master/autopkgtest/examples/default-tests-control.
In case you want other tests, or need other dependencies or restrictions, you can add an explicit test control file and list them according the the spec manually. You can use the aforementioned default-tests-control file as a starting point.
The current tests activated by Testsuite: autopkgtest-pkg-perl are listed below. If some of them have expected failures, they can be skipped by listing the tests in debian/tests/pkg-perl/SKIP. The file can contain empty lines and comment lines starting with the # sign.
This test re-uses the build time test suite present in most CPAN distributions to check that changes in the package's dependencies haven't introduced regressions in the package functionality.
The package and its build dependencies are installed, the t/ directory and test.pl files are copied into a temporary directory, and the build time test suite is run against the installed modules. The temporary directory is used to make sure that the installed modules get used instead of the source tree.
If the test suite needs files outside the t/ directory, those can be listed in the debian/tests/pkg-perl/smoke-files. Empty lines and #-style comment lines are supported. Note that this overrides the default list of t/ and test.pl, so you normally need to include those too.
Some common test files are removed from the temporary directory; they currently include pod.t, pod-coverage.t, boilerplate.t, 04critic.t, and 97_meta.t. If the file debian/tests/pkg-perl/smoke-skip exists, it is used as a list of files to be removed. Empty lines and #-style comments are supported.
The environment variables AUTOMATED_TESTING and NONINTERACTIVE_TESTING are set to 1 by default. These can be overridden and others can be added in the configuration file debian/tests/pkg-perl/smoke-env. Empty lines and comments are allowed. The file syntax is one line per variable setting, in the form KEY=VAL; every such line is eval'ed with a shell so you can compute VAL dynamically, for example: DBDPG_INITDB=$(pg_config --bindir)/initdb.
The tests are run with the prove --merge option, which merges and synchronizes stderr and stdout from the test scripts. In some rare cases this option can break the test suite. It can be disabled by unsetting PKG_PERL_PROVE_ARGS in debian/tests/pkg-perl/smoke-env.
For the benefit of tests looking for something in lib/ or blib/, dummy module files are installed there and included in a dummy MANIFEST file. An empty MANIFEST.SKIP is also provided.
If the file debian/tests/pkg-perl/smoke-setup exists and is executable, it will be executed right before running the tests, with the working directory set to the package source and $TDIR set to the test running directory. The testing gets aborted if this program returns a nonzero value. Please use this facility responsibly and only as a last resort if the other hooks don't work for you.
If the file debian/tests/pkg-perl/smoke-cleanup exists and is executable, it will be executed when the smoke tests exit, with the working directory set to the package source and $TDIR set to the test running directory.
The smoke test by default runs prove with the --recurse option on the t/ directory (prior to 0.37 only the t/*.t tests were run). In case a package should run other test scripts, debian/tests/pkg-perl/smoke-tests can be used to declare which files should be used instead.
Example: t/*/*.t
This test ensures that the runtime dependencies are sufficient for the most basic usage of the module. Warnings from such basic usage are considered failures because they can be quite annoying on consumers of the module.
The package and its runtime dependencies are installed, and the command perl -w -MFoo::Bar -e1 is executed, where Foo::Bar is the name of the main module in the package. If this exits with a nonzero status, or if there is output on stderr, the test is considered failed. The test is repeated with PERL_DL_NONLAZY=1 set in the environment; this makes sure there are no unresolved symbols in binary modules.
The name of the module is read from META.json or META.yml, but it can be overridden in debian/tests/pkg-perl/use-name, which also allows listing multiple module names, one per line. The test is skipped if the name isn't present in one of the above sources.
There are some rare cases where warnings are to be expected even from the most basic usage. Such expected warnings can be listed in debian/tests/pkg-perl/use-whitelist. Each line in this file is interpreted as a regular expression.
This test checks that the runtime dependencies and recommendations cover all the requirements of the modules shipped in the package, and that the modules are syntactically valid Perl.
The test first installs the package and its runtime dependencies, including recommendations. All the .pm files in the package are then run through a syntax check with perl -wc file.pm. If the command exits with a nonzero status, the test is considered failed. Standard error output is not treated as a failure. If there are other files that you'd like to check, you can specify those as regular expressions in debian/tests/pkg-perl/syntax-extra. Don't include the match delimiters (//). Empty lines and #-style comments are supported.
If some of the module files have expected problems passing this test, they can be skipped by listing them in debian/tests/pkg-perl/syntax-skip. The lines are matched as fixed substrings (not regular expressions). Empty lines and #-style comments are supported.
Packages that Suggests other packages are exempt from this test, unless they include a (possibly empty) syntax-skip file. This is because some of the module files are expected to need the suggested packages to work, but there is no mechanism to install them in the environment provided by autopkgtest.
To test all module files despite a Suggests, an empty syntax-skip (with optional comments) can be used since pkg-perl-autopkgtest version 0.45.
Even with the implicit test control file, all the packages would need source uploads to include the Testsuite header, which would take quite a while to happen for all the team packages (currently numbering almost 3000). To speed up adoption, the current plan is to make https://ci.debian.net/ use a whitelist of known good pkg-perl packages that it will treat as having the Testsuite: autopkgtest-pkg-perl header. This whitelist is maintained at https://salsa.debian.org/ci-team/debian-ci-config and is based on test runs of all our packages such as https://lists.debian.org/debian-perl/2014/09/msg00100.html.
The whitelist is considered a temporary measure that can be dropped once we have Testsuite: autopkgtest-pkg-perl headers in all our packages.
As of May 2015, there are 2029 whitelisted packages, 378 packages that declare Testsuite: autopkgtest-pkg-perl, and 604 non-whitelisted pkg-perl packages that currently don't pass the default checks and therefore need manual work.
The list of packages needing manual work is available at https://people.debian.org/~ntyni/pkg-perl/autopkgtest-fail.txt and current as of 20150524, but is not actively updated at the moment (volunteers welcome.)
Once the set of known good packages has stabilized and is live on https://ci.debian.net/, we should do a mass commit in our git repository adding Testsuite: autopkgtest-pkg-perl headers to the known good packages.
A team specific lintian check (activated by lintian --profile pkg-perl) triggers on the lack of the Testsuite header and points to this document. Coupled with the mass commit above, affected packages are expected to be those that need tweaking to get the tests working. Maintainers encountering the lintian warning are encouraged to fix those (other team members are happy to help with that!), but they are also naturally free to ignore the warning and postpone the addition of the header.
When uploading a package with a Testsuite header, uploaders should adopt a habit of verifying that the tests pass so as not to cause unnecessary work to the https://ci.debian.net/ volunteers.
Bugs related to autopkgtest failures should be filed with
User: debian-perl@lists.debian.org Usertags: autopkgtest
so that they show up on the tracking list at https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=autopkgtest;users=debian-perl@lists.debian.org
See https://wiki.debian.org/bugs.debian.org/usertags for more information about usertags.
To set up a sbuild chroot for autopkgtest (formerly known as adt-run), follow the following steps.
You need to setup a chroot type that supports ephemeral sessions for autopkgtest to run properly. It will fail otherwise.
One option is to use lvm-snapshot, so you need to create a logical volume and mount it first. For instance, assuming that /dev/sdb1 is empty and available to be used as a physical volume:
sudo apt-get install lvm2
sudo pvcreate /dev/sdb1
sudo vgcreate sbuild /dev/sdb1
sudo lvcreate -L2G -n sid sbuild
sudo mkfs.ext4 /dev/sbuild/sid
sudo mkdir -p /srv/sbuild/sid
sudo mount /dev/sbuild/sid /srv/sbuild/sid
A simpler alternative would be to use directory type, which doesn't need any previous setup before it is created, just a regular directory (ephemeral sessions will be setup through union filesystem).
Note that sbuild uses schroot underneath to manage the chroots.
Note that installing recommends is needed so all the autopkgtest tools are available later. Especially a missing autodep8 package will lead to grief and sorrow.
sudo apt-get install --install-recommends sbuild autopkgtest
sudo sbuild-createchroot sid /srv/sbuild/sid http://deb.debian.org/debian
At this point you can run sudo umount /srv/sbuild/sid if you used LVM.
sudo adduser {username} sbuild
First, you should rename the file created automatically on the previous step (it's not mandatory but useful):
sudo mv /etc/schroot/chroot.d/sid-amd64-sbuild-* /etc/schroot/chroot.d/sid-amd64-sbuild.conf
Then, you have to setup the chroot properties for the directory type to make sure ephemeral sessions can be created:
sudo sed -i -e 's,^union-type=none,union-type=aufs,' /etc/schroot/chroot.d/sid-amd64-sbuild.conf
Or lvm-snapshot type replacing the contents with these:
[sid-amd64-sbuild]
type=lvm-snapshot
lvm-snapshot-options=--size 2G
device=/dev/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
profile=sbuild
description=Debian sid/amd64 autobuilder
Use this command to update the package list, perform a dist upgrade, and autoremove obsolete packages in the chroot.
sudo sbuild-update -udr sid-amd64-sbuild
In case you need to make other kind of changes to the chroot.
sudo schroot -c source:sid-amd64-sbuild
Play with an ephemeral chroot based on the source chroot setup before (you do not need to be root at this point). Please, be warned that this chroot instance will be automatically removed when you exit the shell.
schroot -c sid-amd64-sbuild
Sometimes you need to mount other filesystem or directories in the chroot (e.g. /home). If you miss anything, you can add that directory in the file /etc/schroot/sbuild/fstab, where sbuild is the profile for the chroots created by sbuild.
You can find several examples in the desktop and default profiles.
See Enrico Zini's blog entry at https://www.enricozini.org/blog/2008/tips/joys-of-schroot/ and/or sbuild-createchroot(8) for details.
You can use eatmydata do improve the speed of your builds inside the chroot pretty easily.
eatmydata is installed in the chroot
You can either install it manually in the source chroot or add --include=eatmydata option before running sbuild-createchroot above.
eatmydata in the chroot configuration
Add command-prefix=eatmydata to /etc/schroot/chroot.d/sid-amd64-sbuild.conf.
See https://wiki.debian.org/sbuild#Using_eatmydata_with_sbuild for more details.
debian/tests/pkg-perl/SKIP: libtest-file-sharedir-perldebian/tests/pkg-perl/smoke-skip: libtext-csv-xs-perldebian/tests/pkg-perl/smoke-files: libnet-ldap-perldebian/tests/pkg-perl/use-name: libnet-radius-perldebian/tests/pkg-perl/syntax-skip: libplack-perldebian/tests/pkg-perl/smoke-env: libsvn-notify-mirror-perlprove --merge in debian/tests/pkg-perl/smoke-env: libterm-progressbar-perldebian/tests/pkg-perl/smoke-setup: libnet-sftp-sftpserver-perldebian/tests/pkg-perl/smoke-cleanup: libmemcached-libmemcached-perlautopkgtest against a package in the archive, assuming your schroot name is sid-amd64-sbuild:autopkgtest libfoo-bar-perl -- schroot sid-amd64-sbuild # since autopkgtest 4.0 adt-run libfoo-bar-perl --- schroot sid-amd64-sbuild
(Make sure that schroot source:sid-amd64-sbuild has a deb-src entry in sources.list and that you have run apt-get update there at least once before trying this example.)
autopkgtest on a .changes file:autopkgtest libfoo-bar-perl_0.01_amd64.changes -- schroot sid-amd64-sbuild # since autopkgtest 4.0 adt-run libfoo-bar-perl_0.01_amd64.changes --- schroot sid-amd64-sbuild
autopkgtest with the control information in the current directory but the actual package from the archive:autopkgtest -B . -- schroot sid-amd64-sbuild # since autopkgtest 4.0 adt-run -B .// --- schroot sid-amd64-sbuild
(The source tree doesn't need to be built, the binary package is installed from the archive with apt.)
See /usr/share/doc/autopkgtest/README.running-tests.rst.gz for more information.
Copyright (c) 2014-2022 by the individual authors and contributors noted above. All rights reserved. This document is free software; you may redistribute it and/or modify it under the same terms as Perl itself
Perl is distributed under your choice of the GNU General Public License or the Artistic License. On Debian GNU/Linux systems, the complete text of the GNU General Public License can be found in `/usr/share/common-licenses/GPL' and the Artistic License in `/usr/share/common-licenses/Artistic'.