=head1 NAME

autopkgtest - Debian Perl Group autopkgtest implementation and team practices

=head1 INTRODUCTION

The C<autopkgtest> package, which provides as-installed testing of Debian
packages, is integrated with pkg-perl packages as of version 3.5.1
(September 2014). The integration is intended to allow for continuous
integration testing at L<https://ci.debian.net/>.

This document describes the implementation and best practices for
the team.

=head1 IMPLEMENTATION

Declaring

 Testsuite: autopkgtest-pkg-perl

in the source package section of C<debian/control> will activate an
implicit test control file that's currently embedded in the C<autopkgtest>
package. This test control file has an indirection layer that
pushes all the test logic into the C<pkg-perl-autopkgtest> package,
maintained inside the team. The entry point for the logic is the
runner (L<https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/blob/master/autopkgtest/scripts/runner>)
script, which runs tests in C</usr/share/pkg-perl-autopkgtest/*.d>
directories.

Thanks to the uniformity of CPAN distributions, this implicit control
file works with most (currently about 75%) of the pkg-perl packages.
The rest will need package specific tweaks, as detailed below.

An explicit test control file that corresponds to the implicit one
described above is included in the C<pkg-perl-autopkgtest> package in
C</usr/share/doc/pkg-perl-autopkgtest/examples> and can be viewed at
L<https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/blob/master/autopkgtest/examples/default-tests-control>.

In case you want other tests, or need other dependencies or
restrictions, you can add an explicit test control file
and list them according the the spec manually. You can use
the aforementioned C<default-tests-control> file as
a starting point.

The current tests activated by C<Testsuite: autopkgtest-pkg-perl> are
listed below. If some of them have expected failures, they can be
skipped by listing the tests in C<debian/tests/pkg-perl/SKIP>.
The file can contain empty lines and comment lines starting with the C<#> sign.

=head2 smoke 

Source: L<https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/blob/master/autopkgtest/scripts/build-deps.d/smoke>

This test re-uses the build time test suite present in most CPAN
distributions to check that changes in the package's dependencies haven't
introduced regressions in the package functionality.

The package and its build dependencies are installed, the C<t/>
directory and C<test.pl> files are copied into a temporary directory,
and the build time test suite is run against the installed modules. The
temporary directory is used to make sure that the installed modules get
used instead of the source tree.

If the test suite needs files outside the C<t/> directory, those can
be listed in the C<debian/tests/pkg-perl/smoke-files>. Empty lines
and #-style comment lines are supported. Note that this overrides the
default list of C<t/> and C<test.pl>, so you normally need to include
those too.

Some common test files are removed from the temporary
directory; they currently include C<pod.t>, C<pod-coverage.t>,
C<boilerplate.t>, C<04critic.t>, and C<97_meta.t>. If the file
C<debian/tests/pkg-perl/smoke-skip> exists, it is used as a list of files
to be removed. Empty lines and #-style comments are supported.

The environment variables C<AUTOMATED_TESTING> and
C<NONINTERACTIVE_TESTING> are set to 1 by default. These can
be overridden and others can be added in the configuration file
C<debian/tests/pkg-perl/smoke-env>. Empty
lines and comments are allowed. The file syntax is one line per variable
setting, in the form C<KEY=VAL>; every such line is C<eval>'ed with a shell
so you can compute C<VAL> dynamically, for example:
C<DBDPG_INITDB=$(pg_config --bindir)/initdb>.

The tests are run with the C<prove> C<--merge> option, which merges and
synchronizes stderr and stdout from the test scripts. In some rare cases
this option can break the test suite. It can be disabled by unsetting
C<PKG_PERL_PROVE_ARGS> in C<debian/tests/pkg-perl/smoke-env>.

For the benefit of tests looking for something in C<lib/> or C<blib/>,
dummy module files are installed there and included in a dummy MANIFEST
file. An empty MANIFEST.SKIP is also provided.

If the file C<debian/tests/pkg-perl/smoke-setup> exists and is executable,
it will be executed right before running the tests, with the working
directory set to the package source and C<$TDIR> set to the test running
directory. The testing gets aborted if this program returns a nonzero
value. Please use this facility responsibly and only as a last resort
if the other hooks don't work for you.

If the file C<debian/tests/pkg-perl/smoke-cleanup>
exists and is executable, it will be executed when the smoke tests exit, with 
the working directory set to the package source and C<$TDIR> set to the test 
running directory.

The smoke test by default runs C<prove> with the C<--recurse> option
on the F<t/> directory (prior to 0.37 only the F<t/*.t> tests were run).
In case a package
should run other test scripts, C<debian/tests/pkg-perl/smoke-tests> can be
used to declare which files should be used instead.

Example:
F<t/*/*.t>

=head2 use.t

Source: L<https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/blob/master/autopkgtest/scripts/runtime-deps.d/use.t>

This test ensures that the runtime dependencies are sufficient for
the most basic usage of the module. Warnings from such basic usage are
considered failures because they can be quite annoying on consumers of
the module.

The package and its runtime dependencies are installed, and the command
C<perl -w -MFoo::Bar -e1> is executed, where C<Foo::Bar> is the name of
the main module in the package. If this exits with a nonzero status,
or if there is output on stderr, the test is considered failed.
The test is repeated with PERL_DL_NONLAZY=1 set in the environment;
this makes sure there are no unresolved symbols in binary modules.

The name of the module is read from C<META.json> or C<META.yml>, but it
can be overridden in C<debian/tests/pkg-perl/use-name>, which also allows
listing multiple module names, one per line.
The test is skipped if the name isn't present in one of the above sources.

There are some rare cases where warnings are to be expected even
from the most basic usage.
Such expected warnings can be listed in
C<debian/tests/pkg-perl/use-whitelist>.
Each line in this file is interpreted as a regular expression.

=head2 syntax.t

Source: L<https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/blob/master/autopkgtest/scripts/runtime-deps-and-recommends.d/syntax.t>

This test checks that the runtime dependencies and recommendations
cover all the requirements of the modules shipped in the package,
and that the modules are syntactically valid Perl.

The test first installs the package and its runtime dependencies,
including recommendations.  All the C<.pm> files in the package are then
run through a syntax check with C<perl -wc file.pm>. If the command
exits with a nonzero status, the test is considered failed. Standard
error output is not treated as a failure. If there are other files that
you'd like to check, you can specify those as regular expressions in
C<debian/tests/pkg-perl/syntax-extra>.
Don't include the match delimiters (//). Empty lines and
#-style comments are supported.

If some of the module files have expected problems passing this test, they
can be skipped by listing them in C<debian/tests/pkg-perl/syntax-skip>.
The lines are
matched as fixed substrings (not regular expressions).
Empty lines and #-style comments are supported.

Packages that I<Suggests> other packages are exempt from this test,
unless they include a (possibly empty) C<syntax-skip> file. This is because some
of the module files are expected to need the suggested packages
to work, but there is no mechanism to install them in the 
environment provided by C<autopkgtest>.

To test all module files despite a I<Suggests>, an empty C<syntax-skip>
(with optional comments) can be used since C<pkg-perl-autopkgtest> version
0.45.

=head1 ci.debian.net

Even with the implicit test control file, all the packages would
need source uploads to include the C<Testsuite> header, which would
take quite a while to happen for all the team packages (currently
numbering almost 3000). To speed up adoption, the current plan is
to make L<https://ci.debian.net/> use a whitelist of known good
pkg-perl packages that it will treat as having the 
C<Testsuite: autopkgtest-pkg-perl> header. This whitelist is maintained at
L<https://salsa.debian.org/ci-team/debian-ci-config>
and is based on test runs of all our packages such as 
L<https://lists.debian.org/debian-perl/2014/09/msg00100.html>.

The whitelist is considered a temporary measure that can be dropped once
we have C<Testsuite: autopkgtest-pkg-perl> headers in all our packages.

As of May 2015, there are 2029 whitelisted packages, 378 packages
that declare C<Testsuite: autopkgtest-pkg-perl>, and 604 non-whitelisted
pkg-perl packages that currently don't pass the default checks and
therefore need manual work.

The list of packages needing manual work is available at
L<https://people.debian.org/~ntyni/pkg-perl/autopkgtest-fail.txt>
and current as of 20150524, but is not actively updated at the moment
(volunteers welcome.)

=head1 Team practices

Once the set of known good packages has stabilized and is live on
L<https://ci.debian.net/>, we should do a mass commit in our git
repository adding C<Testsuite: autopkgtest-pkg-perl> headers to the
known good packages.

A team specific lintian
check (activated by C<lintian --profile pkg-perl>) triggers on the
lack of the C<Testsuite> header and points to this document. Coupled
with the mass commit above, affected packages are expected to be those
that need tweaking to get the tests working. Maintainers encountering
the lintian warning are encouraged to fix those (other team members are
happy to help with that!), but they are also naturally free to ignore
the warning and postpone the addition of the header.

When uploading a package with a C<Testsuite> header, uploaders should
adopt a habit of verifying that the tests pass so as not to cause
unnecessary work to the L<https://ci.debian.net/> volunteers.

Bugs related to autopkgtest failures should be filed with

 User: debian-perl@lists.debian.org
 Usertags: autopkgtest

so that they show up on the tracking list at
L<https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=autopkgtest;users=debian-perl@lists.debian.org>

See L<https://wiki.debian.org/bugs.debian.org/usertags> for more information about usertags.

=head1 SETUP

To set up a C<sbuild> chroot for C<autopkgtest> (formerly known as C<adt-run>),
follow the following steps.

=head2 chroot type

You need to setup a chroot type that supports ephemeral sessions for
autopkgtest to run properly. It will fail otherwise.

One option is to use B<lvm-snapshot>, so you need to create a logical
volume and mount it first. For instance, assuming that C</dev/sdb1> is
empty and available to be used as a physical volume:

    sudo apt-get install lvm2
    sudo pvcreate /dev/sdb1
    sudo vgcreate sbuild /dev/sdb1
    sudo lvcreate -L2G -n sid sbuild
    sudo mkfs.ext4 /dev/sbuild/sid
    sudo mkdir -p /srv/sbuild/sid
    sudo mount /dev/sbuild/sid /srv/sbuild/sid

A simpler alternative would be to use B<directory> type, which doesn't
need any previous setup before it is created, just a regular directory
(ephemeral sessions will be setup through union filesystem).

=head2 sbuild

Note that C<sbuild> uses C<schroot> underneath to manage the chroots.

=over

=item * Create a new chroot

Note that installing recommends is needed so all the autopkgtest tools are
available later. Especially a missing B<autodep8> package will lead to grief
and sorrow.

    sudo apt-get install --install-recommends sbuild autopkgtest
    sudo sbuild-createchroot sid /srv/sbuild/sid http://deb.debian.org/debian

At this point you can run C<sudo umount /srv/sbuild/sid> if you used LVM.

=item * Grant permission to use the chroot

    sudo adduser {username} sbuild

=item * Setup directory chroot

First, you should rename the file created automatically on the previous step
(it's not mandatory but useful):

    sudo mv /etc/schroot/chroot.d/sid-amd64-sbuild-* /etc/schroot/chroot.d/sid-amd64-sbuild.conf

Then, you have to setup the chroot properties for the B<directory> type
to make sure ephemeral sessions can be created:

    sudo sed -i -e 's,^union-type=none,union-type=aufs,' /etc/schroot/chroot.d/sid-amd64-sbuild.conf

Or B<lvm-snapshot> type replacing the contents with these:

    [sid-amd64-sbuild]
    type=lvm-snapshot
    lvm-snapshot-options=--size 2G
    device=/dev/sbuild/sid
    groups=root,sbuild
    root-groups=root,sbuild
    profile=sbuild
    description=Debian sid/amd64 autobuilder

=item * Update chroot source

Use this command to B<update> the package list, perform a B<dist upgrade>,
and B<autoremove> obsolete packages in the chroot.

    sudo sbuild-update -udr sid-amd64-sbuild

=item * Modify chroot source

In case you need to make other kind of changes to the chroot.

    sudo schroot -c source:sid-amd64-sbuild

=item * Instantiate chroot

Play with an ephemeral chroot based on the source chroot setup before
(you do not need to be root at this point). Please, be warned that this
chroot instance will be automatically removed when you exit the shell.

    schroot -c sid-amd64-sbuild

=item * Additional mounts

Sometimes you need to mount other filesystem or directories in the chroot
(e.g. C</home>). If you miss anything, you can add that directory in the
file C</etc/schroot/sbuild/fstab>, where B<sbuild> is the profile for the
chroots created by sbuild.

You can find several examples in the B<desktop> and B<default> profiles.

=back

See Enrico Zini's blog entry at
L<https://www.enricozini.org/blog/2008/tips/joys-of-schroot/>
and/or C<sbuild-createchroot(8)> for details.

=head2 eatmydata

You can use C<eatmydata> do improve the speed of your builds inside the
chroot pretty easily.

=over

=item * Make sure C<eatmydata> is installed in the chroot

You can either install it manually in the source chroot or add
C<--include=eatmydata> option before running C<sbuild-createchroot>
above.

=item * Enable C<eatmydata> in the chroot configuration

Add C<command-prefix=eatmydata> to
C</etc/schroot/chroot.d/sid-amd64-sbuild.conf>.

=back

See L<https://wiki.debian.org/sbuild#Using_eatmydata_with_sbuild> for more details.

=head2 Examples

=over

=item An example package using C<debian/tests/pkg-perl/SKIP>: C<libtest-file-sharedir-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libtest-file-sharedir-perl/blob/master/debian/tests/pkg-perl/SKIP>

=item An example package using C<debian/tests/pkg-perl/smoke-skip>: C<libtext-csv-xs-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libtext-csv-xs-perl/blob/master/debian/tests/pkg-perl/smoke-skip>

=item An example package using C<debian/tests/pkg-perl/smoke-files>: C<libnet-ldap-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libnet-ldap-perl/blob/master/debian/tests/pkg-perl/smoke-files>

=item An example package that uses C<debian/tests/pkg-perl/use-name>: C<libnet-radius-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libnet-radius-perl/blob/master/debian/tests/pkg-perl/use-name>

=item An example package using C<debian/tests/pkg-perl/syntax-skip>: C<libplack-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libplack-perl/blob/master/debian/tests/pkg-perl/syntax-skip>

=item An example package using C<debian/tests/pkg-perl/smoke-env>: C<libsvn-notify-mirror-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libsvn-notify-mirror-perl/blob/master/debian/tests/pkg-perl/smoke-env>

=item An example package disabling C<prove --merge> in C<debian/tests/pkg-perl/smoke-env>: C<libterm-progressbar-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libterm-progressbar-perl/blob/master/debian/tests/pkg-perl/smoke-env>

=item An example package using C<debian/tests/pkg-perl/smoke-setup>: C<libnet-sftp-sftpserver-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libnet-sftp-sftpserver-perl/blob/master/debian/tests/pkg-perl/smoke-setup>

=item An example package using C<debian/tests/pkg-perl/smoke-cleanup>: C<libmemcached-libmemcached-perl>

L<https://salsa.debian.org/perl-team/modules/packages/libmemcached-libmemcached-perl/blob/master/debian/tests/pkg-perl/smoke-cleanup>

=back

=head2 Usage

=over

=item Running C<autopkgtest> against a package in the archive, assuming your
schroot name is C<sid-amd64-sbuild>:

  autopkgtest libfoo-bar-perl -- schroot sid-amd64-sbuild # since autopkgtest 4.0
  adt-run libfoo-bar-perl --- schroot sid-amd64-sbuild

(Make sure that I<schroot source:sid-amd64-sbuild> has a C<deb-src> entry
in C<sources.list> and that you have run C<apt-get update>
there at least once before trying this example.)

=item Running C<autopkgtest> on a C<.changes> file:

  autopkgtest libfoo-bar-perl_0.01_amd64.changes -- schroot sid-amd64-sbuild # since autopkgtest 4.0
  adt-run libfoo-bar-perl_0.01_amd64.changes --- schroot sid-amd64-sbuild

=item Running C<autopkgtest> with the control information in the current
directory but the actual package from the archive:

  autopkgtest -B . -- schroot sid-amd64-sbuild # since autopkgtest 4.0
  adt-run -B .// --- schroot sid-amd64-sbuild

(The source tree doesn't need to be built, the binary package is
installed from the archive with apt.)

See C</usr/share/doc/autopkgtest/README.running-tests.rst.gz> for
more information.

=back

=head1 AUTHORS

=over

=item * Niko Tyni

=item * Alex Muntada

=item * gregor herrmann

=item * intrigeri

=back

=head1 LICENSE

Copyright (c) 2014-2022 by the individual authors and contributors noted
above. All rights reserved. This document is free software; you may
redistribute it and/or modify it under the same terms as Perl itself

Perl is distributed under your choice of the GNU General Public
License or the Artistic License. On Debian GNU/Linux systems,
the complete text of the GNU General Public License can be found
in `/usr/share/common-licenses/GPL' and the Artistic License in
`/usr/share/common-licenses/Artistic'.