Sometimes when a test times out the workspace is still busy when we try
to unmount it, which leaves the runner unusable until it's cleaned up
manually. We try to unmount this in the first step, but that usually
doesn't work since it fails during setup before it starts our workflow.
Move it to the end and make it a lazy unmount so it hopefully works
eventually.
Where our test target is a bigendian system, do an additional build on
the runner host (which is little endian) and test interop between the two.
Should hopefully catch obvious endianness bugs.
It got broken by the sshd-auth change, it's not obvious why, and the
platform lacks the debugging tools (eg gdb, strace) to figure it out.
The upstream project seems effectively dead (6 years since the last
commit, 10 since the last release). It was useful while it lasted
(we found a real bug because of it) but its time seems to have passed.
Instead of trying to infer the type of the self hosted tests in each of
the driver scripts (inconsistently...), set one of the following
variables to "true" in the workflow:
VM: tests run in a virtual machine.
EPHEMERAL: tests run on an ephemeral virtual machine.
PERSISTENT: tests run on a persistent virtual machine
REMOTE: tests run on a physical remote host.
EPHEMERAL VMs can have multiple instances of any given VM can exist
simultaneously and are run by a runner pool. The other types have a
dedicated runner instance and can only run a single test at a time.
Other settings:
SSHFS: We need to sshfs mount over the repo so the workflow can collect
build artifacts. This also implies the tests must be run over ssh.
DEBUG_ACTIONS: enable "set -x" in scripts for debugging.
Some of the selfhosted test targets take a long time to run for various
reasons, so label them for "libvirt-hipri" runners so that they can
start immediately. This should reduce the time to complete all tests.
Previously there was one runner per test target (mostly VMs). This had
a few limitations:
- multiple tests that ran on the same target (eg multiple build
configs) were serialized on availability or that runner.
- it needed manual balancing of VMs over host machines.
To address this, make VMs that use ephemeral disks (ie most of them)
all use a pool of runners with the "libvirt" label. This requires that
we distinguish between "host" and "target" for those. Native runners
and VMs with persistent disks (eg the constantly-updated snapshot ones)
specify the same host and target.
This should improve test throughput.
The default job timeout of 360 (6h) is not enough to complete the
regress tests for some of the slow VMs depending on the load on the host.
Increase to 600 (10h).
If a previous run on a physical runner has failed to clean up, the next
run will fail because it'll try to check out the code to a broken
directory mount. Make cleanup the first step.
Darren Tucker