90 Commits

Author SHA1 Message Date
Tobias Brunner
8f6e3c164a testing: Include the kernel build number 2025-05-28 16:35:26 +02:00
Tobias Brunner
353d5c130b testing: Use tar instead of recursive scp
It seems that scp is sometimes very slow (unclear what causes it as it's
not always the same).  Packing up the files with tar performs a lot
better in these situations.  And copying the files to multiple hosts
in parallel additionally helps to reduce the time required for these
steps.

Using --overwrite and -h preserves existing symlinks (e.g. for the users
file in /etc/freeradius/3.0) and overwrites the target file instead.
The -m option ignores timestamps when extracting the files as some target
files will be newer than the source.  Using -h when packing up files in
load-testconfig allows using symlinks in the test config dirs to files
on the host running the tests.
2025-04-07 14:54:48 +02:00
Tobias Brunner
02c43fa6e4 testing: Move removal of charon.pid into posttest section
Fixes: a103f3a2849f ("testing: Add options to only run pre- or posttest scripts of a scenario")
2025-04-04 12:06:59 +02:00
Tobias Brunner
08428f6b5d testing: Fix loading test config for tests that were never run
The file won't exist in the previous location until load-testconfig was
executed once.  Since it's not modified by the script it's fine to
load it directly from the original location.

Fixes: a103f3a2849f ("testing: Add options to only run pre- or posttest scripts of a scenario")
2025-04-04 12:06:59 +02:00
Tobias Brunner
a103f3a284 testing: Add options to only run pre- or posttest scripts of a scenario
This allows to manually do some testing without having to type commands
to set up a scenario.

Also changes how arguments are parsed (allowing to pass options mixed
with test dirs) and adds some usage output.
2025-03-14 11:49:42 +01:00
Tobias Brunner
1c053bc3f0 testing: Log timestamps relative to the start of the test
Makes it easier to compare multiple runs against each other.
2024-12-03 08:43:45 +01:00
Tobias Brunner
9e88c3f32e testing: Collect test results with an on-guest script in parallel
In particular the swanctl calls all take a while and this allows doing
them in parallel if multiple hosts are involved. This reduces the runtime
of each test by 1-3 seconds.
2024-12-03 08:43:45 +01:00
Tobias Brunner
b3a72c7994 testing: Use an on-guest script to cleanup/initialize and run them in parallel
This is a bit quicker than doing this with separate SSH calls for each
host sequentially (up to half a second per test).
2024-12-03 08:43:45 +01:00
Tobias Brunner
c8cfeeff54 testing: Always use immediate mode for tcpdump
We don't use versions that don't support this anymore and sometimes the
detection didn't work properly and a run without it would get started.
2024-12-03 08:43:45 +01:00
Tobias Brunner
4ba857930c testing: Format total time in a more readable way 2023-07-26 13:06:40 +02:00
Andreas Steffen
67f7d8fe8a testing: Replace deprecated tempfile command by mktemp 2022-07-23 11:28:08 +02:00
Tobias Brunner
63f35993d9 testing: Use sans-serif font for test results
We could later extend the stylesheet more or even add something similar
to actual templates (e.g. a shared header/footer).
2022-05-24 08:11:56 +02:00
Tobias Brunner
994d9d37d4 testing: Copy comments to test log 2022-04-14 18:42:01 +02:00
Tobias Brunner
b47e9919f5 testing: Print the actual start time of a command for do-tests -t
Because the command line, together with the results, is printed after
executing it, there could have been weird delays between commands.
2022-02-09 18:58:08 +01:00
Tobias Brunner
5b9c46b988 testing: Remove support for Debian jessie and FreeRADIUS 2 2021-10-01 15:05:44 +02:00
Tobias Brunner
f2d240954a testing: Skip tests with missing files, don't abort the test run
This allows simple test configs in testing/tests/local that are no
actual test cases.
2020-06-23 16:24:18 +02:00
Tobias Brunner
c7a74fd3e5 testing: Allow enabling only timestamps without verbose command output
-t enables only the timestamps, -v additionally logs command output
(includes -t).
2018-11-21 14:32:25 +01:00
Tobias Brunner
2132031d0e testing: Show config files of FreeRADIUS 3.0 in test results 2018-11-21 14:32:25 +01:00
Tobias Brunner
2fbe44bef3 testing: Remove TNC@FHH dependencies and scenarios that rely on them
While we could continue to use FreeRADIUS 2.x that branch is officially EOL.
So instead of investing time and effort in updating/migrating the patches to
FreeRADIUS 3.x (the module changed quite significantly as it relies solely on
the naeap library in that release), for a protocol that is superseded anyway,
we just remove these scenarios and the dependencies.  Actually, the
complete rlm_eap_tnc module will be removed with FreeRADIUS 4.0.
2018-11-21 14:32:24 +01:00
Tobias Brunner
5cfd7311d0 testing: Print command output if test fails
This is quite helpful to debug why a pattern didn't match.

As it could produce quite a lot of output if something is not found in a
log file, the complete output is only printed in verbose mode, otherwise,
`head` is used to print the first 10 lines of output.

We only get stdout from SSH, so the stderr redirection is only really
for errors ssh itself produces.
2018-06-14 09:29:26 +02:00
Tobias Brunner
4492c9c670 testing: Ignore IP-in-IP SAs created with IPComp SAs that remain in the kernel
The kernel creates such SAs to handle uncompressed small packets.  They
are implicitly created and deleted with IPComp SAs.  The problem is that
when we delete an IPComp SA only that state is deleted and removed from
the SA lists immediately, the IP-in-IP state is not removed until the IPComp
state is eventually destroyed.  This could take a while if there are still
references to it around.  So the IP-in-IP states will keep getting reported
by ip xfrm state until that happens (we also can't flush or explicitly delete
such kernel-created states).

In kernels before 4.14 this wasn't really a problem but since
ec30d78c14a8 ("xfrm: add xdst pcpu cache") the kernel seems to keep the
references to the last used SAs around a lot longer.

Also, usually a test scenario following an IPComp scenario will create
and use new SAs and thus the cached SAs will disappear before the kernel
state is checked again.  However, if a following scenario uses different
hosts the states might remain, which caused some unrelated scenarios to
fail before adding this fix.
2018-02-01 17:10:19 +01:00
Andreas Steffen
67a97c18ae testing: Enable systemd 2017-11-10 11:49:38 +01:00
Tobias Brunner
c11d13c4b9 testing: Add -v option to do-tests to prefix commands with timestamps 2017-08-07 16:55:45 +02:00
Tobias Brunner
f0d051f192 testing: Also capture stderr during test cases
The output was not correct otherwise due to the reordering of commands.
2017-08-07 10:44:05 +02:00
Tobias Brunner
87c6247e0d testing: Clearly mark the tests that failed 2017-08-07 10:44:05 +02:00
Tobias Brunner
99cf64e960 testing: Add support for counting matching lines in tests
Specifying an integer instead of YES in evaltest.dat causes the number to get
compared against the actual number of lines matching the pattern.

This may be used to count matching packets or log lines.
2017-08-07 10:44:05 +02:00
Tobias Brunner
49917f0028 testing: Support running multiple tests with * as wildcard (e.g. ikev2/ocsp-*) 2017-07-07 09:23:14 +02:00
Tobias Brunner
3fb68ac211 testing: load-testconfig script loads config from source dir
It now does replace the IPs too. This way it's easier to play around
with a config (otherwise a do-tests run was required to build the
config files in the build dir).
2017-03-02 11:54:39 +01:00
Tobias Brunner
6307a18fe1 testing: Fix totals if post test checks fail 2016-09-20 15:36:14 +02:00
Tobias Brunner
d8b2980aa5 testing: Log leaks and fail tests if any are detected 2016-09-20 15:36:14 +02:00
Tobias Brunner
ac67aeb100 testing: Add output of iptables-save
This might be helpful to get the complete picture of the installed
rules.  `-c` is currently not used as the counters that are added in
front of every rule make the output quite hard to read and the counters
are already provided in the accompanying `iptables -v -L` output.

Fixes #2111.
2016-09-12 16:15:45 +02:00
Tobias Brunner
fa36699bfa testing: List nat and mangle tables in addition to the filter table
This is useful in scenarios that e.g. use NAT and/or marks.

References #2111.
2016-09-12 16:15:14 +02:00
Tobias Brunner
92ccc0b412 testing: Ignore comments (lines starting with #) in pre-/eval-/posttest.dat 2016-09-09 12:19:14 +02:00
Tobias Brunner
379d94eb70 testing: Try to properly abort a test run after CTRL-C
The run is aborted after the current scenario.  Depending on which
command was interrupted it might be necessary to press CTRL-C multiple
times (e.g. if a later command depends on the interrupted one).

This should fix HTML files and get us some proper console output after
the run.
2016-08-30 16:16:42 +02:00
Tobias Brunner
bdd7c42fc0 testing: Report number of tests per subdirectory in main index 2016-08-30 15:03:40 +02:00
Tobias Brunner
dda06b0439 testing: Mount and serve testresults from the host
This avoids having to copy testresults, makes results of cancelled runs
browsable (runs may actually be followed live) and preserves old results
when rebuilding guest images (e.g. when using the build-strongswan script).
The number of consecutive test runs without any intermittent rebuild of the
guest images is also not limited by the image size anymore.
2016-08-29 18:11:42 +02:00
Tobias Brunner
cf6042453f testing: Serve images in testresults via mod_rewrite and not a symlink 2016-08-29 18:11:42 +02:00
Tobias Brunner
63acd803f0 testing: Wait for packets to be processed by tcpdump
Sometimes tcpdump fails to process all packets during the short running
time of a scenario:

0 packets captured
18 packets received by filter
0 packets dropped by kernel

So 18 packets were captured by libpcap but tcpdump did not yet process
and print them.

This tries to use --immediate-mode if supported by tcpdump (the one
currently in jessie or wheezy does not, but the one in jessie-backports
does), which disables the buffering in libpcap.

However, even with immediate mode there are cases where it takes a while
longer for all packets to get processed.  And without it we also need a
workaround (even though the version in wheezy actually works fine).
That's why there now is a loop checking for differences in captured vs.
received packets.  There are actually cases where these numbers are not
equal but we still captured all packets we're interested in, so we abort
after 1s of retrying.  But sometimes it could still happen that packets
we expected got lost somewhere ("packets dropped by kernel" is not
always 0 either).
2016-06-16 14:36:15 +02:00
Tobias Brunner
71424a2f85 testing: Make sure tcpdump is actually terminated before analyzing/collecting logs 2016-06-16 14:01:47 +02:00
Andreas Steffen
9db530493f testing: Change sql scenarios to swanctl 2016-01-03 06:28:48 +01:00
Andreas Steffen
6789d79d46 testing: Added swanctl --list-algs output 2015-12-11 18:26:54 +01:00
Tobias Brunner
74270c8c86 vici: Don't report memory usage via leak-detective
This slowed down the `swanctl --stats` calls in the test scenarios
significantly, with not much added value.
2015-12-11 18:26:53 +01:00
Tobias Brunner
e873cb5a28 testing: Add test config to create and remove a directory for DBs stored in ramfs 2015-11-09 15:18:38 +01:00
Tobias Brunner
bcad0f761f testing: Report the actual strongSwan and kernel versions 2015-11-09 15:18:37 +01:00
Tobias Brunner
d4908c06c1 testing: Report time required for all scenarios on test overview page 2015-11-09 15:18:37 +01:00
Tobias Brunner
e22a663129 testing: Don't log anything to the console if auth.log or daemon.log do not exist 2015-11-09 15:18:36 +01:00
Tobias Brunner
12f08e07e1 testing: Simplify fetching of swanctl --list-* output 2015-11-09 15:18:36 +01:00
Tobias Brunner
9086f060d3 testing: Let test scenarios fail if IPsec SAs or policies are not removed
The IKE daemon should delete all installed SAs and policies when
everything works properly, so we fail the test if that's not the case.
2015-08-21 18:27:06 +02:00
Tobias Brunner
c91682d1b8 testing: Flush state and policies before every scenario
Similar to conntrack we make sure we are working on a clean slate.
2015-08-21 18:27:06 +02:00
Tobias Brunner
008a9ad12c testing: Don't run do-tests when hosts are not running
running_any is satisfied if at least one host is running.  We could
easily add a running_all() helper to check if all hosts are running if
it turns out that's not strong enough.
2015-08-03 13:34:05 +02:00