From 749ab3f58da16f74982d3e5eca0cf7db78f7b26b Mon Sep 17 00:00:00 2001 From: Sandro Santilli Date: Fri, 28 Jan 2022 17:33:53 +0100 Subject: [PATCH] Improve documentation about runnng tests - Add hints about hunting for names to pass to ctest -R - Add a test environment preparation section in test/README.md - Add section about using docker for PostgreSQL test env setup - Tweak instruction about running python tests manually - avoid $PWD assumption --- tests/README.md | 153 +++++++++++++++++++++++++++++------------------- 1 file changed, 94 insertions(+), 59 deletions(-) diff --git a/tests/README.md b/tests/README.md index b37e60e2f0f..3c65e564ea3 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,52 +1,15 @@ QGIS unit tests =============== -# Build tests +# Building tests Make sure that you have enabled building of tests in CMake. `cmake -DENABLE_TESTS=ON ..` -# Run tests +# Setting up the test environment -You can run all tests using `make check`. -Note you will need `xvfb-run` for that (sudo apt-get install xvfb). - -Individual tests can be run using `ctest`. - -For example if the output of `make check` ends like this: - -``` - The following tests FAILED: - 77 - PyQgsLocalServer (Failed) -``` - -You could re-run the failing test with: - -``` - ctest -V -R PyQgsLocalServer -``` - -The parameter `-V` enables verbose mode and `-R` takes a regular expression as -parameter and will only run matching tests. - -For python tests, you can run a specific test inside a unit file -with something like this: - -``` - QGIS_PREFIX_PATH=output PYTHONPATH=output/python:$PYTHONPATH \ - python ${srcdir}/tests/src/python/test_qgsvectorfilewriter.py - TestQgsVectorLayer.testOverwriteLayer -``` - -If you get `Could not connect to any X display` errors it means that your build -machine does not have an X server. In that case you need to run the test under -`xvfb-run`. For example: - -``` - xvfb-run --server-args=-screen\ 0\ 1024x768x24 ctest -V -R PyQgsServerWMSGetMap -``` - -# Advanced configuration +Some tests require setting up a test environment. +Below some information about how to set such environment up. ## Postgres @@ -86,27 +49,78 @@ service (or your `QGIS_PGTEST_DB` connection string) uses a connection method which allows passing username/password pair to connect (and which allows creating users) -# Write tests +Some tests require a specific PostgreSQL server configuration +(installation of auth certificates) so the most convenient way +to bring up such server would be to (tweak $srcdir appropriately): -Instructions about writing tests for the processing framework -can be found in a [separate README file](../python/plugins/processing/tests/README.md): - - ${TOP_SRCDIR}/python/plugins/processing/tests/README.md - -Information about labeling tests design and organization: - - ${TOP_SRCDIR}/tests/testdata/labeling/README.rst - -WCS testing information can be found in: - - ${TOP_SRCDIR}/tests/testdata/raster/README.WCS - -About benchmark tests you can read: - - ${TOP_SRCDIR}/tests/bench/README + QGIS_WORKSPACE=${srcdir} \ + docker-compose -f .docker/docker-compose-testing-postgres.yml up -d postgres + export PGHOST=`docker inspect docker_postgres_1 | jq -r .[0].NetworkSettings.Networks.docker_default.IPAddress` + export PGUSER=docker + export PGPASSWORD=docker + tests/testdata/provider/testdata_pg.sh + echo "${PGHOST} postgres # for qgis_test, docker" | sudo tee -a /etc/hosts -# Run python tests in GDB +# Running the tests + +Once the test environment is setup, you can run all tests using `make check`. +Note you will need `xvfb-run` for that (sudo apt-get install xvfb). + +Individual tests can be run using `ctest`. + +For example if the output of `make check` ends like this: + +``` + The following tests FAILED: + 77 - PyQgsLocalServer (Failed) +``` + +You could re-run the failing test with: + +``` + ctest -V -R PyQgsLocalServer +``` + +The parameter `-V` enables verbose mode and `-R` takes a regular expression as +parameter and will only run matching tests. + +Note that the name of the test as reported by `make check` does not +always matches the name of the test as known by `cmake`. +When this happens, the `ctest -R ` command will report +`No tests were found!!!` and you'll need to go hunting for the actual name. +A useful hunting tool is `git grep`, which could be used like this: + +``` + $ ctest -V -R testSslRequireNoCaCheck + No tests were found !!! + $ git grep testSslRequireNoCaCheck # Search for the reported testname + tests/src/python/test_authmanager_password_postgres.py: def testSslRequireNoCaCheck(self): + $ git grep test_authmanager_password_postgres # Search for the file containing the reported testname + tests/src/python/CMakeLists.txt: ADD_PYTHON_TEST(PyQgsAuthManagerPasswordPostgresTest test_authmanager_password_postgres.py) + $ ctest -V -R PyQgsAuthManagerPasswordPostgresTest # use the CMakeLists.txt name +``` + +If you get `Could not connect to any X display` errors it means that your build +machine does not have an X server. In that case you need to run the test under +`xvfb-run`. For example: + +``` + xvfb-run --server-args=-screen\ 0\ 1024x768x24 ctest -V -R PyQgsServerWMSGetMap +``` + +## Manually running python tests + +For python tests, you can run a specific test inside a unit file +with something like this (tweak $builddir and $srcdir as appropriate) + +``` + QGIS_PREFIX_PATH=${builddir}/output PYTHONPATH=${builddir}/output/python:$PYTHONPATH \ + python ${srcdir}/tests/src/python/test_qgsvectorfilewriter.py + TestQgsVectorLayer.testOverwriteLayer +``` + +#### Running python tests in GDB First find out the required environment variables by running the test outside the debugger. @@ -131,3 +145,24 @@ Which you can run in gdb with: Now you can start using the usual gdb (`bt` etc.) interface or - if you have installed the [appropriate debug tools (adjust for python3!)](https://wiki.python.org/moin/DebuggingWithGdb) even allows doing python introspection (`py-bt`). + +# Writing tests + +Instructions about writing tests for the processing framework +can be found in a [separate README file](../python/plugins/processing/tests/README.md): + + ${TOP_SRCDIR}/python/plugins/processing/tests/README.md + +Information about labeling tests design and organization: + + ${TOP_SRCDIR}/tests/testdata/labeling/README.rst + +WCS testing information can be found in: + + ${TOP_SRCDIR}/tests/testdata/raster/README.WCS + +About benchmark tests you can read: + + ${TOP_SRCDIR}/tests/bench/README + +