60216 Commits

Author SHA1 Message Date
Andreas Karlsson
23786e9922 Move all pg_tde related tests into contrib/pg_tde
While these tests test our changes to pg_waldump they are quite easy to
overlook right now and where exactly should we draw the line? These
tests are not something we ever want to upstream and in the future when
we figure out how we want to make sure pg_waldump works with encrypted
WAL we likely will want to have the tests for that solution in the same
folder as our other tests anyway.
2025-04-28 10:01:37 +02:00
Andreas Karlsson
bb000ef9e8 Simplify superuser check for pg_tde_set_*_key_using_*_key_provider()
Make the intent a lot cleaner by doing the check when a global provider
us used instead of deciding to do it or not per user facing function.
2025-04-28 09:48:41 +02:00
Andreas Karlsson
7ed3115933 Get rid of global_status enum
This enum was only used in one place and oscured the two dimensions of
provider types (database vs global) and principal keys (server vs
default vs database).
2025-04-28 09:48:41 +02:00
Andreas Karlsson
7177c2df6e Simplify logic for global, local, server, default
Improve readability for the provider vs key type logic.
2025-04-28 09:48:41 +02:00
Andreas Karlsson
dffece72a5 Make ALTER TABLE ... SET ACCESS METHOD logic easier to read
Also add a couple of tests for the DEFAULT case to avoid regressions.
2025-04-28 09:44:07 +02:00
Andreas Karlsson
ae04e97e4b Clean up test SQL file
This removes some unnecessary queries and formats the queries to be
easier to read.
2025-04-28 09:44:07 +02:00
Andreas Karlsson
6a1e1b6495 Remove support for running pg_tde tests in global TDE mode
Now that we no longer run the pg_tde suite in the global TDE mode we can
remove all the code which was there to support it.
2025-04-28 09:42:37 +02:00
Andreas Karlsson
e4c1cc012b Do not run pg_tde tests in make when in global TDE mode
The purpose of the global TDE mode is to run PostgreSQL's normal test
suite but with our extension so running the pg_tde test suite when in
that mode makes no sense.

Meson supports disabling test suites with --no-suite so we only need to
do this for the Makefile.
2025-04-28 09:42:37 +02:00
Andreas Karlsson
b7dfa6897f Only enable TDE_MODE for the tests with global TDE
The TDE_MODE environment variable disables tests we actually want to run
in our Github Actions. This change is also necessary to in a future
commit disable the pg_tde tests in the global TDE mode.
2025-04-28 09:42:37 +02:00
Andreas Karlsson
e979ab71c2 Harmonize the different Github Actions entry points
The meson and the make scripts had diverged a lot, so this commit fixes
that plus some other inconsistencies.
2025-04-28 09:42:37 +02:00
Andreas Karlsson
712dcf5976 Clean up some switch statements and if chains
In tde_keyring.c we do a lot of switching on the keyring type, some of
it which could be done in a slightly nicer way.
2025-04-28 09:42:13 +02:00
Andreas Karlsson
6fef8bdf70 Rename event triggers to something more expressive
Instead of giving them numbers we call them pg_tde_ddl_start and
pg_tde_ddl_end. Since the triggers are not on the same event the names
do not matter for the order they are executed in.
2025-04-26 16:18:00 +02:00
Andreas Karlsson
5a1fe7650d Add more test result and logs to the Github failure artifact
Most of the result files and the logs were missing from the artifact
created when running make check-world.
2025-04-26 14:58:17 +02:00
Andreas Karlsson
5f64c75c5e Make failure artifact names consistent in Github Actions
One of them even always had meson in the name even when make was ran.
2025-04-26 14:58:17 +02:00
Andreas Karlsson
e355f3827d Clean up .gitignore 2025-04-25 22:51:43 +02:00
Andreas Karlsson
435ef90a35 Move pg_tde_change_key_provider .gitignore entry
The .gitignore entry was left in the old location when the source for
the executable was moved.
2025-04-25 22:51:43 +02:00
Andreas Karlsson
f1d12f17b1 Stop ignoring nodtags.h for pgindent
Since upstream does not ignore it neither should we.
2025-04-25 22:51:43 +02:00
Andreas Karlsson
d51c498991 Remove leftovers from when we remove tde_heap_basic
These pgindent excludes are no longer relevant.
2025-04-25 22:51:43 +02:00
Andreas Karlsson
d602c2ec67 Remove unused return value from check_percona_api_version()
There is no user of this return value and furthermore the function can
only ever return true.
2025-04-25 22:51:21 +02:00
Artem Gavrilov
b7d52ab9a1 Highlight that code coverage badge in main readme file related to pg_tde 2025-04-25 16:46:11 +02:00
Artem Gavrilov
082cc11c8c Run simple tests for code coverage needs 2025-04-25 16:46:11 +02:00
Artem Gavrilov
198af9e357 Extract code coverage jobs into separate CI workflow 2025-04-25 16:46:11 +02:00
Artem Gavrilov
c1da756335 Specifiy extensions required for tde test in EXTRA_INSTALL
To run pg_tde tests with `make check` we have to add pg_buffercache and
test_decoding extensions to temporary pg installation.
2025-04-25 16:46:11 +02:00
Anders Åstrand
aeaae4e282 Remove double declarations
For some reason these functions were declared twice, once using the
macro and once without it.
2025-04-25 16:07:52 +02:00
Andreas Karlsson
b639210ef4 Remove duplicate pg_regress alternate expected file
We apparently had the same alternate file twice in our repo.
2025-04-25 15:50:37 +02:00
Andreas Karlsson
e7fa26f8d3 Consistently use HeapTupleIsValid()
HeapTupleIsValid() is actually just a null check but PostgreSQL's
codebase almost always uses this macro and we had a confusion where we
both had a null check and called this macro so we at least should pick
just one of the two ways to write it. And here I picked the most
commonly used way in the PostgreSQL codebase.
2025-04-25 15:50:10 +02:00
Andreas Karlsson
f758cc70bb Make local variable no longer static
Presumably this variable used to actually be used as a static variable
at some point in time but that is no longer the case.
2025-04-25 15:49:57 +02:00
Andreas Karlsson
3251aef9e7 Rename the files under $PGDATA/pg_tde
The name pg_tde_<OID>_keyring was confusing to users due to making it
sound like it would contain keys. And the name pg_tde_<OID>_map did not
tell a user anything. The new names are <OID>_providers for the key
providers and <OID>_keys for the relation/WAL keys.

While changing the suffixes to be more descriptive I also dropped the
pg_tde_ prefix since it is just noise when they all are in the pg_tde
directory.
2025-04-25 14:11:05 +02:00
Andrew Pogrebnoy
16ba8eeeeb Fix release of the not-held lock
Fixes PG-1573
2025-04-25 15:00:10 +03:00
Anders Åstrand
6b508a0f74 Make 013_crash_recovery.pl work on repeated runs
If the keyring from a previous run is still present, the test fails.
2025-04-25 13:45:53 +02:00
Andreas Karlsson
57ac8c8918 Fix typo in out param of pg_tde_*_key_info() 2025-04-25 10:35:46 +02:00
Andrew Pogrebnoy
c8419b81ac Add crash recovery tests
Tests to trigger redo routines after the server crash. It mostly checks
invariants when different redo functions might rewrite WAL keys created on the
init stage.

For PG-1539, PG-1541, PG-1468, PG-1413
2025-04-24 15:14:27 +03:00
Andrew Pogrebnoy
1fa786fa29 Don't rewrite _map files on the save_principal_key redo
We create a new WAL key during the extension init, which happens before
the redo. This means that in case of a crash,
pg_tde_save_principal_key_redo was rewriting a WAL _map file and destroying
a newly created key.

Since we emit an XLog record after the key was successfully written to
the file (the file was created), we can safely assume that we should
not change the file if it exists.
2025-04-24 15:14:27 +03:00
Anders Åstrand
e735727c8e Do not encourage root token use for vault
We should probably not encourage anyone to use a root token for vault
keyring providers. We use "secret_token" elsewhere, so use that
consistently.
2025-04-24 11:40:53 +02:00
Artem Gavrilov
b7d2f1599f Fix typo 2025-04-23 18:06:27 +02:00
Andreas Karlsson
e450170e03 PG-1441 Do not replicate relation keys
Instead of replicating relation keys we generate new ones on replay of
the XLOG_TDE_ADD_RELATION_KEY record at the replica server. This means a
replica and its master server will end up with different sets of
relation keys making a simple binary diff impossible but that is a
dubious advantage since the WAL keys will differ anyway and on on the
flip-side the new code is simpler and easier to reason about. Especially
since now WAL keys and relation keys are treated in a bit more similar
ways.

To prevent duplicate keys in the key file we skip generating and adding
a key if there already is an entry in the file for the same relation.
2025-04-23 14:33:45 +02:00
Andreas Karlsson
8fe368b6f9 PG-1441 Do not generate relation keys in the SMGR on redo
Make sure we can never generate relation keys on a streaming replica or
in recovery in the SMGR code. Instead the key should always have been
already generated when replaying the XLOG_TDE_ADD_RELATION_KEY record.
2025-04-23 14:33:45 +02:00
Andreas Karlsson
4d9756dd17 PG-1441 Write simple TAP test for replication of pg_tde
This way we can avoid obvious regression when refactoring the code for
replicating keys in future commits. This test can in the future be
expanded to test more interesting cases.
2025-04-23 14:33:45 +02:00
Andreas Karlsson
53f71222c9 PG-1441 Refactor code for generating keys in SMGR code
The old code was harder to read than necessary since it had exactly two
callers of which one had each value of the boolean flag. Breaking it up
into two functions makes the intent clearer. While at it we also clean
up the flow a bit more.
2025-04-23 14:33:45 +02:00
Andrew Pogrebnoy
4724ecbfc6 Redo of the key provider info now needs a lock
Although it may be technically omitted during redo, write_key_provider_info() now checks if there is a lock held
2025-04-23 14:51:18 +03:00
Anders Åstrand
157230de39 PG-1419 Validate key provider access
This adds some validation to make sure we can access the key provider
when it's created to make the user experience a little nicer. The actual
access validation is very rudimentary for now but can easily be
expanded.
2025-04-23 10:31:30 +02:00
Anders Åstrand
1b8513c21e Refactor some lower level functions in tde_keyring
Previously write_key_provider_info() was a bit of a "do everything"
function that had very different behavior depending on what parameters
was passed to it. This commit reworks it to a "dumb" function that just
writes the data without asking questions and have the callers take
responsibility for data validity.

This is to make it easier to validate the data in different ways
depending on the caller's needs without further complicating
write_key_provider_info().
2025-04-23 10:31:30 +02:00
Anders Åstrand
65fa2e241b Rename KeyringProviderXLRecord
The new name, KeyringProviderRecordInFile, describes what it is rather
than what it's used for. But the real reason is that I want to use it
for other things than the WAL in future commits.
2025-04-23 10:31:30 +02:00
Anders Åstrand
d6eb31f59a Fix error message in keyring_file.c
The colon was on the wrong side of the space.
2025-04-23 10:31:30 +02:00
Anders Åstrand
ce63840ca9 Remove non-test
This regression file didn't really test anything. The tests it was
supposed to do was removed here e270322f72
So now it doesn't test anything that key_provider.sql doesn't already do
for us.
2025-04-23 10:31:30 +02:00
Anders Åstrand
043232e31e Change some keyring function to return void
No callers cared about the return value of these functions anyway.
2025-04-23 10:31:30 +02:00
Andreas Karlsson
14a2bd2849 Clean up a couple comments in pg_tde_tdemap.c 2025-04-23 09:23:20 +02:00
Andreas Karlsson
1885236fa9 Simplify error handling in pg_tde_write_map_keydata_file() 2025-04-23 09:23:20 +02:00
Andreas Karlsson
15ea8358f3 Do not switch between using prev_pos and curr_pos
Be consistent about always passing curr_pos when reading or writing the
map file. The code is easier to understand if only one variable is used
for positioning in the file.
2025-04-23 09:23:20 +02:00
Andreas Karlsson
0b2dbd2ccd Consistently update the position in a pointer passed to the function
This UX is not nice but let's at least be consistent about it.
2025-04-23 09:23:20 +02:00