1983 Commits

Author SHA1 Message Date
Yann Collet
4c097b4939 refactor fullbench
to make it easier to add new scenarios
2024-12-20 10:36:58 -08:00
Yann Collet
09964c6276 prepare fullbench for a refactor 2024-12-20 10:36:58 -08:00
Yann Collet
0b013b2688 added unit tests to ZSTD_compressSequencesAndLiterals()
seems to work as expected,
correctly control that `litSize` and `srcSize` are exactly correct.
2024-12-20 10:36:58 -08:00
Yann Collet
c97522f7fb codemod: ZSTD_sequenceFormat_e -> ZSTD_SequenceFormat_e
since it's a type name.

Note: in contrast with previous names, this one is on the Public API side.
So there is a #define, so that existing programs using ZSTD_sequenceFormat_e still work.
2024-12-20 10:36:56 -08:00
Yann Collet
477a01067f codemod: symbolEncodingType_e -> SymbolEncodingType_e 2024-12-20 10:36:56 -08:00
Yann Collet
a2245721ca codemod: seqStore_t -> SeqStore_t
same idea, SeqStore_t is a type name, it should start with a Capital letter.
2024-12-20 10:36:55 -08:00
Yann Collet
9671813375 codemod: seqDef -> SeqDef
SeqDef is a type name, so it should start with a Capital letter.
It's an internal symbol, no impact on public API.
2024-12-20 10:36:55 -08:00
Diego Nieto Cid
4a4786bef0 tests: fix non-regular file test on GNU/Hurd
Since commit b21b03ca6 [1] the behaviour of writes to /dev/zero has been
fixed and now the non-regular file removal test no longer needs to be
done on /dev/random (which no longer works as random is not writable now).

[1] https://git.savannah.gnu.org/cgit/hurd/hurd.git/commit/?id=b21b03ca624b89caeedfe58430cea4b40317d39f
2024-12-10 10:30:52 -03:00
Yann Collet
bbaba45589 change experimental parameter name
from ZSTD_c_useBlockSplitter to ZSTD_c_splitAfterSequences.
2024-10-31 13:43:40 -07:00
Yann Collet
4f93206d62 changed variable name to ZSTD_c_blockSplitterLevel
suggested by @terrelln
2024-10-29 11:12:09 -07:00
Yann Collet
f593ccda04 removed trace left over 2024-10-28 16:57:01 -07:00
Yann Collet
37706a677c added a test
test both that the new parameter works as intended,
and that the over-split protection works as intended
2024-10-28 16:31:15 -07:00
Yann Collet
4e1a87976f update regression results 2024-10-25 16:33:00 -07:00
Yann Collet
c63b5d2294 update regression results 2024-10-24 14:47:55 -07:00
Yann Collet
ea082692aa added block splitter variant for greedy & lazy (levels 5 to 7) 2024-10-24 13:27:01 -07:00
Yann Collet
70c77d20d6 update regression results
first block is no longer splitted since adding the @savings over-split protection
2024-10-24 11:43:16 -07:00
Yann Collet
bbda1acf85 update regression results 2024-10-23 15:56:56 -07:00
Yann Collet
d2eeed53dc updated compression results
due to integration of `sample5` strategy, leading to better compression ratios on a range of levels
2024-10-23 11:50:57 -07:00
Yann Collet
dac26eaeac updated regression test results 2024-10-23 11:50:57 -07:00
Yann Collet
433f4598ad fixed minor conversion warnings on Visual 2024-10-23 11:50:56 -07:00
Yann Collet
80a912dec1 fixed zstreamtest 2024-10-23 11:50:56 -07:00
Yann Collet
f83ed087f6 fixed RLE detection test 2024-10-23 11:50:56 -07:00
Yann Collet
41d870fbbf updated regression tests results 2024-10-17 11:06:26 -07:00
Yann Collet
61d08b0e42 fix test
a margin of 4 is insufficient to guarantee compression success.
2024-10-17 09:37:23 -07:00
Yann Collet
730d2dce41 fix test 2024-10-15 18:44:40 -07:00
Yann Collet
de6cc98e07 fix incorrect pointer manipulation
fix #4155
2024-10-01 09:25:26 -07:00
Pawel Czarnecki
1f5df587fa tests/decodecorpus: add more advanced options
- add option to force a specific block type
- add option to force a specific literal tyle
- add option to generate only frame headers
- add option to skip generating magic numbers

Co-authored-by: Maciej Dudek <mdudek@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Co-authored-by: Roman Dobrodii <rdobrodii@antmicro.com>
2024-07-26 09:47:16 +02:00
Yann Collet
3de0541aef
Merge pull request #4079 from elasota/truncated-huff-state-error
Throw error if Huffman weight initial states are truncated
2024-06-30 16:17:03 -07:00
elasota
0938308ff6 Throw error if Huffman weight initial states are truncated 2024-06-20 17:46:16 -04:00
Dimitri Papadopoulos
2d736d9c50
Fix new typos found by codespell 2024-06-20 20:12:16 +02:00
Quentin Boswank
f19c98228f Fix $filter and Msys/Cygwin
- switched the patter and input of $filter into the right places
- added pattern wildcard to MSYS_NT & CYGWIN_NT as they change with windows versions
- correctly identify MSYS2, even in an env like MINGW64
2024-06-05 18:37:27 +02:00
Elliot Gorokhovsky
be6a182006
Unit test for external sequence producer + static CCtx + streaming (#4063) 2024-06-03 12:42:27 -04:00
Yann Collet
ff7a151f2e update test name 2024-04-21 10:44:50 -07:00
Alex Murkoff
01cea2e1e2 fixed ISO C incompatibility
for good this time... probably
2024-04-13 02:37:13 +07:00
Alex Murkoff
1d5e9705db fixed memory leaks and almost all undefined behaviour 2024-04-13 02:24:14 +07:00
Po-Chuan Hsieh
103a85e6f6
Use md5sum rather than gmd5sum for FreeBSD
Reference:	https://man.freebsd.org/cgi/man.cgi?query=md5sum
2024-03-27 20:53:52 +08:00
Nick Terrell
81a5e5d438 [fuzz] Turn off -Werror by default
This was causing OSS-Fuzz errors, due to compiler differences.
* Fix the issue
* Also turn off -Werror so we don't fail fuzzer builds for warnings
* Turn on -Werror in our CI
2024-03-26 16:34:36 -07:00
Elliot Gorokhovsky
dc1f7b560b
fix -Werror=pointer-arith in fuzzers (#3983) 2024-03-21 15:16:38 -04:00
Yann Collet
6679d0ca7b
Merge pull request #3982 from embg/fuzzer_readme
Document the process for adding a new fuzzer
2024-03-21 11:10:03 -07:00
Yann Collet
1d3f664fce
Merge pull request #3979 from yoniko/Werror-fuzz
Fail on errors when building fuzzers
2024-03-21 10:41:34 -07:00
Nick Terrell
731f4b70fc Fix & fuzz ZSTD_generateSequences
This function was seriously flawed:
* It didn't do output bounds checks
* It produced invalid sequences when an uncompressed or RLE block was emitted
* It produced invalid sequences when the block splitter was enabled
* It produced invalid sequences when ZSTD_c_targetCBlockSize was enabled

I've attempted to fix these issues, but this function is just a bad idea,
so I've marked it as deprecated and unsafe. We should replace it with
`ZSTD_extractSequences()` which operates on a compressed frame.
2024-03-21 07:18:05 -07:00
Elliot Gorokhovsky
741b87bbe1
Fuzzing and bugfixes for magicless-format decoding (#3976)
* fuzzing and bugfixes for magicless format

* reset dctx before each decompression

* do not memcmp empty buffers

* nit: decompressor errata
2024-03-20 19:22:34 -04:00
Elliot Gorokhovsky
f62b2663b9 Add docs on how to add a new fuzzer 2024-03-19 14:05:23 -07:00
Yonatan Komornik
3487a60950 Fail on errors when building fuzzers
Fails on errors when building fuzzers with `fuzz.py` (adds `Werror`).
Currently allows `declaration-after-statement`, `c++-compat` and
`deprecated` as they are abundant in code (some fixes to
`declaration-after-statement` are presented in this commit).
2024-03-18 15:51:28 -07:00
Yonatan Komornik
6a0052a409
Fix bugs in simple decompression fuzzer (#3978)
Fixes 2 issue in `simple_decompress.c`:
1. Wrong type used for storing the results of `ZSTD_findDecompressedSize` resulting in never matching to `ZSTD_CONTENTSIZE_ERROR` or `ZSTD_CONTENTSIZE_UNKNOWN`.

2. Experimental API is used (`ZSTD_findDecompressedSize`) without defining `ZSTD_STATIC_LINKING_ONLY`.
2024-03-18 15:36:40 -07:00
Elliot Gorokhovsky
7d970bd83c
Implement one-shot fallback for magicless format (#3971) 2024-03-18 10:55:53 -04:00
W. Felix Handte
2215101cad Add a Few Tests 2024-03-13 16:32:04 -04:00
Yann Collet
4bd911ae41
Merge pull request #3954 from facebook/zeroSeq_sample
Add the zeroSeq sample
2024-03-12 15:31:01 -07:00
Elliot Gorokhovsky
f65b9e27ce
Exercise ZSTD_findDecompressedSize() in the simple decompression fuzzer (#3959)
* Improve decompression fuzzer

* Fix legacy frame header fuzzer crash, add unit test
2024-03-12 17:07:06 -04:00
Elliot Gorokhovsky
92fbd42894
Export ZSTD_LEGACY_SUPPORT in tests/Makefile (#3955)
This doesn't affect most of the targets, but will help me sleep better at night knowing that future refactors won't break the legacy support.

Should have been included in https://github.com/facebook/zstd/pull/3943 but I noticed after that merged, so putting up a separate PR.
2024-03-12 14:36:54 -04:00