sharpetronics/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-10-06 00:04:13 -04:00
Code Issues Projects Releases Wiki Activity
9,060 Commits 38 Branches 77 Tags
Commit Graph

9060 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
sen
c7afbec4c1
Merge pull request #2780 from senhuang42/blocksplit_stack_reduce 
Reduce stack usage of block splitter
 2021-09-23 09:48:04 -04:00
senhuang42
1d8143c84f Move block splitter from stack to CCtx 2021-09-23 00:02:31 -04:00
Nick Terrell
e3feec74d3
Merge pull request #2795 from terrelln/linux-fixes 
[contrib][linux] Reduce stack usage by 80 bytes
 2021-09-22 19:56:53 -07:00
Nick Terrell
162491f601 [contrib][linux] Reduce stack usage by 80 bytes 
Instead of calling `ZSTD_compress_advanced()` and
`ZSTD_initCStream_advanced()`, which each take a `ZSTD_parameters` by
value, use the new advanced API.

Stack usage went from 2024 -> 1944.
 2021-09-22 18:18:47 -07:00
Nick Terrell
b25d57461e
Merge pull request #2794 from terrelln/spdx-license 
[contrib][linux] Fix up SPDX license identifiers
 2021-09-22 17:33:31 -07:00
Yann Collet
441eb2654b
Merge pull request #2793 from facebook/rdoc 
Minor documentation update
 2021-09-22 17:07:49 -07:00
Nick Terrell
4d347a902b [contrib][linux] Fix up SPDX license identifiers 
Correctly identify that we are GPL v2+ or BSD 3 clause, as pointed out
in issue #2663.
 2021-09-22 15:09:42 -07:00
Yann Collet
8150891939 regenerated zstdless.1 2021-09-22 14:48:51 -07:00
Yann Collet
3addf2f277 updated zstdgrep man page 2021-09-22 14:30:59 -07:00
Yann Collet
70b36c2308 update zstdgrep doc 
to mention ripgrep alternative which transparently supports zstd-compressed files.
 2021-09-22 14:30:06 -07:00
Yann Collet
999f8778af updated man pages 
using ronn-ng
 2021-09-22 14:18:24 -07:00
Yann Collet
2832bbbbdc emphasize usage of -r in documentation 
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
 2021-09-22 14:00:20 -07:00
sen
044c8b4722
Merge pull request #2779 from senhuang42/fse_fix 
Fix NCountWriteBound
 2021-09-22 13:51:21 -04:00
sen
1e99d36361
Merge pull request #2788 from senhuang42/param_switch 
Use new paramSwitch enum for row matchfinder and block splitter
 2021-09-22 13:27:55 -04:00
Nick Terrell
5208e14ae0
Merge pull request #2791 from terrelln/huf-asm-fixes 
[huf] Fix compilation when DYNAMIC_BMI2=0 && BMI2 is supported
 2021-09-22 09:59:02 -07:00
senhuang42
99b5e7b8c2 Add test case for FSE over-write 2021-09-22 12:03:46 -04:00
Nick Terrell
9450876a9d [huf] Fix compilation when DYNAMIC_BMI2=0 && BMI2 is supported 
* Fix compilation issues pointed out in PR #2790.
* Add test cases to GitHub actions that test all combinations of
  `DYNAMIC_BMI2` BMI2 support.
 2021-09-21 16:49:13 -07:00
senhuang42
06f42c3bfd Use new paramSwitch enum for LDM 2021-09-21 14:22:09 -04:00
senhuang42
b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
Nick Terrell
8385355fac
Merge pull request #2722 from terrelln/huf-asm 
Assembly implementation of 4X1 & 4X2 Huffman
 2021-09-20 17:32:30 -07:00
Nick Terrell
a5f2c45528 Huffman ASM 2021-09-20 14:46:43 -07:00
Nick Terrell
51b123d5f7
Merge pull request #2784 from terrelln/huf-asm-fuzzer 
[fuzzer] Add Huffman decompression fuzzer
 2021-09-20 11:57:27 -07:00
Yann Collet
847dcb65bf
Merge pull request #2787 from animalize/msvc2019 
add msvc2019 to build.generic.cmd
 2021-09-20 11:14:51 -07:00
Ma Lin
4eef208889 add msvc2019 to build.generic.cmd 2021-09-19 10:28:16 +08:00
Nick Terrell
d7542aacd9 [fuzzer] Add huf_decompress fuzzer 
Add a fuzzer for Huffman decompression. Fix several bugs in Huffman
decompression, mostly related to `op == NULL` and pointer underflow.
 2021-09-17 15:00:49 -07:00
Nick Terrell
6392c382bf
Merge pull request #2783 from terrelln/huf-asm-makefiles 
[build] Add support for ASM files in Make + CMake
 2021-09-17 14:56:47 -07:00
Nick Terrell
8bf699aa59 [build] Add support for ASM files in Make + CMake 
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
 2021-09-17 14:13:53 -07:00
sen
9d2a45a705
Merge pull request #2778 from senhuang42/opt_inlining_revert 
Revert opt outlining change
 2021-09-15 14:22:10 -04:00
Sen Huang
a7aa2c5df6 Fix NCountWriteBound 2021-09-15 09:51:42 -07:00
Sen Huang
bd84e4a9d3 Revert opt outlining change 2021-09-15 09:08:41 -07:00
Nick Terrell
2fabd370bb
Merge pull request #2777 from terrelln/oss-fuzz-fix 
[rsyncable] Fix test failures
 2021-09-14 13:20:22 -07:00
Nick Terrell
9d9e2ed00b [rsyncable] Fix test failures 
Test failures showed up on the daily cron job. They didn't show up
in CI because the condition is somewhat rare, and didn't trigger
during the CI tests.

This PR fixes up the logic in `findSynchronizationPoint()` to correctly
handle the edge case. It also un-comments an assert that helps catch the
issue, and verify that rsyncable mode is calculating the correct hash.

After the fix, the test that failed passes:

```
./zstreamtest --newapi -t1 --no-big-tests -s9680
```
 2021-09-14 12:28:53 -07:00
Yann Collet
2e6f5bc0d8
Merge pull request #2771 from facebook/opt_investigation 
Improve optimal parser performance on small data
 2021-09-14 10:36:34 -07:00
Nick Terrell
d22bbed5db
Merge pull request #2776 from terrelln/oss-fuzz-fix 
[rsyncable] Ensure ZSTD_compressBound() is respected
 2021-09-14 09:37:43 -07:00
Yann Collet
fd94b9d1c9 Merge branch 'dev' into opt_investigation 2021-09-14 01:15:51 -07:00
sen
29f595ef76
Merge pull request #2773 from senhuang42/huffman_bugix 
Fix dictionary training huffman segfault and small speed improvement
 2021-09-13 21:02:42 -04:00
Nick Terrell
a418b4e478 [rsyncable] Ensure ZSTD_compressBound() is respected 
In degenerate cases `--rsyncable` could create very small blocks (1
byte). This causes the compressed output to be larger than
`ZSTD_compressBound()`. Fix the issue by ensuring that rsyncable mode
never outputs blocks smaller than 128 KB.

The minimum job size is 512 KB, so we shouldn't lose many
synchronization points from skipping any that cause blocks smaller than
128 KB. And even if we do, that is fine, because we'll find the next
one.

This fixes the `raw_dictionary_round_trip` oss-fuzz assert.

Credit to OSS-Fuzz
 2021-09-13 17:14:07 -07:00
Sen Huang
d45d0ad9d8 Update regression test 2021-09-13 12:41:02 -04:00
Sen Huang
4a498fb9c3 Add a dictionary training large corpus test 2021-09-13 12:29:17 -04:00
Sen Huang
1daf3c8dbc Use 32 buckets for log2 bucketing in huffman sort 2021-09-13 12:29:16 -04:00
Yann Collet
b6b2855b80 updated regression tests 2021-09-12 10:22:35 -07:00
Yann Collet
f58e63bee7 Merge branch 'dev' into opt_investigation 2021-09-12 01:42:49 -07:00
Yann Collet
640c5b1f77 fix automated_benchmarking 
make it able to process text output sent into either stdout or stderr
 2021-09-12 01:36:18 -07:00
Yann Collet
c10067c44e
Merge pull request #2775 from eli-schwartz/meson 
meson: fix type error for integer option
 2021-09-10 05:47:52 -07:00
Eli Schwartz
193aa49673
meson: fix type error for integer option 
meson forgave using the wrong type, but this isn't guaranteed. muon
simply failed.
 2021-09-09 23:40:58 -04:00
Felix Handte
d68aa19a2f
Merge pull request #2749 from felixhandte/zstd-fast-pipelined 
Pipelined Implementation of ZSTD_fast (~+5% Speed)
 2021-09-09 17:05:30 -04:00
Yann Collet
5449ede2e6 make automated-benchmarking faster 
by employing parallel compilation of object files.
 2021-09-08 15:12:28 -07:00
Yann Collet
b7f46ebc23 use ZSTD_memcpy() for better portability 
notably within kernel space
 2021-09-08 14:45:53 -07:00
Yann Collet
4f0b1b9ee5 update regression tests 2021-09-08 14:37:42 -07:00
Yann Collet
7fce9a41b5 change update rate to 12/11/11/11 
better for large files, and sources with relatively "stable" entropy,
like silesia.tar.
slightly worse for files with rapidly changing entropy,
like Calgary.tar/.

Updated small files tests in fuzzer
 2021-09-08 14:05:57 -07:00