sharpetronics/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-12-06 00:02:05 -05:00
Code Issues Projects Releases Wiki Activity
8,980 Commits 38 Branches 77 Tags
Commit Graph

3903 Commits

Author SHA1 Message Date
Yann Collet
2901b5e675 Merge branch 'dev' into cli-dll 2021-01-07 10:24:09 -08:00
Thomas Waldmann
f9802d80a0 fix typos (work done by Andrea Gelmini) 2021-01-07 18:47:23 +01:00
senhuang42
c2c9b8a7ec Address comments, clean up interface/internals 2021-01-07 12:29:12 -05:00
senhuang42
22b7bff2bc Add unit test, improve documentation 2021-01-07 12:29:12 -05:00
senhuang42
ea52fc3606 Use XXHash for hash function, create a sensible public interface 2021-01-07 12:29:12 -05:00
senhuang42
7c1a79f232 Add debuglog statements 2021-01-07 12:29:11 -05:00
senhuang42
d1a6a9d285 Reference requested dict ID at decompression time 2021-01-07 12:29:11 -05:00
senhuang42
5a6d3eef2b Allocate memory for DDict hash set when parameter is set 2021-01-07 12:29:11 -05:00
senhuang42
fd5b608f1c Add parameter to control multiple DDicts 2021-01-07 12:29:11 -05:00
senhuang42
f933668d3f Implement hashset for dictIDs 2021-01-07 12:29:11 -05:00
Yann Collet
cefdc023f7 The CLI can be linked to libzstd dynamic library 
invoking target zstd-dll
 2021-01-06 18:00:24 -08:00
Yann Collet
9866148e22 removed redundant tests 
clang v3.8 tests are either flacky or redundant,
prefer using clang-latest.
 2021-01-06 15:40:20 -08:00
Nick Terrell
58476bcf7f Don't shrink window log in ZSTD_getCParams() 
Treat ZSTD_getCParams() and ZSTD_adjustCParams() in the same way
we treat streaming compression. Choose parameters based on the
dictionary size + source size, and assume the source size is small
if unkown. But, don't shrink the window log down in
ZSTD_adjustCParams_internal().
 2021-01-04 15:54:09 -08:00
Nick Terrell
9d31c704d5 Don't shrink window log when streaming with a dictionary 
Fixes #2442.

1. When creating a dictionary keep the same behavior as before.
   Assume the source size is 513 bytes when adjusting parameters.
2. When calling ZSTD_getCParams() or ZSTD_adjustCParams() keep
   the same behavior as before.
3. When attaching a dictionary keep the same behavior of ignoring
   the dictionary size. When streaming this will select the
   largest parameters and not adjust them down. But, the CDict
   will use the correctly sized parameters, which seems like the
   right tradeoff.
4. When not attaching a dictionary (either forced not to, or
   using a prefix dictionary) we select parameters based on the
   dictionary size + source size, and assume the source size is
   small, which is the same behavior as before. But, now we don't
   adjust the window log (and hash and chain log) down when the
   source size is unknown.

When the source size is unknown all cdicts should attach, except
when the user disables attaching, or `forceWindow` is used. This
means that when streaming with a CDict we end up in the good case
where we get small CDict parameters, and large source parameters.

TODO: Add a streaming + dictionary regression test case.
 2021-01-04 15:54:09 -08:00
Nick Terrell
66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
Yann Collet
cfff4c1cd5
Merge pull request #2439 from senhuang42/skippable_frame_api 
Generate skippable frame API
 2020-12-28 11:22:07 -08:00
Gregory Szorc
dd1a7e41ee Add ifndef guards for _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE 
This ensures the symbols aren't redefined, which would result in a compiler
error.

I was getting redefined symbols for _LARGEFILE64_SOURCE when building for
32-bit x86 Linux on an older CentOS release in a CI environment. With this
change, I'm able to compile the single file library in this environment.

Closes #2443.
 2020-12-26 10:02:45 -07:00
senhuang42
5c41490bfe Use pre-defined constants 2020-12-21 11:52:05 -05:00
senhuang42
7e11bd012b Implement skippable frame function 2020-12-21 11:13:22 -05:00
Yann Collet
9648bf027b try to keep libzstd.a "as is" once created 
to be compatible with scenarios such as
`make -j allmost`
 2020-12-20 17:10:57 -08:00
Yann Collet
7c495e8ea2 updated version number to v1.4.8 2020-12-18 15:52:11 -08:00
Yann Collet
a7cb4af573 added emphasis on the alignment condition of workspace 
and made it a programming mistake (`assert()`)
rather than a runtime error.
 2020-12-18 15:04:09 -08:00
Nick Terrell
ae85676d44 Fix alignment of scratchBuffer in HUF_compressWeights() 
The scratch buffer must be 4-byte aligned. This causes test failures in
32-bit systems, where the stack isn't aligned.

Fixes Issue #2428.
 2020-12-17 14:30:27 -08:00
Yann Collet
0b39531d75 moving all references to release branch 
was previously `master`
 2020-12-16 23:00:35 -08:00
Yann Collet
f647a759fe updated version number to v1.4.7 
and updated doc
 2020-12-15 20:53:05 -08:00
Yann Collet
b8c3a473ec
Merge pull request #2420 from terrelln/huf-comment 
[huf_compress] Refactor and comment HUF_buildCTable()
 2020-12-14 16:14:07 -08:00
Yann Collet
c56723ab03 replace final links by direct copy 
link can behave slightly differently from real binaries,
breaking a few scripts relying on "real binary" assumption.
 2020-12-10 13:25:08 -08:00
Felix Handte
f861e8c07b
Merge pull request #2421 from felixhandte/pc-no-sed 
Don't Use Regexes to Build Pkg-Config File
 2020-12-09 18:58:17 -05:00
W. Felix Handte
9dab03db90 Create Enum to Represent Static/Dynamic Allocation Distinction in cwksp 2020-12-09 14:57:37 -05:00
W. Felix Handte
db9e73cb07 Don't ASAN-Poison Statically-Allocated Workspaces 
Addresses #2286.
 2020-12-09 13:00:47 -05:00
W. Felix Handte
b521183c74 Avoid Use of Regexes in Building Package-Config File 2020-12-08 20:10:05 -05:00
Nick Terrell
1bbcf07bd5 [huf_compress] Refactor and comment HUF_buildCTable() 
Comment and refactor `HUF_buildCTable()` and the helper functions
it calls as I read and understand the code. Hopefully this refactor
makes the code a bit more clear.
 2020-12-08 13:57:01 -08:00
Yann Collet
b86e3c9304
Merge pull request #2415 from facebook/fix_aliasing 
fix gcc-10 strict aliasing warnings
 2020-12-04 21:30:57 -08:00
Nick Terrell
fad175f9c1
Merge pull request #2412 from animalize/dict_compressionlevel 
use ZSTD_CLEVEL_DEFAULT in zdict.c
 2020-12-04 17:09:30 -08:00
Yann Collet
6132df8dd3 fix gcc-10 strict aliasing warnings 
by exposing HUF_CElt declaration.
 2020-12-04 16:43:19 -08:00
Yann Collet
68c14bdff2 minor speed improvement to HUF_readCTable() 
faster by ~+1-2%
 2020-12-04 16:33:39 -08:00
Nick Terrell
c238db046f
Merge pull request #2414 from terrelln/mt-progress 
[lib] Ensure that multithreaded compression always makes some progress
 2020-12-04 16:30:08 -08:00
Nick Terrell
4c58cb8383 [lib] Ensure that multithreaded compression always makes some progress 2020-12-03 20:25:14 -08:00
animalize
1aec77ea89 use ZSTD_CLEVEL_DEFAULT in zdict.c 2020-12-03 12:46:57 +08:00
Nick Terrell
6672689e7e
Merge pull request #2406 from terrelln/linux-wrapper-api 
[linux] Add the linux wrapper API
 2020-12-02 16:49:03 -08:00
Yann Collet
91c1b57be9
Merge pull request #2409 from facebook/test_makefile 
Minor refactor
 2020-12-02 15:33:54 -08:00
Nick Terrell
894ae36675
Merge pull request #2390 from animalize/clamp_level 
Clamp compression level
 2020-12-02 14:35:58 -08:00
senhuang42
2cbd038528 Move max nb seq check to per-block 2020-12-02 12:11:32 -05:00
Nick Terrell
3cda5fae77 [minor][lib] Remove double semicolon 2020-12-02 01:08:08 -08:00
Yann Collet
9f8b180d5d fixed API documentation 2020-12-02 00:15:07 -08:00
Yann Collet
6112b82526
Merge pull request #2348 from dscheg/dev 
Fix dll path in case of cross-compilation
 2020-12-01 17:59:56 -08:00
senhuang42
3efe9c902b Add sequence nb validation to compressSequences(), adjust minMatch comparisons 2020-12-01 10:54:45 -05:00
senhuang42
4c5f337248 Use cctx's minMatch instead of global MINMATCH, make fuzzer use validation 2020-11-30 15:41:20 -05:00
Dmitriy Titarenko
61f71753d4 Pass dictBufferCapacity to COVER_selectDict() 
closes #2371
 2020-11-22 23:45:18 +05:00
sen
c5fbd55dac
Merge pull request #2387 from senhuang42/compress_sequence_API 
[RFC] New sequence compression API
 2020-11-20 16:54:20 -05:00