sharpetronics/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-11-04 00:02:59 -05:00
Code Issues Projects Releases Wiki Activity
7,730 Commits 37 Branches 77 Tags
Commit Graph

49 Commits

Author SHA1 Message Date
Nick Terrell
4e0515916d [lib] Fix repcode validation in no dict mode 2020-05-12 11:57:15 -07:00
Nick Terrell
5fcbc484c8
Merge pull request #2040 from caoyzh/dev-2 
Optimize by prefetching on aarch64
 2020-04-08 13:14:47 -07:00
Nick Terrell
ac58c8d720 Fix copyright and license lines 
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
 2020-03-26 17:02:06 -07:00
caoyzh
7201980650 Optimize by prefetching on aarch64 2020-03-14 15:25:59 +08:00
Nick Terrell
659e9f05cf Fix null pointer addition 2019-11-20 18:36:04 -08:00
Nick Terrell
ddab2a94e8 Pass iend into ZSTD_storeSeq() to allow ZSTD_wildcopy() 2019-09-20 00:56:20 -07:00
Yann Collet
facbe8b2c2 factored the logic selecting lowest match index 
as suggested by @terrelln
 2019-08-05 15:18:43 +02:00
Yann Collet
98692c2838 fixed compression ratio regression when dictionary-compressing medium-size inputs at levels 1-3 2019-08-01 15:58:17 +02:00
Yann Collet
eaeb7f00b5 updated the _extDict variant of double fast 2019-07-12 14:17:17 -07:00
Yann Collet
e8a7f5d3ce double-fast: changed the trade-off for a smaller positive change 
same number of complementary insertions, just organized differently
(long at `ip-2`, short at `ip-1`).
 2019-07-12 11:34:53 -07:00
Yann Collet
d1327738c2 updated double_fast complementary insertion 
in a way which is more favorable to compression ratio,
though very slightly slower (~-1%).

More details in the PR.
 2019-07-11 15:25:22 -07:00
Yann Collet
80d6ccea79 removed UINT32_MAX 
apparently not guaranteed on all platforms,
replaced by UINT_MAX.
 2019-05-31 17:27:07 -07:00
Yann Collet
fce4df3ab7 fixed wrong assert in double_fast 2019-05-31 17:06:28 -07:00
Yann Collet
d605f482c7 make double_fast compatible with new index invalidation strategy 2019-05-31 16:50:04 -07:00
Yann Collet
e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
W. Felix Handte
bad74c4781 Use Working Ctx Logs when not in DMS Mode 
We pre-hash the ptr for the dict match state sometimes. When that actually
happens, a hashlog of 0 can produce undefined behavior (right shift a long
long by 64). Only applies to unoptimized compilations, since when
optimizations are applied, those hash operations are dropped when we're not
actually in dms mode.
 2018-09-28 17:12:54 -07:00
W. Felix Handte
22fcb8d4c7 Support Split Logs in ZSTD_dfast 2018-09-28 17:12:54 -07:00
W. Felix Handte
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
b76c888497 ZSTD_dfast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:24:25 -07:00
Yann Collet
357c648c3f changed a few variable names 
to unify naming convention
 2018-06-04 17:10:50 -07:00
Yann Collet
2108decb41 Fixed a nasty corruption bug 
recently introduce into the new dictionary mode.
The bug could be reproduced with this command :
./zstreamtest -v --opaqueapi --no-big-tests -s4092 -t639

error was in function ZSTD_count_2segments() :
the beginning of the 2nd segment corresponds to prefixStart
and not the beginning of the current block (istart == src).
This would result in comparing the wrong byte.
 2018-06-01 18:54:34 -07:00
W. Felix Handte
f86796639e Remove Incorrect and Extraneous Repcode Bounds Check 2018-05-31 17:02:29 -04:00
W. Felix Handte
5b292b5685 Check Long + 1 Matches in Both Prefix and Dict in Bothe Short Match Paths 2018-05-25 13:13:57 -04:00
W. Felix Handte
88b733b380 Interleave Prefix and Dict Searches 2018-05-25 13:13:57 -04:00
W. Felix Handte
1850025156 Refactor ZSTD_dfast to Use gotos 2018-05-25 13:13:57 -04:00
W. Felix Handte
43606f9c83 ... When I Said "HashTable", I Meant "ChainTable" 2018-05-25 13:13:28 -04:00
W. Felix Handte
ec7efe88f5 Fix Off-By-One Error 2018-05-25 13:13:28 -04:00
W. Felix Handte
2bfe43267e Disallow Too-Long Repcodes When Using an Attached Dict 2018-05-25 13:13:28 -04:00
W. Felix Handte
b97ad3f457 Port Changes Made to ZSTD_fast to ZSTD_dfast 2018-05-25 13:13:28 -04:00
W. Felix Handte
2313cca1b7 Implement Second Repcode Check 2018-05-25 13:13:28 -04:00
W. Felix Handte
0998f10813 Implement First Repcode Check 2018-05-25 13:13:28 -04:00
W. Felix Handte
50c5b2bb90 Find Dict Hash Table Matches 2018-05-25 13:13:28 -04:00
W. Felix Handte
7a25f7ef5b Existing Repcode Check Only Applies to noDict Case 2018-05-25 13:13:28 -04:00
W. Felix Handte
8b241da4df Properly Initialize Repcode Values 2018-05-25 13:13:28 -04:00
W. Felix Handte
7097a03749 Add Necessary Dict Variables 2018-05-25 13:13:28 -04:00
W. Felix Handte
aacbbf4f9a Rename 'lowest' to 'localLowest' to Prepare to Introduce Dict Indices 2018-05-25 13:13:28 -04:00
W. Felix Handte
c10d1b4011 Skeleton for In-Place Impl for ZSTD_dfast 2018-05-25 13:13:28 -04:00
Yann Collet
ad4524d605 fix ZSTD_compressBlock() associated with CDict 
reported by @let-def.

It's actually a bug in ZSTD_compressBegin_usingCDict()
which would pass a wrong pledgedSrcSize value (0 instead of ZSTD_CONTENTSIZE_UNKNOWN)
resulting in wrong window size, resulting in downsized seqStore,
resulting in segfault when writing into the seqStore later in the process.

Added a test in fuzzer to cover this use case (fails before the patch).
 2018-05-07 12:54:13 -07:00
Nick Terrell
295ab0dbfa Only load extra table positions for CDicts 
Zstdmt uses prefixes to load the overlap between segments. Loading extra
positions makes compression non-deterministic, depending on the previous
job the context was used for. Since loading extra position takes extra
time as well, only do it when creating a `ZSTD_CDict`.

Fixes #1077.
 2018-04-02 14:41:30 -07:00
Nick Terrell
7e5e226cbf Split the window state into substructure 2018-02-26 13:29:57 -08:00
Yann Collet
5188749e1c ensure compression parameters are updated when only compression level is changed 2018-02-02 16:31:20 -08:00
Nick Terrell
887cd4e35e Split ZSTD_CCtx into smaller sub-structures 2018-01-16 11:17:50 -08:00
Nick Terrell
9a211d1f05 Load more dictionary positions into table if empty 
If the hash table is empty load positions into the hash table
that we would otherwise skip.

| Level | Data Set     | Improvement |
|-------|--------------|-------------|
| 1     | github       | 0.44%       |
| 1     | hg-changelog | 0.13%       |
| 1     | hg-commands  | 1.28%       |
| 1     | hg-manifest  | 0.70%       |
| 3     | github       | 0.74%       |
| 3     | hg-changelog | 0.87%       |
| 3     | hg-commands  | 1.74%       |
| 3     | hg-manifest  | 0.23%       |
 2018-01-12 16:17:22 -08:00
Yann Collet
ee441d5d2b renamed zstd_compress.h into zstd_compress_internal.h 
to emphasize the fact that all definitions it contains
must remain private, accross lib/compress modules.
 2017-11-07 16:15:23 -08:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Nick Terrell
721726d688 Split parsers out of zstd_compress.c 2017-09-05 17:10:25 -07:00