sharpetronics/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-11-04 00:02:59 -05:00
Code Issues Projects Releases Wiki Activity
8,291 Commits 37 Branches 77 Tags
Commit Graph

8291 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
senhuang42
674c9b9235 Add in proper block repcode histories 2020-11-12 15:34:37 -05:00
senhuang42
06c7f14066 Let block reps persist 2020-11-12 12:24:44 -05:00
senhuang42
396275068c Fix incorrect repcode setting 2020-11-12 11:57:01 -05:00
senhuang42
1a8af0de73 Improve unit test 2020-11-12 11:09:09 -05:00
senhuang42
4d4fd2c55f Overhaul repcode handling logic 2020-11-12 10:59:35 -05:00
Yann Collet
69b8361b0c
Merge pull request #2388 from facebook/fix2386 
fix incorrect assert
 2020-11-06 11:38:08 -08:00
sen
f62edf0fe9
Merge pull request #2381 from senhuang42/expand_sequence_extraction_api 
Add enum to define ZSTD_Sequence type and update sequence extraction API
 2020-11-06 13:00:31 -05:00
Yann Collet
95e74616d5 fix multiple minor conversion warnings 
unrelated to #2386, just cleaning up while I'm updating this file ...
 2020-11-06 09:57:05 -08:00
Yann Collet
2769e4d459 fix incorrect assert 
fix #2386, reported by @Neumann-A
 2020-11-06 09:44:04 -08:00
senhuang42
7d1dea070c Update unit tests 2020-11-06 11:10:37 -05:00
senhuang42
779df995c6 Implement mergeGeneratedSequences() 2020-11-06 10:55:46 -05:00
senhuang42
51abd58208 Rename getSequences() to generateSequences() 2020-11-06 10:53:22 -05:00
senhuang42
261ea69661 Add new mergeGeneratedSequences() function 2020-11-06 10:52:34 -05:00
Nick Terrell
07ee8bfe33
Merge pull request #2385 from LuAPi/add-ZSTD_getDictID_fromCDict-single-commit 
Add ZSTD_getDictID_fromCDict function to experimental section
 2020-11-05 10:36:13 -08:00
Luke Pitt
eac309c71b Add ZSTD_getDictID_fromCDict function to experimental section 2020-11-04 11:37:37 +00:00
senhuang42
f782cac3d4 Change block delimiter removing to linear time approach 2020-11-02 17:06:20 -05:00
senhuang42
3c9b43da1d Remove trailing comma 2020-11-02 11:53:58 -05:00
senhuang42
3434049c1f Use ZSTD_memmove() instead of memmove() 2020-11-02 11:43:19 -05:00
senhuang42
c54a25b666 Revert compressibility change 2020-11-02 11:38:58 -05:00
senhuang42
d4d0346b40 Update name of enum, clarify documentation 2020-11-02 11:38:17 -05:00
senhuang42
9102f30dbf Update unit test 2020-11-02 11:30:31 -05:00
senhuang42
e6178f837f Revert unnecessary seqCollector adjustment 2020-11-02 10:59:20 -05:00
senhuang42
e8501e00b8 Fix incorrect index increment in merge algorithm 2020-11-02 10:58:41 -05:00
senhuang42
a36fdada57 Add algorithm to remove all delimiters 2020-11-02 10:46:52 -05:00
senhuang42
435a3a0428 Update seqCollector definition 2020-11-02 10:19:26 -05:00
senhuang42
3327932609 Update ZSTD_getSequences function signature 2020-11-02 10:17:59 -05:00
senhuang42
7397d0102f Add new enum for different sequence formats for ingestion/extraction 2020-11-02 10:15:53 -05:00
sen
f54109c589
Merge pull request #2378 from senhuang42/free_cress_ptr 
[minor] Pass cRess_t by const ptr instead of by value
 2020-11-01 20:33:25 -05:00
Nick Terrell
37d546c445
Merge pull request #2379 from terrelln/regression-test 
[regression] Updates results.csv & add README
 2020-10-30 15:09:38 -07:00
Nick Terrell
7205e609a9
Merge pull request #2354 from terrelln/stable-buffer 
Add ZSTD_c_stable{In,Out}Buffer and optimize when set
 2020-10-30 15:06:56 -07:00
Nick Terrell
a446fa33dc [regression] Add README explaining the test 2020-10-30 13:55:52 -07:00
Nick Terrell
222916a5d3 [regression] Update results.csv 
https://github.com/facebook/zstd/pull/2339 removes the single-pass zstdmt API.
This changes the compressed size, because we no longer take the # of threads into
account when deciding the job size.
 2020-10-30 13:54:30 -07:00
sen
c37c714ef1
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api 
Refine external ZSTD_Sequence API
 2020-10-30 15:47:25 -04:00
Nick Terrell
2ebf6d5588 [test] Add unit tests for ZSTD_c_stable{In,Out}Buffer 2020-10-30 10:55:34 -07:00
Nick Terrell
d4e021fe35 [lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set 
We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.
 2020-10-30 10:55:34 -07:00
Nick Terrell
24f72789e2 [lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set 
Compress directly from the `ZSTD_inBuffer`. We still allocate the input
buffer. A following commit will remove that allocation.
 2020-10-30 10:55:34 -07:00
Nick Terrell
6bd6b6f7d3 [cwksp] Return NULL when 0 bytes are requested 
This ensures that the buffer is never used.
 2020-10-30 10:55:34 -07:00
Nick Terrell
fcf81cee5e [lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set 
We compress directly to the `ZSTD_outBuffer` so we don't need to
allocate it.
 2020-10-30 10:55:34 -07:00
Nick Terrell
6d5dc93d4e [lib] Compress directly into output when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer always compress directly into the
`ZSTD_outBuffer`. We are allowed to return `dstSizeTooSmall`.
 2020-10-30 10:55:34 -07:00
Nick Terrell
987cb4ca6a [lib] Take the shortcut when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.
 2020-10-30 10:55:34 -07:00
Nick Terrell
809b2f2071 [lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2() 
Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.

An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.
 2020-10-30 10:55:34 -07:00
Nick Terrell
c74be3f6de [lib] Validate buffers when ZSTD_c_stable{In,Out}Buffer is set 
Adds the validation of the input/output buffers only. They are still
unused.
 2020-10-30 10:55:34 -07:00
Nick Terrell
e3e0775cc8 [API] Add ZSTD_c_stable{In,Out}Buffer parameters 
This commit adds the parameters and sets the value in the CCtxParams
but it does not do anything with the value.
 2020-10-30 10:54:39 -07:00
Nick Terrell
e2581d9572 [lib] Set appliedParams in zstdmt mode 
Previously only `nbWorkers` was set. Set all parameters, because that is
what is expected. This is needed for the `ZSTD_c_stable{In,Out}Buffer`
parameters.
 2020-10-30 10:54:38 -07:00
senhuang42
f0da97642a Specify that getSequences() will always emit block boundary sequences 2020-10-30 12:31:17 -04:00
senhuang42
536e89c723 Sequence extractor should update CBlockState 2020-10-30 12:13:19 -04:00
senhuang42
32cac2627a Emit last literals of 0 size as well, to indicate block boundary 2020-10-29 16:41:17 -04:00
senhuang42
69bd5f0654 Correct literalsRead calculation to include longLength 2020-10-29 14:49:37 -04:00
sen
ff93440fc6
Merge pull request #2375 from senhuang42/ldm_oss_fuzz_testcase 
Add a test case for LDM + opt parser with small uncompressible block
 2020-10-29 09:32:05 -04:00
senhuang42
7198ebb213 Un-mix declarations and code 2020-10-28 18:51:03 -04:00