sharpetronics/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-10-15 00:02:02 -04:00
Code Issues Projects Releases Wiki Activity
9,942 Commits 38 Branches 77 Tags
Commit Graph

643 Commits

Author SHA1 Message Date
Elliot Gorokhovsky
bce0382c82
Bugfixes for the External Matchfinder API (#3433) 
* external matchfinder bugfixes + tests

* small doc fix
 2023-01-19 10:41:24 -05:00
daniellerozenblit
dc1c6cc5df
Merge pull request #3418 from daniellerozenblit/fuzz-max-block-size 
Fuzz on maxBlockSize
 2023-01-19 08:18:04 -05:00
Elliot Gorokhovsky
5d8cfa6b96
Deprecate advanced streaming functions (#3408) 
* deprecate advanced streaming functions

* remove internal usage of the deprecated functions

* nit

* suppress warnings in tests/zstreamtest.c

* purge ZSTD_initDStream_usingDict

* nits

* c90 compat

* zstreamtest.c already disables deprecation warnings!

* fix initDStream() return value

* fix typo

* wasn't able to import private symbol properly, this commit works around that

* new strategy for zbuff

* undo zbuff deprecation warning changes

* move ZSTD_DISABLE_DEPRECATE_WARNINGS from .h to .c
 2023-01-13 14:51:47 -05:00
Danielle Rozenblit
14b8defb86 move ZSTD_BLOCKSIZE_MAX_MIN to static linking only section 2023-01-13 07:00:50 -08:00
Nick Terrell
5b266196a4 Add support for in-place decompression 
* Add a function and macro ZSTD_decompressionMargin() that computes the
  decompression margin for in-place decompression. The function computes
  a tight margin that works in all cases, and the macro computes an upper
  bound that will only work if flush isn't used.
* When doing in-place decompression, make sure that our output buffer
  doesn't overlap with the input buffer. This ensures that we don't
  decide to use the portion of the output buffer that overlaps the input
  buffer for temporary memory, like for literals.
* Add a simple unit test.
* Add in-place decompression to the simple_round_trip and
  stream_round_trip fuzzers. This should help verify that our margin stays
  correct.
 2023-01-12 16:28:08 -08:00
Danielle Rozenblit
06b096db47 additional tests and documentation updates + allow maxBlockSize to be set to 0 (goes to default) 2023-01-12 13:41:50 -08:00
Danielle Rozenblit
53eb5a758c add simple test for maxBlockSize expected functionality 2023-01-12 08:55:39 -08:00
Danielle Rozenblit
1fffcfe01d update minimum threshold for max block size 2023-01-11 11:09:57 -08:00
Danielle Rozenblit
fe08137d9a resolve max block value in cctx and use when calculating the max block size 2023-01-09 07:53:53 -08:00
daniellerozenblit
d913417f72
Merge branch 'dev' into fuzz-max-block-size 2023-01-04 16:34:07 -05:00
Danielle Rozenblit
908e812733 initial commit 2023-01-04 13:01:54 -08:00
Yann Collet
9fbbd74871
Merge pull request #3400 from danlark1/dev 
Move deprecated annotation before static to allow C++ compilation for clang
 2022-12-28 15:50:26 -08:00
Yann Collet
00c85b28e7 update ZSTD_CCts_setCParams() inline documentation 
specify behavior when changing compression parameters during MT compression,
reported by @embg
 2022-12-28 15:08:18 -08:00
Yann Collet
481a2e1010
Merge pull request #3403 from facebook/setCParams 
ZSTD_CCtx_setCParams
 2022-12-28 14:07:13 -08:00
Elliot Gorokhovsky
2a402626dd
External matchfinder API (#3333) 
* First building commit with sample matchfinder

* Set up ZSTD_externalMatchCtx struct

* move seqBuffer to ZSTD_Sequence*

* support non-contiguous dictionary

* clean up parens

* add clearExternalMatchfinder, handle allocation errors

* Add useExternalMatchfinder cParam

* validate useExternalMatchfinder cParam

* Disable LDM + external matchfinder

* Check for static CCtx

* Validate mState and mStateDestructor

* Improve LDM check to cover both branches

* Error API with optional fallback

* handle RLE properly for external matchfinder

* nit

* Move to a CDict-like model for resource ownership

* Add hidden useExternalMatchfinder bool to CCtx_params_s

* Eliminate malloc, move to cwksp allocation

* Handle CCtx reset properly

* Ensure seqStore has enough space for external sequences

* fix capitalization

* Add DEBUGLOG statements

* Add compressionLevel param to matchfinder API

* fix c99 issues and add a param combination error code

* nits

* Test external matchfinder API

* C90 compat for simpleExternalMatchFinder

* Fix some @nocommits and an ASAN bug

* nit

* nit

* nits

* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h

* nit

* nit

* nits

* Update copyright headers

* Fix CMake zstreamtest build

* Fix copyright headers (again)

* typo

* Add externalMatchfinder demo program to make contrib

* Reduce memory consumption for small blockSize

* ZSTD_postProcessExternalMatchFinderResult nits

* test sum(matchlen) + sum(litlen) == srcSize in debug builds

* refExternalMatchFinder -> registerExternalMatchFinder

* C90 nit

* zstreamtest nits

* contrib nits

* contrib nits

* allow block splitter + external matchfinder, refactor

* add windowSize param

* add contrib/externalMatchfinder/README.md

* docs

* go back to old RLE heuristic because of the first block issue

* fix initializer element is not a constant expression

* ref contrib from zstd.h

* extremely pedantic compiler warning fix, meson fix, typo fix

* Additional docs on API limitations

* minor nits

* Refactor maxNbSeq calculation into a helper function

* Fix copyright
 2022-12-28 16:45:14 -05:00
Yann Collet
89342d1e07 New xp library symbol : ZSTD_CCtx_setCParams() 
Inspired by #3395,
offer a new capability to set all parameters defined in a ZSTD_compressionParameters structure
with a single symbol invocation
to improve user code brevity.
 2022-12-27 23:49:22 -08:00
Daniel Kutenin
48f4aa7307
Move deprecated annotation before static to allow C++ compilation for clang 
This fixes last 2 instances of https://github.com/facebook/zstd/issues/3250
 2022-12-23 12:07:31 +00:00
Nick Terrell
40a7188130 Fix make clangbuild & add CI 
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
 2022-12-21 17:31:04 -08:00
Nick Terrell
e4018c4e7f [docs] Clarify dictionary loading documentation 
Reinforce that loading a new dictionary clears the current dictionary.
Except for the multiple-ddict mode.
 2022-12-20 11:10:49 -08:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
 2022-12-20 12:52:34 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms' 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
 2022-12-20 12:37:57 -05:00
Yann Collet
2f4238e47a make ZSTD_DECOMPRESSBOUND() compatible with input size 0 
for environments with stringent compilation warnings.
 2022-12-16 16:05:39 -08:00
Yann Collet
51355e1f70
Merge pull request #3362 from facebook/compressBound 
check potential overflow of compressBound()
 2022-12-16 14:22:22 -08:00
Nick Terrell
358a237484 [api][visibility] Make the visibility macros more consistent 
1. Follow the scheme introduced in PR #2501 for both `zdict.h` and `zstd_errors.h`.
2. If the `*_VISIBLE` macro isn't set, but the `*_VISIBILITY` macro is, use that.
   Also make this change for `zstd.h`, since we probably shouldn't have changed
   that macro name without backward compatibility in the first place.
3. Change all references to `*_VISIBILITY` to `*_VISIBLE`.

Fixes #3359.
 2022-12-16 12:54:45 -08:00
Yann Collet
97f63ce2b5 added unit tests for compressBound() 
and rephrased the code documentation, as suggested by @terrelln
 2022-12-16 12:35:14 -08:00
Yann Collet
45ed0df18a check potential overflow of compressBound() 
fixed #3323, reported by @nigeltao

Completed documentation around this risk
(which is largely theoretical,
I can't see that happening in any "real world" scenario,
but an erroneous @srcSize value could indeed trigger it).
 2022-12-15 15:23:15 -08:00
Elliot Gorokhovsky
e1e82f74f1 Reserve two fields in ZSTD_frameHeader 2022-12-13 17:45:05 -08:00
Fangrui Song
5635827ede Move ZSTD_DEPRECATED before ZSTDLIB_API/ZSTDLIB_STATIC_API 
Clang doesn't allow [[deprecated(...)]] attribute after __attribute__.
Move [[deprecated(...)]] before __attribute__ to fix C++14/C++17 uses
with Clang.

Fix #3250
 2022-09-22 12:30:44 -07:00
Yann Collet
434ffe979c minor: refactor publication of ZSTD_copyCCtx() 
for improved clarity
 2022-09-22 11:14:21 -07:00
Danielle Rozenblit
aa82998821 add sequence bound function 2022-09-09 12:34:25 -07:00
Dimitri Papadopoulos
0015308c0f
Fix typos found by codespell 2022-09-08 23:17:00 +02:00
Elliot Gorokhovsky
ef60302af9
Merge pull request #3230 from grossws/fix3229-docs 
Add description for ZSTD_decompressStream and ZSTD_initDStream
 2022-08-16 12:48:23 -04:00
Konstantin Gribov
1c847e2e32 Add description for ZSTD_decompressStream and ZSTD_initDStream 
With that these functions become visible in generated docs.

Fixes #3229
 2022-08-08 18:02:50 +03:00
Felix Handte
7e6278a706
Merge pull request #3196 from mileshu/dev 
[T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd
 2022-08-02 12:34:04 -04:00
Miles HU
c450f9f952 [T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd 
The discussion for this task is here: facebook/zstd#3128.

This task can probably be scoped to the first part: marking these functions deprecated.
We'll later look at removal when we roll out v1.6.0.
 2022-08-01 22:45:52 -07:00
Nick Terrell
0f4fd28a64
Deprecate ZSTD_getDecompressedSize() (#3225) 
Fixes #3158.

Mark ZSTD_getDecompressedSize() as deprecated and replaced by ZSTD_getFrameContentSize().
 2022-08-01 11:52:14 -07:00
Miles HU
a5655e4017 Revert "T119975957" 
This reverts commit 962746edffa5340315136af34ac3331eba82c3c8.
 2022-07-12 11:17:25 -07:00
Miles HU
962746edff T119975957 
Signed-off-by: Miles HU <yuanpu@fb.com>
 2022-07-08 15:01:36 -07:00
Elliot Gorokhovsky
5c382bf110 1.5.3 version bump 2022-06-29 14:45:53 -04:00
Elliot Gorokhovsky
93b89fb24b Add docs 2022-06-22 16:13:07 -04:00
Elliot Gorokhovsky
2a128110d0 Add prefetchCDictTables CCtxParam 2022-06-22 16:13:07 -04:00
Dominique Pelle
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Yann Collet
bad7f82300
Merge pull request #2974 from facebook/fix2966_part3 
Lazy parameters adaptation (part 1 - ZSTD_c_stableInBuffer)
 2022-01-27 06:14:04 -08:00
Yann Collet
7616e39f3b adding traces to better track processing of literals 2022-01-26 14:47:21 -08:00
Yann Collet
27d336b099 minor behavior refinements 
specifically, there is no obligation to start streaming compression with pos=0.
stableSrc mode is now compatible with this setup.
 2022-01-26 10:31:24 -08:00
Yann Collet
37b87add7a make stableSrc compatible with regular streaming API 
including flushStream().

Now the only condition is for `input.size` to continuously grow.
 2022-01-26 10:31:24 -08:00
Yann Collet
c0c5ffa973 streaming compression : lazy parameter adaptation with stable input 
effectively makes ZSTD_c_stableInput compatible ZSTD_compressStream()
and zstd_e_continue operation mode.
 2022-01-26 10:31:24 -08:00
Yann Collet
87dcd3326a fix sequence compression API in Explicit Delimiter mode 2022-01-25 13:33:41 -08:00
Tom Rix
2b957afec7 cleanup double word in comment. 
Remove the second 'a' and 'into'

Signed-off-by: Tom Rix <trix@redhat.com>
 2022-01-24 12:43:39 -08:00
W. Felix Handte
46ad9377e8 Bump Version Number to 1.5.2 2022-01-07 14:14:26 -05:00