3949 Commits

Author SHA1 Message Date
Nick Terrell
ebc2dfa821
Merge pull request #2524 from terrelln/huf-stack-reduction
[huf] Reduce stack usage of HUF_readDTableX2 by ~972 bytes
2021-03-22 12:37:54 -07:00
Nick Terrell
634bfd339f [FSE] Clean up workspace using dynamically sized struct 2021-03-22 11:07:07 -07:00
Sen Huang
dff4a0e867 Make ZSTD_estimateCCtxSize_internal() loop through all srcSize parameter sets as well 2021-03-21 16:15:31 -07:00
Yann Collet
f5434663ea Refactor prefetching for the decoding loop
Following #2545,
I noticed that one field in `seq_t` is optional,
and only used in combination with prefetching.
(This may have contributed to static analyzer failure to detect correct initialization).

I then wondered if it would be possible to rewrite the code
so that this optional part is handled directly by the prefetching code
rather than delegated as an option into `ZSTD_decodeSequence()`.

This resulted into this refactoring exercise
where the prefetching responsibility is better isolated into its own function
and `ZSTD_decodeSequence()` is streamlined to contain strictly Sequence decoding operations.
Incidently, due to better code locality,
it reduces the need to send information around,
leading to simplified interface, and smaller state structures.
2021-03-19 15:48:17 -07:00
Nick Terrell
756bd59322 [huf][fse] Clean up workspaces
* Move `counting` to a struct in `FSE_decompress_wksp_body()`
* Fix error code in `FSE_decompress_wksp_body()`
* Rename a variable in `HUF_ReadDTableX2_Workspace`
2021-03-17 16:50:37 -07:00
ihsinme
a5bf09d764
simple fix for using bit operator.
good day.
It seems to me that the developer intended to use a logical operator.
so I suggest a simple fix.
2021-03-17 11:37:42 +03:00
Sen Huang
77ae664ba6 Fix ZSTD_dedicatedDictSearch_isSupported() requirements 2021-03-16 17:36:05 -07:00
senhuang42
386111adec Add a nbSeq argument to compressSequences()
Refactor ZSTD_compressBlock_internal() to do the block header write within and add nbSeq argument to compressSequences()
2021-03-16 14:04:22 -07:00
senhuang42
98764493cf Move block header write into compressBlock_internal() 2021-03-16 14:04:22 -07:00
Nick Terrell
ea288e0d8e [lib] Bump zstd version number 2021-03-16 11:47:27 -07:00
Nick Terrell
cd1551d261 [lib][tracing] Add ZSTD_NO_TRACE macro
When defined, it disables tracing, and avoids including the header.
2021-03-16 11:47:27 -07:00
Yann Collet
3d6c903056
Merge pull request #2521 from animalize/doc_free
doc: ZSTD_free*() functions accept NULL pointer
2021-03-06 21:33:28 -08:00
Nick Terrell
3b1aba42cc [fse] Reduce stack usage of FSE_decompress_wksp() by 512 bytes
* Move `counting` into the workspace
* Inrease `HUF_DECOMPRESS_WORKSPACE_SIZE` by 512 bytes
2021-03-05 13:24:27 -08:00
Nick Terrell
0f18059a4e [huf] Reduce stack usage of HUF_readDTableX2 by ~460 bytes
* Use `HUF_readStats_wksp()`
* Use workspace in `HUF_fillDTableX2*()`
* Clean up workspace usage to use a workspace struct
2021-03-05 12:39:46 -08:00
Nick Terrell
b5fd348a85
Merge pull request #2523 from terrelln/huf-stack-reduction
Add HUF_writeCTable_wksp() function
2021-03-05 12:35:09 -08:00
Nick Terrell
5df2a21f1e Add HUF_writeCTable_wksp() function
This saves ~700 bytes of stack space in HUF_writeCTable.
2021-03-05 10:29:18 -08:00
Nick Terrell
e50f88ca4c
Merge pull request #2522 from terrelln/stack-reduction
Reduce stack usage of ZSTD_buildCTable()
2021-03-04 20:55:58 -08:00
Nick Terrell
27498ff00f Reduce stack usage of ZSTD_buildCTable()
It is a stack high-point for some compression strategies and has an easy
fix. This moves the normalized count into the entropy workspace.
2021-03-04 16:12:11 -08:00
animalize
0933775d79 doc: ZSTD_free*() functions accept NULL pointer 2021-03-04 12:02:52 +08:00
Yann Collet
029f974ddc strengthen compilation flags 2021-03-02 15:43:52 -08:00
W. Felix Handte
d7db928f72 Bump Library Version 1.4.8 -> 1.4.9 2021-03-01 17:45:30 -05:00
Yann Collet
61b63e9060
Merge pull request #2492 from niacat/dev
Use standard md5 tool on NetBSD.
2021-02-24 16:38:10 -08:00
Nick Terrell
7736549bea [bug-fix] Make simple single-pass functions ignore advanced parameters
The simple compression functions are intended to ignore the advanced
parameters, but they were accidentally using them. All the
`ZSTD_parameters` were set correctly, but any extra parameters were
used as-is. E.g. `ZSTD_c_format`.

This PR makes all the simple single-pass functions listed below ignore
the advanced parameters, as intended.

* `ZSTD_compressCCtx()`
* `ZSTD_compress_usingDict()`
* `ZSTD_compress_usingCDict()`
* `ZSTD_compress_advanced()`
* `ZSTD_compress_usingCDict_advanced()`

It also adds a test case that ensures that each of these functions
ignore the advanced parameters.
2021-02-12 19:11:23 -08:00
Nick Terrell
c62eb05964 [lib] Set appliedParams.compressionLevel correctly
Forward the correct compressionLevel to the appliedParams in all cases.
It was already correct for the advanced API, so only the old single-pass
functions needed to be fixed.

This compression level is unused by the library, but is set so that the
tracing framework can consume it.
2021-02-12 15:00:14 -08:00
Nick Terrell
f520f6dfbe [trace] Minor fixes found during integration
* Mark `ZSTD_CCtx_getParameter()` as const
* Add `extern "C"` guards to `zstd_trace.h`
2021-02-11 16:20:04 -08:00
Yann Collet
8884cb887d
Merge pull request #2483 from mpu/ldmgear
New algorithms for the long distance matcher
2021-02-11 08:38:23 -08:00
nia
74f85818a6 Use standard md5 tool on NetBSD.
This avoids a GNU coreutils dependency.

-n is used to match the output format of coreutils:
http://man.netbsd.org/md5.1
2021-02-11 10:50:11 +01:00
Quentin Carbonneaux
552efcac2d relocate large arrays from the stack to ldmState_t 2021-02-10 16:16:54 +01:00
Nick Terrell
e59c9459a5 [trace] Keep track of a uint64_t tracing context
The most common information that you want to track between begin() and
end() is the timestamp of the begin function, so you can measure the
duration of the (de)compression call. Allow the tracing library to put
this information inside the `ZSTD_TraceCtx`, so it doesn't need to keep
a global map in this case. If a single uint64_t is not enough, the
tracing library can return a unique identifier (like the context
pointer) instead, and use it as a key in a map.

This keeps the simple case simple.
2021-02-09 11:37:05 -08:00
Quentin Carbonneaux
e2ad174d73 fix some compiler warnings 2021-02-08 20:19:16 +01:00
Nick Terrell
54a4998a80 Add basic tracing functionality 2021-02-05 16:28:52 -08:00
Nick Terrell
f9b1e711ba [zstd] Fix NULL pointer addition in ZSTD_checkContinuity()
Don't start a new section when `dstSize == 0` to avoid NULL
pointer addition.
2021-02-05 12:18:06 -08:00
Yann Collet
b9748757b0 fixed minor cast warning 2021-02-05 09:55:54 -08:00
Quentin Carbonneaux
874a590e5c deal safely with short inputs in ZSTD_ldm_generateSequences
The fuzzer CI found this bug.
2021-02-04 11:15:24 +01:00
Quentin Carbonneaux
9f327c02fd new core ldm algorithm 2021-02-03 22:24:07 +01:00
Quentin Carbonneaux
aee3dc877f fix a variable name to reflect its nature 2021-01-22 02:24:19 -08:00
Quentin Carbonneaux
d6e3de77dc fix warning and remove one more occurrence of makeEntryAndInsertByTag 2021-01-20 01:39:16 -08:00
Quentin Carbonneaux
e0d5eca8fa fix forgotten numTagBits in getTagMask 2021-01-20 00:54:20 -08:00
Quentin Carbonneaux
1e65711ca5 a couple performance improvement changes for ldm 2021-01-20 00:54:20 -08:00
Yann Collet
0bad3e5c0f parallel make build on linux
fix #2474
2021-01-18 11:33:03 -08:00
yumeyao
821d9acd17
Fix visibility of symbols in .so (#2441)
Fix visibility of symbols in .so and add an alias for renamed API
2021-01-08 14:27:31 -08:00
sen
69085db61c
Merge pull request #2446 from senhuang42/multiple_ddicts_v3
[RFC] Support references to multiple DDicts
2021-01-08 16:49:45 -05:00
Yann Collet
33b73db33c
Merge pull request #2457 from facebook/cli-dll
zstd CLI can now be linked to libzstd dynamic library
2021-01-07 17:10:13 -08:00
Yann Collet
c416015ab5
Merge pull request #2459 from ThomasWaldmann/fix-typos
fix typos (work done by Andrea Gelmini)
2021-01-07 16:19:10 -08:00
senhuang42
9ae0dd9336 Fix Visual and staticanalyze warnings 2021-01-07 17:58:37 -05:00
Thomas Waldmann
92a2b5ccc9 fixup: lits means literals 2021-01-07 23:30:42 +01:00
Yann Collet
2901b5e675 Merge branch 'dev' into cli-dll 2021-01-07 10:24:09 -08:00
Thomas Waldmann
f9802d80a0 fix typos (work done by Andrea Gelmini) 2021-01-07 18:47:23 +01:00
senhuang42
c2c9b8a7ec Address comments, clean up interface/internals 2021-01-07 12:29:12 -05:00
senhuang42
22b7bff2bc Add unit test, improve documentation 2021-01-07 12:29:12 -05:00