Yann Collet
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Yann Collet
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
Yann Collet
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
Yann Collet
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Yann Collet
49af41820d
clarified status of zstdmt_compress.h API
2017-07-05 17:21:13 -07:00
Yann Collet
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
Yann Collet
2cb9774f5e
more precise estimation of amount to flush at end of stream (single thread mode)
...
also : can use DEBUGLEVEL variable in /tests
2017-07-04 12:39:26 -07:00
Yann Collet
6383372dec
fixed : 0-copy in NULL is UB
2017-07-04 10:36:41 -07:00
Yann Collet
2de2396a36
refactor ZSTDMT_compress()
2017-07-03 16:23:36 -07:00
Yann Collet
2084b041f4
fixed comments
2017-07-03 15:52:19 -07:00
Yann Collet
5a77361595
fixed wrong function name in comment
2017-07-03 15:21:24 -07:00
cyan4973
1bafe393e4
fix : ZSTDMT_compressStream_generic() can accept NULL input
...
also : converge implementations towards new version of ZSTDMT_compressStream_generic()
2017-07-01 06:59:24 -07:00
Yann Collet
58bd0e70fc
fixed : dictionary compression with new advanced API in Multi-threading mode
2017-06-30 16:01:02 -07:00
Yann Collet
d8b33a598d
Optimized ZSTDMT single-pass mode speed on large sources
...
by ensuring job sizes remain "not too large"
2017-06-30 15:44:57 -07:00
Yann Collet
d5c046c609
implemented shortcut for zstd_compress_generic() in MT mode
...
added ZSTDMT_compress_advanced() API
2017-06-30 14:51:01 -07:00
Yann Collet
a3d9926c40
compression optimization opportunity
...
switch to single-pass mode directly into output buffer
when outputSize >= ZSTD_compressBound(inputSize).
Speed gains observed with fullbench (~+15% on level 1)
2017-06-29 14:44:49 -07:00
Yann Collet
037466245f
refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget()
...
use less macro statements
the initial version was meant to work with STATIC_ASSERT
but since it doesn't work and needs assert()
it's possible to rewrite it using normally compiled code
which is better for compiler.
Downside : the error message is less precise.
There is a DEBUGLOG(3,) to compensate.
2017-06-28 20:24:08 -07:00
Yann Collet
2bf428df45
Merge branch 'advancedAPI2' into refPrefix
2017-06-28 16:35:49 -07:00
Yann Collet
1ca76039af
fixed -Wdeclaration-after-statement
2017-06-28 15:40:21 -07:00
Yann Collet
813535105b
added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0]
...
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
2017-06-28 15:34:56 -07:00
Yann Collet
adbe74a8ac
adjusted compression levels to guarantee a monotonically increasing memory budget
2017-06-28 13:22:37 -07:00
Yann Collet
33a6639039
fixed ZSTD_refPrefix with Multithread-enabled CCtx
2017-06-28 11:09:43 -07:00
Yann Collet
2e4274262d
controlled dictMode
2017-06-27 17:09:12 -07:00
Yann Collet
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
Yann Collet
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
Yann Collet
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
Yann Collet
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
Yann Collet
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
Yann Collet
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
Yann Collet
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
Yann Collet
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
Yann Collet
ecb0f46866
add controls over streaming buffers
2017-06-21 17:25:01 -07:00
Yann Collet
204b6b7ef6
fixed streaming buffered allocation with CDict compression
2017-06-21 15:13:00 -07:00
Yann Collet
83095970e6
free cdictLocal faster, suggested by @terrelln
2017-06-21 12:26:40 -07:00
Yann Collet
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
Yann Collet
e51d51bdf7
fixed memcpy() overlap
2017-06-20 17:44:55 -07:00
Yann Collet
466f92eaa6
removed one useless streaming compression stage, detected by @terrelln
2017-06-20 16:25:29 -07:00
Yann Collet
c3bce24ef4
fixed potential dangling pointer, detected by @terrelln
2017-06-20 16:09:11 -07:00
Yann Collet
b44ab82f7a
ensure new ZSTD_strategy starts at value 1
2017-06-20 14:11:49 -07:00
Yann Collet
c08e649e95
first implementation of bench.c with new API ZSTD_compress_generic()
...
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.
Also : fixed a long standing bug in ZSTDMT streaming API.
2017-06-19 18:25:35 -07:00
Nick Terrell
55f9cd4942
[libzstd] Fix UBSAN failure
2017-06-19 15:12:28 -07:00
Yann Collet
bf99150be3
update new api presentation in zstd.h and manual
2017-06-19 12:56:25 -07:00
Yann Collet
d7a3bffba9
new api : setting compression parameters is refused if a dictionary is already loaded
2017-06-19 11:53:01 -07:00
Yann Collet
381e66cfbd
added ZSTD_clampCParams()
...
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
2017-06-16 17:34:54 -07:00
Yann Collet
aee916e37c
fixed +/-1 error for pledgedSrcSizePlusOne
2017-06-16 17:02:35 -07:00
Yann Collet
d3de3d51a3
fix attempt 2 : Visual sign conversion warning
2017-06-16 16:51:33 -07:00
Yann Collet
944be54774
fixed attempt : minor Visual sign conversion warning
2017-06-16 14:05:01 -07:00
Yann Collet
b26728c9c8
added ZSTD_startNewCompression()
2017-06-16 14:00:46 -07:00
Yann Collet
a0ba849fe6
changed frameContentSize field to pledgedSrcSizePlusOne
...
pledgedSrcSize is proper : it's a promise, not yet fulfilled.
It will be controlled at the end.
PlusOne is meant to have 0 (default) == unknown
2017-06-16 13:29:17 -07:00
Yann Collet
2cf7755da7
fix : pledgedSrcSize correctly reset to unknown in "continue" mode
2017-06-16 12:34:41 -07:00
