sharpetronics/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-11-27 00:05:09 -05:00
Code Issues Projects Releases Wiki Activity

Merge pull request #2643 from facebook/workers32

Browse Source 
reduce ZSTDMT_NBWORKERS_MAX in 32-bit mode
This commit is contained in:
Yann Collet 2021-05-12 14:18:31 -07:00 committed by GitHub
commit 705a62b612
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 9 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG
View File

@ -13,6 +13,7 @@ api: Several experimental functions have been deprecated and will emit a compile
`ZSTD_initCStream_advanced()` `ZSTD_initCStream_advanced()`
`ZSTD_initCStream_usingCDict_advanced()` `ZSTD_initCStream_usingCDict_advanced()`
`ZSTD_resetCStream()` `ZSTD_resetCStream()`
api: ZSTDMT_NBWORKERS_MAX reduced to 64 for 32-bit environments (@Cyan4973)
perf: Significant speed improvements for middle compression levels (#2494, @senhuang42 @terrelln) perf: Significant speed improvements for middle compression levels (#2494, @senhuang42 @terrelln)
perf: Block splitter to improve compression ratio, enabled by default for high compression levels (#2447, @senhuang42) perf: Block splitter to improve compression ratio, enabled by default for high compression levels (#2447, @senhuang42)
perf: Decompression loop refactor, speed improvements on `clang` and for `--long` modes (#2614 #2630, @Cyan4973) perf: Decompression loop refactor, speed improvements on `clang` and for `--long` modes (#2614 #2630, @Cyan4973)

6
lib/compress/zstdmt_compress.h
View File

@ -32,10 +32,10 @@
/* === Constants === */ /* === Constants === */
#ifndef ZSTDMT_NBWORKERS_MAX /* can be modified at compile time */ #ifndef ZSTDMT_NBWORKERS_MAX /* a different value can be selected at compile time */
# define ZSTDMT_NBWORKERS_MAX 200 # define ZSTDMT_NBWORKERS_MAX ((sizeof(void*)==4) /*32-bit*/ ? 64 : 256)
#endif #endif
#ifndef ZSTDMT_JOBSIZE_MIN /* can be modified at compile time */ #ifndef ZSTDMT_JOBSIZE_MIN /* a different value can be selected at compile time */
# define ZSTDMT_JOBSIZE_MIN (512 KB) # define ZSTDMT_JOBSIZE_MIN (512 KB)
#endif #endif
#define ZSTDMT_JOBLOG_MAX (MEM_32bits() ? 29 : 30) #define ZSTDMT_JOBLOG_MAX (MEM_32bits() ? 29 : 30)

3
programs/README.md
View File

@ -224,7 +224,8 @@ Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVE
that `zstd` will use for compression, which by default is `1`. that `zstd` will use for compression, which by default is `1`.
This functionality only exists when `zstd` is compiled with multithread support. This functionality only exists when `zstd` is compiled with multithread support.
`0` means "use as many threads as detected cpu cores on local system". `0` means "use as many threads as detected cpu cores on local system".
The max # of threads is capped at: `ZSTDMT_NBWORKERS_MAX==200`. The max # of threads is capped at `ZSTDMT_NBWORKERS_MAX`,
which is either 64 in 32-bit mode, or 256 for 64-bit environments.
This functionality can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments. This functionality can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments.
One such scenario is `tar --zstd`. One such scenario is `tar --zstd`.

2
programs/zstd.1
View File

@ -105,7 +105,7 @@ Display information related to a zstd compressed file, such as size, ratio, and
\fB\-\-fast[=#]\fR: switch to ultra\-fast compression levels\. If \fB=#\fR is not present, it defaults to \fB1\fR\. The higher the value, the faster the compression speed, at the cost of some compression ratio\. This setting overwrites compression level if one was set previously\. Similarly, if a compression level is set after \fB\-\-fast\fR, it overrides it\. \fB\-\-fast[=#]\fR: switch to ultra\-fast compression levels\. If \fB=#\fR is not present, it defaults to \fB1\fR\. The higher the value, the faster the compression speed, at the cost of some compression ratio\. This setting overwrites compression level if one was set previously\. Similarly, if a compression level is set after \fB\-\-fast\fR, it overrides it\.
. .
.IP "\(bu" 4 .IP "\(bu" 4
\fB\-T#\fR, \fB\-\-threads=#\fR: Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX==200\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\. \fB\-T#\fR, \fB\-\-threads=#\fR: Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to \fBZSTDMT_NBWORKERS_MAX\fR, which is either 64 in 32\-bit mode, or 256 for 64\-bit environments\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\.
. .
.IP "\(bu" 4 .IP "\(bu" 4
\fB\-\-single\-thread\fR: Does not spawn a thread for compression, use a single thread for both I/O and compression\. In this mode, compression is serialized with I/O, which is slightly slower\. (This is different from \fB\-T1\fR, which spawns 1 compression thread in parallel of I/O)\. This mode is the only one available when multithread support is disabled\. Single\-thread mode features lower memory usage\. Final compressed result is slightly different from \fB\-T1\fR\. \fB\-\-single\-thread\fR: Does not spawn a thread for compression, use a single thread for both I/O and compression\. In this mode, compression is serialized with I/O, which is slightly slower\. (This is different from \fB\-T1\fR, which spawns 1 compression thread in parallel of I/O)\. This mode is the only one available when multithread support is disabled\. Single\-thread mode features lower memory usage\. Final compressed result is slightly different from \fB\-T1\fR\.

3
programs/zstd.1.md
View File

@ -115,7 +115,8 @@ the last one takes effect.
* `-T#`, `--threads=#`: * `-T#`, `--threads=#`:
Compress using `#` working threads (default: 1). Compress using `#` working threads (default: 1).
If `#` is 0, attempt to detect and use the number of physical CPU cores. If `#` is 0, attempt to detect and use the number of physical CPU cores.
In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX==200. In all cases, the nb of threads is capped to `ZSTDMT_NBWORKERS_MAX`,
which is either 64 in 32-bit mode, or 256 for 64-bit environments.
This modifier does nothing if `zstd` is compiled without multithread support. This modifier does nothing if `zstd` is compiled without multithread support.
* `--single-thread`: * `--single-thread`:
Does not spawn a thread for compression, use a single thread for both I/O and compression. Does not spawn a thread for compression, use a single thread for both I/O and compression.