mirror of https://github.com/facebook/zstd.git synced 2025-12-08 00:03:24 -05:00

History

Yann Collet 5235d8d6ba first implementation of delayed update for btlazy2

This is a pretty nice speed win.

The new strategy consists in stacking new candidates as if it was a hash chain.
Then, only if there is a need to actually consult the chain, they are batch-updated,
before starting the match search itself.
This is supposed to be beneficial when skipping positions,
which happens a lot when using lazy strategy.

The baseline performance for btlazy2 on my laptop is :
15#calgary.tar       :   3265536 ->    955985 (3.416),  7.06 MB/s , 618.0 MB/s
15#enwik7            :  10000000 ->   3067341 (3.260),  4.65 MB/s , 521.2 MB/s
15#silesia.tar       : 211984896 ->  58095131 (3.649),  6.20 MB/s , 682.4 MB/s
(only level 15 remains for btlazy2, as this strategy is squeezed between lazy2 and btopt)

After this patch, and keeping all parameters identical,
speed is increased by a pretty good margin (+30-50%),
but compression ratio suffers a bit :
15#calgary.tar       :   3265536 ->    958060 (3.408),  9.12 MB/s , 621.1 MB/s
15#enwik7            :  10000000 ->   3078318 (3.249),  6.37 MB/s , 525.1 MB/s
15#silesia.tar       : 211984896 ->  58444111 (3.627),  9.89 MB/s , 680.4 MB/s

That's because I kept `1<<searchLog` as a maximum number of candidates to update.
But for a hash chain, this represents the total number of candidates in the chain,
while for the binary, it represents the maximum depth of searches.
Keep in mind that a lot of candidates won't even be visited in the btree,
since they are filtered out by the binary sort.

As a consequence, in the new implementation,
the effective depth of the binary tree is substantially shorter.

To compensate, it's enough to increase `searchLog` value.
Here is the result after adding just +1 to searchLog (level 15 setting in this patch):
15#calgary.tar       :   3265536 ->    956311 (3.415),  8.32 MB/s , 611.4 MB/s
15#enwik7            :  10000000 ->   3067655 (3.260),  5.43 MB/s , 535.5 MB/s
15#silesia.tar       : 211984896 ->  58113144 (3.648),  8.35 MB/s , 679.3 MB/s

aka, almost the same compression ratio as before,
but with a noticeable speed increase (+20-30%).

This modification makes btlazy2 more competitive.
A new round of paramgrill will be necessary to determine which levels are impacted and could adopt the new strategy.

2017-12-28 16:58:57 +01:00

common

fix a subtle issue in continue mode

2017-12-19 09:43:03 +01:00

compress

first implementation of delayed update for btlazy2

2017-12-28 16:58:57 +01:00

decompress

re-added test case

2017-12-12 14:01:54 -08:00

deprecated

fixed zbufftest

2017-10-19 14:06:02 -07:00

dictBuilder

no longer supported starting C++17

2017-12-04 18:00:53 -08:00

dll

fixed a bunch of headers after license change (#825 )

2017-08-31 11:24:54 -07:00

legacy

no longer supported starting C++17

2017-12-04 18:00:53 -08:00

.gitignore

makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819 )

2017-09-11 14:09:34 -07:00

BUCK

Update BUCK files

2017-10-25 12:47:57 -07:00

libzstd.pc.in

updated pkg config file

2016-11-30 11:06:58 -08:00

Makefile

makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819 )

2017-09-11 14:09:34 -07:00

README.md

programs/Makefile : better support for GNU conventions

2017-09-06 16:53:59 -07:00

zstd.h

version bump to v1.3.3

2017-12-14 16:11:20 -08:00

README.md

Zstandard library files

The lib directory is split into several sub-directories, in order to make it easier to select or exclude specific features.

Building

Makefile script is provided, supporting the standard set of commands, directories, and variables (see https://www.gnu.org/prep/standards/html_node/Command-Variables.html).

make : generates both static and dynamic libraries
make install : install libraries in default system directories

API

Zstandard's stable API is exposed within lib/zstd.h.

Advanced API

Optional advanced features are exposed via :

lib/common/zstd_errors.h : translates size_t function results into an ZSTD_ErrorCode, for accurate error handling.
ZSTD_STATIC_LINKING_ONLY : if this macro is defined before including zstd.h, it unlocks access to advanced experimental API, exposed in second part of zstd.h. These APIs shall never be used with dynamic library ! They are not "stable", their definition may change in the future. Only static linking is allowed.

Modular build

Directory lib/common is always required, for all variants.
Compression source code lies in lib/compress
Decompression source code lies in lib/decompress
It's possible to include only compress or only decompress, they don't depend on each other.
lib/dictBuilder : makes it possible to generate dictionaries from a set of samples. The API is exposed in lib/dictBuilder/zdict.h. This module depends on both lib/common and lib/compress .
lib/legacy : source code to decompress older zstd formats, starting from v0.1. This module depends on lib/common and lib/decompress. To enable this feature, it's necessary to define ZSTD_LEGACY_SUPPORT = 1 during compilation. Typically, with gcc, add argument -DZSTD_LEGACY_SUPPORT=1. Using higher number limits the number of version supported. For example, ZSTD_LEGACY_SUPPORT=2 means : "support legacy formats starting from v0.2+". The API is exposed in lib/legacy/zstd_legacy.h. Each version also provides a (dedicated) set of advanced API. For example, advanced API for version v0.4 is exposed in lib/legacy/zstd_v04.h .

Multithreading support

Multithreading is disabled by default when building with make. Enabling multithreading requires 2 conditions :

set macro ZSTD_MULTITHREAD
on POSIX systems : compile with pthread (-pthread compilation flag for gcc for example)

Both conditions are automatically triggered by invoking make lib-mt target. Note that, when linking a POSIX program with a multithreaded version of libzstd, it's necessary to trigger -pthread flag during link stage.

Multithreading capabilities are exposed via :

private API lib/compress/zstdmt_compress.h. Symbols defined in this header are currently exposed in libzstd, hence usable. Note however that this API is planned to be locked and remain strictly internal in the future.
advanced API ZSTD_compress_generic(), defined in lib/zstd.h, experimental section. This API is still considered experimental, but is designed to be labelled "stable" at some point in the future. It's the recommended entry point for multi-threading operations.

Windows : using MinGW+MSYS to create DLL

DLL can be created using MinGW+MSYS with the make libzstd command. This command creates dll\libzstd.dll and the import library dll\libzstd.lib. The import library is only required with Visual C++. The header file zstd.h and the dynamic library dll\libzstd.dll are required to compile a project using gcc/MinGW. The dynamic library has to be added to linking options. It means that if a project that uses ZSTD consists of a single test-dll.c file it should be linked with dll\libzstd.dll. For example:

    gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll

The compiled executable will require ZSTD DLL which is available at dll\libzstd.dll.

Deprecated API

Obsolete API on their way out are stored in directory lib/deprecated. At this stage, it contains older streaming prototypes, in lib/deprecated/zbuff.h. Presence in this directory is temporary. These prototypes will be removed in some future version. Consider migrating code towards supported streaming API exposed in zstd.h.

Miscellaneous

The other files are not source code. There are :

LICENSE : contains the BSD license text
Makefile : make script to build and install zstd library (static and dynamic)
BUCK : support for buck build system (https://buckbuild.com/)
libzstd.pc.in : for pkg-config (used in make install)
README.md : this file