ecf90ca24b
Reproduction steps: ``` make zstreamtest CC=clang CFLAGS="-O3 -g -fsanitize=memory -fsanitize-memory-track-origins" ./zstreamtest -vv -t4178 -i4178 -s4531 ``` How to get to the error in gdb (may be a more efficient way): * 2 breaks at zstd_compress.c:2418 -- in ZSTD_compressContinue_internal() * 2 breaks at zstd_compress.c:2276 -- in ZSTD_compressBlock_internal() * 1 break at zstd_compress.c:1547 Why the error occurred: When `zc->forceWindow == 1`, after calling `ZSTD_loadDictionaryContent()` we have `zc->loadedDictEnd == zc->nextToUpdate == 0`. But, we've really loaded up to `iend` into the dictionary. Then in `ZSTD_compressBlock_internal()` we see that `current > zc->nextToUpdate + 384`, so we load the last 192 bytes a second time. In this case the bytes we are loading are a block of all 0s, starting in the previous block. So when we are loading the last 192 bytes, we find a `match` in the future, 183 bytes beyond `ip`. Since the block is all 0s, the match extends to the end of the block. But in `ZSTD_count()` we only check that `pIn < pInLoopLimit`, but since `pMatch > pIn`, `pMatch` eventually points past the end of the buffer, causing the MSAN failure. The fix: The line changed sets sets `zc->nextToUpdate` to the end of the dictionary. This is the behavior that existed before `ZSTD_p_forceWindow` was introduced. This fixes the exposing test case. Since the code doesn't fail without `zc->forceWindow`, it makes sense that this works. I've run the command `./zstreamtest -T2mn` 64 times without failures. CI should also verify nothing obvious broke.
Zstandard library files
The lib directory contains several directories. Depending on target use case, it's enough to include only files from relevant directories.
API
Zstandard's stable API is exposed within zstd.h,
at the root of lib directory.
Advanced API
Some additional API may be useful if you're looking into advanced features :
- common/error_public.h : transforms
size_tfunction results into anenum, for precise error handling. - ZSTD_STATIC_LINKING_ONLY : if you define this macro before including
zstd.h, it will give access to advanced and experimental API. These APIs shall never be used with dynamic library ! They are not "stable", their definition may change in the future. Only static linking is allowed.
Modular build
Directory common/ is required in all circumstances.
You can select to support compression only, by just adding files from the compress/ directory,
In a similar way, you can build a decompressor-only library with the decompress/ directory.
Other optional functionalities provided are :
-
dictBuilder/: source files to create dictionaries. The API can be consulted indictBuilder/zdict.h. This module also depends oncommon/andcompress/. -
legacy/: source code to decompress previous versions of zstd, starting fromv0.1. This module also depends oncommon/anddecompress/. Library compilation must include directiveZSTD_LEGACY_SUPPORT = 1. The main API can be consulted inlegacy/zstd_legacy.h. Advanced API from each version can be found in their relevant header file. For example, advanced API for versionv0.4is inlegacy/zstd_v04.h.
Using MinGW+MSYS to create DLL
DLL can be created using MinGW+MSYS with the make libzstd command.
This command creates dll\libzstd.dll and the import library dll\libzstd.lib.
The import library is only required with Visual C++.
The header file zstd.h and the dynamic library dll\libzstd.dll are required to
compile a project using gcc/MinGW.
The dynamic library has to be added to linking options.
It means that if a project that uses ZSTD consists of a single test-dll.c
file it should be linked with dll\libzstd.dll. For example:
gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll
The compiled executable will require ZSTD DLL which is available at dll\libzstd.dll.
Obsolete streaming API
Streaming is now provided within zstd.h.
Older streaming API is still available within deprecated/zbuff.h.
It will be removed in a future version.
Consider migrating code towards newer streaming API in zstd.h.
Miscellaneous
The other files are not source code. There are :
- LICENSE : contains the BSD license text
- Makefile : script to compile or install zstd library (static and dynamic)
- libzstd.pc.in : for pkg-config (
make install) - README.md : this file