9912 Commits

Author SHA1 Message Date
Danielle Rozenblit
2def93bf10 Merge branch 'optimal-huff-depth-speed' of github.com:daniellerozenblit/zstd into optimal-huff-depth-speed 2022-12-20 12:28:07 -08:00
Danielle Rozenblit
482689b995 huf log speed optimization: unidirectional scan of logs + break when regressing 2022-12-20 12:27:38 -08:00
W. Felix Handte
e3f2c8b11c Pin actions/checkout Dependency to Specific Commit Hash
It's a bit silly, because if we can't trust GitHub, what are we doing here?
But OSSF complains about it, so let's fix it.
2022-12-20 14:53:12 -05:00
Felix Handte
ea199876e2
Merge pull request #3378 from facebook/dependabot/github_actions/github/codeql-action-2.1.37
Bump github/codeql-action from 1.0.26 to 2.1.37
2022-12-20 14:17:33 -05:00
Felix Handte
5f0543a508
Merge pull request #3377 from facebook/dependabot/github_actions/ossf/scorecard-action-2.1.0
Bump ossf/scorecard-action from 2.0.6 to 2.1.0
2022-12-20 14:16:10 -05:00
Felix Handte
3a7e2eec21
Merge pull request #3340 from facebook/dependabot/github_actions/actions/upload-artifact-3
Bump actions/upload-artifact from 1 to 3
2022-12-20 14:15:43 -05:00
Nick Terrell
e4018c4e7f [docs] Clarify dictionary loading documentation
Reinforce that loading a new dictionary clears the current dictionary.
Except for the multiple-ddict mode.
2022-12-20 11:10:49 -08:00
Felix Handte
f302ad8811
Merge pull request #3173 from felixhandte/update-copyright-company
Update Copyright Comments
2022-12-20 13:51:01 -05:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
2022-12-20 12:44:56 -05:00
W. Felix Handte
4c999cf968 Update test-license.py 2022-12-20 12:43:16 -05:00
W. Felix Handte
36d5c2f326 Update Copyright Year ('2021' -> 'present')
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f);
do
  sed -i 's/\-2021/-present/' $f;
done

g co HEAD -- .github/workflows/dev-short-tests.yml # fix bad match
```
2022-12-20 12:42:50 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
W. Felix Handte
a05cca6208 Manually Update VS Code Copyright Definitions 2022-12-20 12:36:06 -05:00
Yann Collet
6a90c0f6fb
Merge pull request #3289 from nmoinvaz/cmake/playtest-win
Don't attempt playTests.sh cmake test if running on Windows.
2022-12-19 16:39:38 -08:00
Yonatan Komornik
a8add436ce
Merge pull request #3364 from yoniko/fix-windows-mt-thread-resize-bug
Windows MT layer bug fixes
2022-12-19 15:54:01 -08:00
Yonatan Komornik
26f1bf7d70 CR fixes 2022-12-19 15:13:43 -08:00
Nick Terrell
e9797b5dc5 [pzstd] Fixes for Windows build
* Add `Portability.h` to fix min/max issues.
* Fix conversion warnings
* Assert that windowLog <= 23, which is currently always the case.
  This could be loosened, but we aren't looking to add new functionality.

Fixes on top of PR #3375 by @eli-schwartz, which added Windows CI for contrib & programs.
2022-12-19 14:09:43 -08:00
Eli Schwartz
67cd24b25b CI: build contrib directory on meson-windows 2022-12-19 14:09:43 -08:00
Eli Schwartz
3cee69a1cc CI: build programs on meson-windows too 2022-12-19 14:09:43 -08:00
Yann Collet
9073fe00cc
Merge pull request #3376 from facebook/split2
Block splitter : minor reformatting
2022-12-19 13:00:27 -08:00
dependabot[bot]
79a00f8dcf
Bump github/codeql-action from 1.0.26 to 2.1.37
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 1.0.26 to 2.1.37.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](5f53256358...959cbb7472)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 05:07:15 +00:00
dependabot[bot]
65545969d5
Bump ossf/scorecard-action from 2.0.6 to 2.1.0
Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.0.6 to 2.1.0.
- [Release notes](https://github.com/ossf/scorecard-action/releases)
- [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
- [Commits](99c53751e0...937ffa90d7)

---
updated-dependencies:
- dependency-name: ossf/scorecard-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 05:07:09 +00:00
Yann Collet
832c1a6a1c minor reformatting
and minor reliability and maintenance changes
2022-12-18 11:26:57 -08:00
Yonatan Komornik
aaa38b29bb meson: zstreamtests should now pass on Windows 2022-12-17 13:38:02 -08:00
Yonatan Komornik
ec42c92aaa Fix race condition in the Windows thread / pthread translation layer
When spawning a Windows thread we have small worker wrapper function that translates
between the interfaces of Windows and POSIX threads.
This wrapper is given a pointer that might get stale before the worker starts running,
resulting in UB and crashes.
This commit adds synchronization so that we know the wrapper has finished reading the data
it needs before we allow the main thread to resume execution.
2022-12-17 13:38:02 -08:00
Yonatan Komornik
500f02eb66 Fixes two bugs in the Windows thread / pthread translation layer
1. If threads are resized the threads' `ZSTD_pthread_t` might move
while the worker still holds a pointer into it (see more details in #3120).
2. The join operation was waiting for a thread and then return its `thread.arg`
as a return value, but since the `ZSTD_pthread_t thread` was passed by value it
would have a stale `arg` that wouldn't match the thread's actual return value.

This fix changes the `ZSTD_pthread_join` API and removes support for returning
a value. This means that we are diverging from the `pthread_join` API and this
is no longer just an alias.
In the future, if needed, we could return a Windows thread's return value using
`GetExitCodeThread`, but as this path wouldn't be excised in any case, it's
preferable to not add it right now.
2022-12-17 13:38:02 -08:00
Yonatan Komornik
fba704f8bb
Merge pull request #3374 from yoniko/fix-meson-ci-dependancy
Fix Meson-Windows CI test by pulling a more recent version msvc-dev-cmd
2022-12-17 13:36:56 -08:00
Yonatan Komornik
3867c41552 meson: Fix Windows CI test by pulling a more recent version msvc-dev-cmd 2022-12-17 10:33:49 -08:00
Yonatan Komornik
ce61cb836d
Merge pull request #3120 from eli-schwartz/meson-fixup
Meson test fixups
2022-12-16 18:28:54 -08:00
Yann Collet
3a484efb3e
Merge pull request #3373 from facebook/decompressBound
decompressBound tests and fix
2022-12-16 16:55:30 -08:00
Yann Collet
2f4238e47a make ZSTD_DECOMPRESSBOUND() compatible with input size 0
for environments with stringent compilation warnings.
2022-12-16 16:05:39 -08:00
Yann Collet
ea24b88667 decompressBound() tests
fixed an overflow in an intermediate result on 32-bit platform.
Checked that the new test catch this bug in 32-bit mode.
2022-12-16 15:43:26 -08:00
Yann Collet
51355e1f70
Merge pull request #3362 from facebook/compressBound
check potential overflow of compressBound()
2022-12-16 14:22:22 -08:00
Nick Terrell
2f7b8d47fb [zdict] Fix static linking only include guards
Fix `zdict.h` static linking only section so if you include it twice it
still exposes the static linking only symbols. E.g. this pattern:

```
```

This can easily happen when a header you include includes `zdict.h`.
2022-12-16 12:55:20 -08:00
Nick Terrell
0c42424a1e [build] Fix ZSTD_LIB_MINIFY build option
`ZSTD_LIB_MINIFY` broke in 8bf699aa59372d7c2bb4216bcf8037cab7dae51e.

This commit fixes the macro and the static library shrinks from ~600K to 324K
with ZSTD_LIB_MINIFY set.

Fixes #3066.
2022-12-16 12:55:05 -08:00
Nick Terrell
358a237484 [api][visibility] Make the visibility macros more consistent
1. Follow the scheme introduced in PR #2501 for both `zdict.h` and `zstd_errors.h`.
2. If the `*_VISIBLE` macro isn't set, but the `*_VISIBILITY` macro is, use that.
   Also make this change for `zstd.h`, since we probably shouldn't have changed
   that macro name without backward compatibility in the first place.
3. Change all references to `*_VISIBILITY` to `*_VISIBLE`.

Fixes #3359.
2022-12-16 12:54:45 -08:00
Yann Collet
97f63ce2b5 added unit tests for compressBound()
and rephrased the code documentation, as suggested by @terrelln
2022-12-16 12:35:14 -08:00
Nick Terrell
58508398f4 [CI] Re-enable versions-test
It seems like with the deletion of Travis CI we didn't successfully transfer the
version compatibility test. Attempt to enable the version compatibility test.
2022-12-16 12:04:21 -08:00
Eli Schwartz
6747ba4ef5
meson: mark a known test failure on Windows 2022-12-16 14:35:31 -05:00
Eli Schwartz
937e9d3b62
meson: add Windows CI
There are a couple of oddities here. We don't attempt to build e.g.
contrib, because that doesn't seem to work at the moment. Also notice
that each command is its own step. This happens because github actions
runs in powershell, which doesn't seem to let you abort on the first
failure.
2022-12-16 14:35:22 -05:00
Eli Schwartz
7f29c1847d
meson: add Linux CI
Travis is no longer run, but this wasn't ported to something else.
2022-12-16 14:27:47 -05:00
Eli Schwartz
9c3e18f7fe
meson: add support for running both fast and slow version of tests
playTests.sh has an option to run really slow tests. This is enabled by
default in Meson, but what we really want is to do like the Makefile,
and run the fast ones by default, but with an option to run the slow
ones instead.
2022-12-16 14:27:46 -05:00
Eli Schwartz
26134b4565
meson: don't require valgrind tests
It's entirely possible some people don't have valgrind installed, but
still want to run the tests. If they don't have it installed, then they
probably don't intend to run those precise test targets anyway.

Also, this solves an error when running the tests in an automated
environment. The valgrind tests have a hard dependency on behavior such
as `./zstd` erroring out with the message "stdin is a console, aborting"
which does not work if the automated environment doesn't have a console.
As a rough heuristic, automated environments lacking a console will
*probably* also not have valgrind, so avoiding that test definition
neatly sidesteps the issue.

Also, valgrind is not easily installable on macOS, at least homebrew
says it isn't available there. This makes it needlessly hard to
enable the testsuite on macOS.
2022-12-16 14:27:46 -05:00
Nick Terrell
ef566c8d68 [circleci] Try to re-enable aarch64build
Update CircleCI to focal and try to re-enable aarch64build.

Fixes #2785 if it works.
2022-12-16 10:21:28 -08:00
Yonatan Komornik
8745229612
Merge pull request #3368 from eli-schwartz/meson-execinfo-bool
meson: fix broken commit that broke the build
2022-12-16 09:14:53 -08:00
Nick Terrell
ee6475cbbd Add missing parens around macro definition
Fixes #3301.
2022-12-15 17:18:23 -08:00
Eli Schwartz
626425dce0
meson: fix warning for using too-new features
In commit 031de3c69ccbf3282ed02fb49369b476730aeca8 a feature of Meson
0.50.0 was added, but the minimum specified version of Meson is 0.48.0.
Meson therefore emitted a warning:

WARNING: Project targets '>=0.48.0' but uses feature introduced in '0.50.0': required arg in compiler.has_header.

And if anyone actually used Meson 0.48.0 to build with, it would error
out with mysterious claims that the build file itself is invalid, rather
than telling the user to install a newer version of Meson.

Solve this by bumping the minimum version to align with reality. This
e.g. drops support for Debian oldstable (buster)'s packaged version of
Meson, but still works if backports are enabled, or if the user can
`pip install` a newer version.
2022-12-15 20:08:35 -05:00
Eli Schwartz
e8401e9e8d
meson: fix broken commit that broke the build
In commit 031de3c69ccbf3282ed02fb49369b476730aeca8 some code was added
that returned a boolean, but was treated as if it returned a dependency
object. This wasn't tested and could not work. Moreover, zstd no longer
built at all unless the entire programs directory was disabled and not
even evaluated.

Fix the return type checking.
2022-12-15 20:08:28 -05:00
Yann Collet
26e8c65aab Merge branch 'dev' of github.com:facebook/zstd into dev 2022-12-15 15:58:42 -08:00