11236 Commits

Author SHA1 Message Date
李子建
d95123f2e6 Improve speed of ZSTD_compressSequencesAndLiterals() using RVV 2025-06-02 17:21:02 +08:00
Dave Vasilevsky
f9938c217d lz4: Remove ancient test helpers
Building lz4 as root was causing `make clean` to fail with permission
errors.

We used to have to install lz4 from source back in Ubuntu 14.04, but
nowadays the installed lz4 is fine. Get rid of ancient helpers and
cruft!
2025-05-07 22:01:49 -07:00
Dave Vasilevsky
448a09ff78 seekable_format: Fix conversion warnings in parallel_compression 2025-05-07 22:01:49 -07:00
Dave Vasilevsky
13cb7a10ae seekable_format: Add test for parallel_compression memory usage
Use ulimit to fail the test if we use O(filesize) memory, rather than
O(threads).
2025-05-07 22:01:49 -07:00
Dave Vasilevsky
01c973de8d seekable_format: Fix race in parallel_processing
There was no memory barrier between writing and reading `done`, which
would allow reordering to cause races. With so little data to handle
after each job completes, we might as well just join.
2025-05-07 22:01:49 -07:00
Dave Vasilevsky
6fc8455a72 seekable_format: Cleanup POOL in parallel_compression 2025-05-07 22:01:49 -07:00
Dave Vasilevsky
2d4cff69c4 seekable_format: Make parallel_compression use memory properly
Previously, parallel_compression would only handle each job's results
after ALL jobs were successfully queued. This caused all src/dst
buffers to remain in memory until then!

It also polled to check whether a job completed, which is racy without
any memory barrier.

Now, we flush results as a side effect of completing a job. Completed
frames are placed in an ordered linked-list, and any eligible frames
are flushed. This may be zero or multiple frames, depending on the
order in which jobs finish.

This design also makes it simple to support streaming input, so that
is now available. Just pass `-` as the filename, and stdin/stdout will
be used for I/O.
2025-05-07 22:01:49 -07:00
Dave Vasilevsky
f5b6531902 seekable_format: Link against multi-threaded libzstd.a
Some of these examples are intended to be parallel, and don't make
sense to link against single-threaded libzstd.

The filename of mt and nomt libzstd are identical, so it's still
possible to link against the single-threaded one, just harder.
2025-05-07 22:01:49 -07:00
Dave Vasilevsky
6b0039abcf seekable_format: Build with $(MAKE)
This passes make flags, such as `-jN` for building in parallel, to
the underlying make.
2025-05-07 22:01:49 -07:00
Thomas Devoogdt
769723aee2 [cmake] only require a CXX compiler when tests are build
Signed-off-by: Thomas Devoogdt <thomas@devoogdt.com>
2025-05-06 12:20:26 -07:00
Nobuhiro Iwamatsu
2d224dc745 Add License variable to pkg-config file
The pkg-config file has License variable that allows you to set the license for
the software. This sets 'BSD-3-Clause OR GPL-2.0-only' to License.

Ref: https://github.com/pkgconf/pkgconf/blob/master/man/pc.5#L116
Signed-off-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>
2025-05-06 12:16:28 -07:00
Yann Collet
e26dde3dcf
Merge pull request #4378 from Cyan4973/fix_zstreamtest_minor
fix minor warning in zstreamtest
2025-04-29 14:01:14 -07:00
Yann Collet
453e252015
Merge pull request #4371 from ecordonnier-sc/eco/fix-macos-duplicate-lc-rpath
Fix duplicate LC_RPATH error on MacOS
2025-04-29 14:00:41 -07:00
Yann Collet
7f907d5c23 fix minor warning in zstreamtest 2025-04-29 09:56:00 -07:00
Etienne Cordonnier
8929d3b09f Fix duplicate LC_RPATH error on MacOS
After the update to MacOS 15.4, the dynamic loader dyld treats duplicated LC_RPATH as an error.
The `FLAGS` variable already contains `LDFLAGS`, thus using both `FLAGS` and `LDFLAGS`
duplicates all `LDFLAGS`, including `-Wl,rpath` parameters.

The duplicate LC_RPATH causes this kind of errors:

```
dyld[29361]: Library not loaded: @loader_path/../lib/libzstd.1.dylib
      Referenced from: <7131C877-3CF0-33AC-AA05-257BA4FDD770> /Users/foobar/...
      Reason: tried: '/Users/foobar/..../lib/libzstd.1.dylib' (duplicate LC_RPATH '/usr/mypath.../lib')
```

Closes https://github.com/facebook/zstd/issues/4369

Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
2025-04-18 15:59:06 +02:00
Yann Collet
d654fca786
Merge pull request #4362 from facebook/dependabot/github_actions/actions/setup-java-4.7.1
Bump actions/setup-java from 4.7.0 to 4.7.1
2025-04-14 03:19:07 -07:00
dependabot[bot]
28ffb100ab
Bump actions/setup-java from 4.7.0 to 4.7.1
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](3a4f6e1af5...c5195efecf)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 4.7.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-04-14 05:21:33 +00:00
Yann Collet
f5d4da09f5
Merge pull request #4353 from afq984/sparse
Check regular file for sparse support after opening
2025-04-07 03:48:41 -07:00
Li-Yu Yu
2a12bbaf90 Update cli-tests/file-stat tests 2025-03-28 22:46:26 +00:00
Li-Yu Yu
3bd5aa3404 Check regular file for sparse support after opening
A regular file may be created by the open call.
Checking after opening allows sparseFileSupport even
if dstFileName does not already exist.
2025-03-28 21:23:10 +00:00
Yann Collet
c5926fbab8
Merge pull request #4349 from Cyan4973/devfd
Support process substitution for `--filelist=`
2025-03-28 10:35:45 -07:00
Yann Collet
2f9627863f update error message 2025-03-28 10:02:37 -07:00
Yann Collet
0bdeb1d204 fix test 2025-03-28 09:18:17 -07:00
Yann Collet
8626da73b6 add error message clarification for inputs as process substitution 2025-03-28 09:18:17 -07:00
Yann Collet
a293cdcb85 added CI test 2025-03-28 09:18:17 -07:00
Yann Collet
0bde39a9b3 visual studio fix 2025-03-28 09:18:17 -07:00
Yann Collet
94cfa0b5a0 minor: restore some code comment 2025-03-28 09:18:17 -07:00
Yann Collet
76c2fdc7b7 better naming
and more narrow scope of local variables
2025-03-28 09:18:17 -07:00
Yann Collet
7630870b47 second implementation, better structured for improved maintenance 2025-03-28 09:18:17 -07:00
Yann Collet
165e52ce62 first implementation supporting Process Substitution 2025-03-28 09:18:17 -07:00
Yann Collet
7f0519dcfd
Merge pull request #4346 from Cyan4973/wintime
fix a risk of overflow on a time counter on Windows
2025-03-27 21:07:57 -07:00
Yann Collet
23e8812f5b
Merge pull request #4350 from Cyan4973/patchapply
add --patch-apply command
2025-03-27 09:57:20 -07:00
Yann Collet
eb168a0afc add --patch-apply command
as an equivalent for `-d --patch-from`.
Requested by @sergeevabc in #2173.
2025-03-25 14:50:39 -07:00
Yann Collet
12ea5f6e30
Merge pull request #4314 from facebook/dependabot/github_actions/ossf/scorecard-action-2.4.1
Bump ossf/scorecard-action from 2.4.0 to 2.4.1
2025-03-24 18:33:51 -07:00
Yann Collet
ec40252066
Merge pull request #4345 from Cyan4973/block128
[doc] minor clarification for maximum block size
2025-03-24 18:33:01 -07:00
Yann Collet
64dc08f47f
Merge pull request #4344 from Cyan4973/assert1
add an assert
2025-03-24 18:32:37 -07:00
Yann Collet
334af72280 update zstd --fast benchmark result
use --fast=4, instead of --fast=3,
to make it more representative of a lzo equivalent scenario.
2025-03-23 12:06:58 -07:00
Yann Collet
c59f7ac51e update benchmark result
brotli uses -1 level, like all other compressors
2025-03-23 12:02:51 -07:00
Yann Collet
33f4d40d1e fix a risk of overflow on a time counter on Windows
closes #4126
2025-03-23 11:42:41 -07:00
Yann Collet
bea1e942f7 [doc] minor clarification for maximum block size
closes #4339
2025-03-22 23:52:45 -07:00
Yann Collet
2fec3989c1 add an assert
to help static analyzers understand there is no overflow risk there.
2025-03-22 18:23:31 -07:00
Yann Collet
6775af5b2e
Merge pull request #4341 from clan/pragma
lib/zstd.h: don't put pragma after static
2025-03-20 19:49:08 -07:00
Z. Liu
cd8ca9d92e lib/zstd.h: move pragma before static
otherwise will cause dev-python/zstandard build failed when compiling with
clang as reported at https://bugs.gentoo.org/950259

the root cause is pycparser, which is unfixed since reported 2.5 years
ago, :(

Signed-off-by: Z. Liu <zhixu.liu@gmail.com>
2025-03-20 03:40:42 +00:00
Yann Collet
eca205fc78
Merge pull request #4336 from Cyan4973/qsort_v2
fix #4312 - musl compilation compatibility
2025-03-12 10:31:54 -07:00
Yann Collet
4d53e27144 removed OpenBSD specificity 2025-03-12 09:55:14 -07:00
Yann Collet
ddcb41a282 updated documentation 2025-03-11 14:10:35 -07:00
Yann Collet
a9b8fef2e8 add support for C11 Annex K qsort_s()
standard defined re-entrant variant of qsort().
Unfortunately, Annex K is optional.
2025-03-11 14:10:35 -07:00
Yann Collet
dcf675886b re-design qsort() selection in cover
centralizes auto detection tests,
then distribute the outcome in all the places where it's active.
2025-03-11 14:10:35 -07:00
Yann Collet
51b6e79f65 fix #4312
and upgraded the test so that it would fail, both at compile time and at run time, without the fix
2025-03-11 14:10:35 -07:00
Yann Collet
9a57bdc0bd attempt to reduce length of long cli tests by invoking -T0 2025-03-11 14:10:35 -07:00