54 Commits

Author SHA1 Message Date
Yonatan Komornik
c78f434aa4
Fix zstd-dll build missing dependencies (#3496)
* Fixes zstd-dll build (https://github.com/facebook/zstd/issues/3492):
- Adds pool.o and threading.o dependency to the zstd-dll target
- Moves custom allocation functions into header to avoid needing to add dependency on common.o
- Adds test target for zstd-dll
- Adds github workflow that buildis zstd-dll
2023-02-12 12:32:31 -08:00
Yann Collet
ea684c335a added c89 build test to CI 2023-01-19 14:59:30 -08:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
Yonatan Komornik
500f02eb66 Fixes two bugs in the Windows thread / pthread translation layer
1. If threads are resized the threads' `ZSTD_pthread_t` might move
while the worker still holds a pointer into it (see more details in #3120).
2. The join operation was waiting for a thread and then return its `thread.arg`
as a return value, but since the `ZSTD_pthread_t thread` was passed by value it
would have a stale `arg` that wouldn't match the thread's actual return value.

This fix changes the `ZSTD_pthread_join` API and removes support for returning
a value. This means that we are diverging from the `pthread_join` API and this
is no longer just an alias.
In the future, if needed, we could return a Windows thread's return value using
`GetExitCodeThread`, but as this path wouldn't be excised in any case, it's
preferable to not add it right now.
2022-12-17 13:38:02 -08:00
Yann Collet
b7d55cfa0d fix issue #3119
fix segfault error when running zstreamtest with MALLOC_PERTURB_
2022-10-12 23:04:23 -07:00
Yonatan Komornik
1598e6c634
Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
Yann Collet
6211bfee5e fixed backup prototype for POOL_sizeof() 2021-12-30 14:33:21 -08:00
Yann Collet
b1978d60ee POOL_sizeof() only needs a const read-only reference 2021-12-30 14:08:51 -08:00
Nick Terrell
a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
Nick Terrell
66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
Martin Liska
b684900a4a Allow external creation of POOLs that can be shared. 2020-10-07 12:44:33 +02:00
Nick Terrell
c465f24457 ZSTD_ prefix mem{cpy,move,set},malloc,calloc,free 2020-08-26 12:26:03 -07:00
Nick Terrell
a686d306d2 Rename ZSTD_{malloc,calloc,free} to ZSTD_custom{Malloc,Calloc,Free} 2020-08-26 12:25:08 -07:00
Nick Terrell
80f577baa2 Move standard includes to zstd_deps.h 2020-08-26 12:25:08 -07:00
yoshihitoh
c6548eac8e Rename static vars to avoid redefinition error. 2020-06-29 10:51:50 +09:00
Nick Terrell
ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
Nick Terrell
243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Nick Terrell
a8daa2d683 Signal before unlocking in pool.c 2018-11-08 10:45:53 -08:00
Yann Collet
fbd5dfc1b1 changed POOL_resize() return type to int
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
2018-06-22 12:14:59 -07:00
Yann Collet
243cd9d8bb add a cond_broadcast after resize
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
2018-06-21 18:04:58 -07:00
Yann Collet
818e72b4d5 added extended POOL test
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
2018-06-21 14:58:59 -07:00
Yann Collet
6de249c1c6 fixed: bug when counting nb of active threads
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
2018-06-20 18:28:49 -07:00
Yann Collet
6b48eb12c0 change control of threadLimit
now limits maximum nb of active threads
even when queueSize > 1.
2018-06-20 14:35:39 -07:00
Yann Collet
62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet
166901dc72 reduced POOL_resize() restriction
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
2018-06-19 18:07:18 -07:00
Yann Collet
4567c57199 finalized POOL_resize()
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)

The function may fail, and returns a NULL pointer in this case.
2018-06-19 16:03:12 -07:00
Yann Collet
1c714fda3f introduced POOL_resize()
not complete yet :
finalize behavior in case of unfinished expansion
2018-06-18 20:46:39 -07:00
Yann Collet
997e4d0ccd added POOL_tryAdd() 2018-01-18 14:39:51 -08:00
Yann Collet
61e5a1adfc removed direct call to malloc() from pool.c 2017-10-31 17:43:24 -07:00
Yann Collet
86b4fe5b45 adjustCParams : restored previous behavior
unknowns srcSize presumed small if there is a dictionary (dictSize>0)
and presumed large otherwise.
2017-09-28 18:14:28 -07:00
Nick Terrell
6c41adfb28 [libzstd] pthread function prefixed with ZSTD_
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
2017-09-27 11:48:48 -07:00
Yann Collet
3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Nick Terrell
423b133568 [POOL] Allow free on NULL when multithreading is disabled 2017-09-05 11:18:13 -07:00
Nick Terrell
02033be08c [pool] Visual Studios disallows empty structs 2017-08-28 17:19:01 -07:00
Nick Terrell
26dc040a7b [pool] Accept custom allocators 2017-08-24 17:01:41 -07:00
Nick Terrell
89dc856cae [pool] Fix formatting 2017-08-24 16:48:32 -07:00
Yann Collet
7db552676e reduced pool queue to 0 to save memory
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
Yann Collet
32fb407c9d updated a bunch of headers
for the new license
2017-08-18 16:52:05 -07:00
Stella Lau
73ba58955f Signal after finishing job when queueSize=0 2017-08-01 20:12:06 -07:00
Stella Lau
1d76da1d87 Replace marker with queueEmpty variable and update pool.h comment 2017-08-01 12:30:16 -07:00
Stella Lau
5adceeed01 Allow queueSize=0 in pool.c and update poolTests 2017-07-31 10:10:16 -07:00
Yann Collet
a90b16e150 Visual blind fix 2 2017-07-20 15:57:55 -07:00
Yann Collet
16261e6951 buffer pool can be invoked from multiple threads 2017-07-11 14:14:07 -07:00
Yann Collet
6056e4c3eb added POOL_sizeof() for single-thread 2017-06-02 11:36:47 -07:00
Yann Collet
c4a5a21c5c created ZSTDMT_sizeof_CCtx() and POOL_sizeof()
required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*
2017-06-01 17:56:14 -07:00
Nick Terrell
e628eaf87a Fix pool.c threading.h import 2017-01-26 15:29:10 -08:00
cyan4973
2e3b659ae1 fixed minor warnings (Visual, conversion, doxygen) 2017-01-20 14:43:09 -08:00
Yann Collet
0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00
Nick Terrell
bb13387d7d Fix pool for threading.h 2016-12-31 19:10:47 -05:00