sharpetronics/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-10-04 00:02:33 -04:00
Code Issues Projects Releases Wiki Activity
9,436 Commits 38 Branches 77 Tags
Commit Graph

9436 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
f2d9652ad8 more usage of new error code stabilityCondition_notRespected 
as suggested by @terrelln
 2022-01-26 18:30:55 -08:00
Yann Collet
dda4c10f07 added ZSTD_compressStream2() + ZSTD_c_stableInBuffer test 2022-01-26 13:33:04 -08:00
Yann Collet
cbff372d10 added helper function inBuffer_forEndFlush() 2022-01-26 11:05:57 -08:00
Yann Collet
b99ece96b9 converted checks into user validation generating error codes 
had to create a new error code for this condition,
none of the existing ones were fitting enough.
 2022-01-26 10:43:50 -08:00
Yann Collet
af3d9c506e added streaming test starting from non-0 pos 2022-01-26 10:31:25 -08:00
Yann Collet
c1668a00d2 fix extended case combining stableInBuffer with continue() and flush() modes 2022-01-26 10:31:25 -08:00
Yann Collet
270f9bf005 better consistency in accessing @input 
as suggested by @terrelln.

Also : commented zstreamtest more
to ensure ZSTD_stableInBuffer is tested/
 2022-01-26 10:31:24 -08:00
Yann Collet
8296be4a0a pretend consuming input to provide a sense of forward progress 2022-01-26 10:31:24 -08:00
Yann Collet
4b9d1dd9ff fixed incorrect comment 2022-01-26 10:31:24 -08:00
Yann Collet
27d336b099 minor behavior refinements 
specifically, there is no obligation to start streaming compression with pos=0.
stableSrc mode is now compatible with this setup.
 2022-01-26 10:31:24 -08:00
Yann Collet
37b87add7a make stableSrc compatible with regular streaming API 
including flushStream().

Now the only condition is for `input.size` to continuously grow.
 2022-01-26 10:31:24 -08:00
Yann Collet
c0c5ffa973 streaming compression : lazy parameter adaptation with stable input 
effectively makes ZSTD_c_stableInput compatible ZSTD_compressStream()
and zstd_e_continue operation mode.
 2022-01-26 10:31:24 -08:00
Yann Collet
5684bae4f6 minor refactoring 
on streaming compression implementation.
 2022-01-26 10:31:23 -08:00
Yann Collet
cc7d23bcec
Merge pull request #2965 from facebook/offbase 
Converge sumtype (offset | repcode) numeric representation towards offBase
 2022-01-24 15:47:42 -08:00
Yonatan Komornik
70df5de1b2
AsyncIO compression part 1 - refactor of existing asyncio code (#3021) 
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h

* Bugfix - rename fileio_asycio to fileio_asyncio

* Added copyrights & license to new files

* CR fixes
 2022-01-24 14:43:02 -08:00
Nick Terrell
87f81d0796
Merge pull request #3026 from trixirt/from-linux-fix 
cleanup double word in comment.
 2022-01-24 14:31:16 -08:00
Tom Rix
2b957afec7 cleanup double word in comment. 
Remove the second 'a' and 'into'

Signed-off-by: Tom Rix <trix@redhat.com>
 2022-01-24 12:43:39 -08:00
Yann Collet
feaaf7a6b1 slightly shortened status and summary lines in very verbose mode 2022-01-21 21:38:35 -08:00
binhdvo
17017ac8db
Change zstdless behavior to align with zless (#2909) 
* Change zstdless behavior to align with zless
 2022-01-21 19:57:19 -05:00
Yann Collet
b27356fd27
Merge pull request #3005 from cwoffenden/faster-amalgamate 
Use faster Python script to amalgamate
 2022-01-21 16:12:22 -08:00
Yann Collet
24318093cc slightly shortened compression status update line 
to fit within 80 columns limit.
 2022-01-21 14:08:46 -08:00
Yonatan Komornik
8ab95f24da
Merge pull request #2985 from yoniko/zstd-output-file-buffer 
ZSTD CLI: Use buffered output
 2022-01-21 13:57:05 -08:00
Yonatan Komornik
1598e6c634
Async write for decompression (#2975) 
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
 2022-01-21 13:55:41 -08:00
Nick Terrell
2f03c1996f
Merge pull request #3013 from WojciechMula/simplify-asm 
Simplify HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop
 2022-01-21 11:13:40 -08:00
Felix Handte
4a35912b3b
Merge pull request #3018 from felixhandte/gh-action-release-artifacts-trigger-fix 
Trigger Release Artifact Generation on Publish
 2022-01-20 21:08:59 -05:00
Yann Collet
71921e596f
Merge pull request #2983 from facebook/minLitPricev2 
[opt] minor compression ratio improvement
 2022-01-20 16:02:31 -08:00
W. Felix Handte
fa9cb4510a Trigger Release Artifact Generation on Publish 
We previously triggered release artifact generation on release creation. We
sometimes observed that the action failed to run. I hypothesized that we were
hitting rate limiting or something. I just stumbled across [this documentat-
ion](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release), which says:

> Note: Workflows are not triggered for the `created`, `edited`, or `deleted`
> activity types for draft releases. When you create your release through the
> GitHub browser UI, your release may automatically be saved as a draft.

This must have been what was happening. This commit therefore changes the
trigger to the `published` activity. This should be more reliable.

This does have the unfortunate side effect that artifacts won't be generated
or attached until *after* the release has been published, which is what I was
trying to avoid by using the `created` activity. Oh well.
 2022-01-20 17:36:28 -05:00
Felix Handte
330c97d2bf
Merge pull request #3015 from felixhandte/enable-cet-test-illegal-instruction 
Add GitHub Action Checking that Zstd Runs Successfully Under CET
 2022-01-20 16:18:42 -05:00
Elliot Gorokhovsky
a8f1aa2f6d
Merge pull request #3016 from embg/macro_lint 
Minor lint fix
 2022-01-20 13:05:02 -07:00
Felix Handte
c7e8315d88
Merge pull request #2994 from hjl-tools/hjl/cet-report/dev 
x86: Append -z cet-report=error to LDFLAGS
 2022-01-20 14:58:13 -05:00
Elliot Gorokhovsky
f936dd89cb Minor lint fix 2022-01-20 11:54:43 -07:00
H.J. Lu
d6fcdd123c x86: Append -z cet-report=error to LDFLAGS 
Append -z cet-report=error to LDFLAGS if -fcf-protection is enabled by
default in compiler to catch the missing Intel CET marker:

compiling multi-threaded dynamic library 1.5.1
/usr/local/bin/ld: obj/conf_f408b4c825de923ffc88f7f21b6884b1/dynamic/huf_decompress_amd64.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
...
LINK obj/conf_dbc0b41e36c44111bb0bb918e093d7c1/zstd
/usr/local/bin/ld: obj/conf_dbc0b41e36c44111bb0bb918e093d7c1/huf_decompress_amd64.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
 2022-01-20 08:32:01 -08:00
W. Felix Handte
7cf80cb94c Add GitHub Action Checking that Zstd Runs Successfully Under CET 2022-01-20 11:25:52 -05:00
Felix Handte
4dfc4eca9a
Merge pull request #2992 from hjl-tools/hjl/cet/dev 
x86-64: Enable Intel CET
 2022-01-20 11:24:47 -05:00
Carl Woffenden
3f181b6192 More descriptive exclusion error; updated docs and copyright 2022-01-20 14:50:31 +01:00
Wojciech Muła
e74ca7979e Simplify HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop 
Get rid of three divisions. The original expression was:

    opmin := min((oend0 - op0) / 10, (oend1 - op1) / 10, (oend2 - op2) / 10, (oend3 - op3) / 10)
    r15   := min(r15, opmin)

The division by 10 can be moved outside the `min`:

    opmin := min(oend0 - op0, oend1 - op1, oend2 - op2, oend3 - op3)
    r15   := min(r15, opmin/10)
 2022-01-19 18:38:46 +01:00
Carl Woffenden
dc983e7d68 Typo (and missing commit) 2022-01-19 18:05:35 +01:00
Elliot Gorokhovsky
40bc9ee6c4
Merge pull request #2999 from embg/ci_instructions_update 
Update CI documentation
 2022-01-19 10:03:07 -07:00
Elliot Gorokhovsky
8250faa01b Update CI documentation 2022-01-19 09:58:31 -07:00
Carl Woffenden
786263ea85 Suggestion from code review 2022-01-19 17:48:10 +01:00
Carl Woffenden
566ebce347
Python style change 
Co-authored-by: Alexandre Bury <alexandre.bury@gmail.com>
 2022-01-19 17:33:20 +01:00
Carl Woffenden
5fd6ddaf8b Fixed bugs found in other projects 
When testing amalgamating other projects it was found: invalid Unicode errors were tripping Python's text IO, and the header search order appears differs from the shell version.
 2022-01-19 16:56:03 +01:00
Carl Woffenden
dd7d29a19c Updated README 2022-01-19 15:57:33 +01:00
Carl Woffenden
7d90f0b520 Test and tidy 
Made the Python more Python-like. Added notes and general tidy. Tested exclusions and building with various options. Tested all scripts.
 2022-01-19 11:32:53 +01:00
Nick Terrell
ef1eafded3
Merge pull request #3009 from terrelln/asm-flags 
[build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
 2022-01-18 15:48:11 -08:00
Nick Terrell
3369a106eb
Merge pull request #2998 from floppym/issue2997 
Avoid updating timestamps when the destination is stdout
 2022-01-18 15:14:06 -08:00
Nick Terrell
8ea3d57de4 [build][asm] Pass ASFLAGS to the assembler instead of CFLAGS 
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`

Fixes #3006.
 2022-01-18 15:11:29 -08:00
Carl Woffenden
8f1e51f99f Feature parity with original shell script; needs further testing 2022-01-18 19:07:18 +01:00
Elliot Gorokhovsky
fe8fefb28d
Merge pull request #3000 from embg/mt_memory_readability 
Documentation and minor refactor to clarify MT memory management
 2022-01-18 10:18:01 -07:00
Elliot Gorokhovsky
9b6dfedf0c Documentation and minor refactor to clarify MT memory management. 2022-01-18 09:43:05 -07:00