mirror of
https://github.com/facebook/zstd.git
synced 2025-10-05 00:07:15 -04:00
d993a288e0
Summary: Added an option -p# where -p0 (default) sets the aggregation method to fastest speed while -p1 sets the aggregation method to median. Also added a new column in the csv file to report this option's value. Test Plan: `` $ ./largeNbDicts -1 --nbDicts=1 -D ~/benchmarks/html/html_8_16K.32K.dict ~/benchmarks/html/html_8_16K/* loading 7450 files... created src buffer of size 83.4 MB split input into 7450 blocks loading dictionary /home/zhuhan/benchmarks/html/html_8_16K.32K.dict compressing at level 1 without dictionary : Ratio=3.03 (28827863 bytes) compressed using a 32768 bytes dictionary : Ratio=4.28 (20410262 bytes) generating 1 dictionaries, using 0.1 MB of memory Compression Speed : 306.0 MB/s Fastest Speed : 310.6 MB/s $ ./largeNbDicts -1 --nbDicts=1 -p1 -D ~/benchmarks/html/html_8_16K.32K.dict ~/benchmarks/html/html_8_16K/* loading 7450 files... created src buffer of size 83.4 MB split input into 7450 blocks loading dictionary /home/zhuhan/benchmarks/html/html_8_16K.32K.dict compressing at level 1 without dictionary : Ratio=3.03 (28827863 bytes) compressed using a 32768 bytes dictionary : Ratio=4.28 (20410262 bytes) generating 1 dictionaries, using 0.1 MB of memory Compression Speed : 306.9 MB/s Median Speed : 298.4 MB/s ```
largeNbDicts
largeNbDicts is a benchmark test tool
dedicated to the specific scenario of
dictionary decompression using a very large number of dictionaries.
When dictionaries are constantly changing, they are always "cold",
suffering from increased latency due to cache misses.
The tool is created in a bid to investigate performance for this scenario, and experiment mitigation techniques.
Command line :
largeNbDicts [Options] filename(s)
Options :
-z : benchmark compression (default)
-d : benchmark decompression
-r : recursively load all files in subdirectories (default: off)
-B# : split input into blocks of size # (default: no split)
-# : use compression level # (default: 3)
-D # : use # as a dictionary (default: create one)
-i# : nb benchmark rounds (default: 6)
--nbBlocks=#: use # blocks for bench (default: one per file)
--nbDicts=# : create # dictionaries for bench (default: one per block)
-h : help (this text)
Advanced Options (see zstd.h for documentation) :
--dedicated-dict-search
--dict-content-type=#
--dict-attach-pref=#