Former Amazon VP Adrian cockcroft on Twitterbroke the newssaid that after switching the compression algorithm of AWS from gzip to zstd, a huge amount of memory is saved, the compressed S3 storage is reduced by about 30%, and the space saved can reach the scale of EB (1 EB = 1024 PB = 1024 * 1024 TB).

An Amazon employee added to the statement, saying that AmazonWhat has changed is not how the data stored by the customer is compressed, but The way S3 itself stores service data (mainly logs) – S3 itself switched from gzip logs to ztsd logs, reducing storage costs by 30%.

However, Amazon did not issue any announcements related to changing compression technology. Careful Holmes netizens found that Amazon S3 had a zip at the end of November 2021.31% off which corresponds to a 30% reduction in storage costs.

About Zstd

Zstd (Zstandard) is a fast lossless compression algorithm open sourced by Facebook. It is mainly used in real-time compression scenarios at the zlib level and has a better compression ratio.

Zstd can also provide stronger compression ratios at the expense of compression speed, with the ratio of speed to compression ratio configurable incrementally.

Zstd is an excellent compression technology,and zlib, lz4, xz and other compression algorithms are different. Zstd seeks a solution that combines compression performance and compression ratio. Zstd alsoA special compression mode “dictionary compression” is provided for small data, which supports generating dictionary files in training mode to improve the compression ratio of small data packets.

The following is a performance comparison of common compression algorithms: