Compression

gzip, zlib, and DEFLATE.

On this page

valk.compress implements gzip, zlib and DEFLATE, plus the CRC-32 and Adler-32 checksums. gzip is the format of .gz files and HTTP gzip; zlib is what HTTP calls deflate.

Compress data in memory#

Valk

use valk.compress

fn main() {
    let text = "hello hello hello"
    let packed = compress.gzip(text)
    let smallest = compress.gzip(text, 9)
    let unpacked = compress.gunzip(packed) ! {
        println("Corrupt data")
        return
    }
    println(unpacked == text) // true
}

Compress	Decompress	Format
`gzip(data, level)`	`gunzip(data, max_size)`	gzip
`zlib(data, level)`	`unzlib(data, max_size)`	zlib
`deflate(data, level)`	`inflate(data, max_size)`	raw DEFLATE
`compress(data, format, level)`	`decompress(data, format, max_size)`	a `compress.Format`

The level runs from 0 (store only) to 9 (smallest); the default is 6. Decompressing throws a CompressError on corrupt or truncated input.

Limit untrusted input#

Valk

use valk.compress

fn main() {
    let packed = compress.gzip("a".repeat(10000))
    let text = compress.gunzip(packed, 1000) ! {
        println(E.code == E.too_large) // true
        return
    }
}

A small compressed upload can expand to gigabytes. Pass max_size to stop at that many output bytes; 0 means no limit.

Compress a stream#

Valk

use valk.compress
use valk.fs
use valk.io

fn main() {
    let file = fs.stream("log.txt.gz", fs.OpenOptions { read: false, write: fs.WriteMode.truncate, create: true }) ! return
    let writer = compress.Compressor.new(file, compress.Format.gzip)
    writer.write("first line\n") ! return
    writer.close() ! return
    file.close() ! return

    let input = fs.stream("log.txt.gz") ! return
    let reader = compress.Decompressor.new(input, compress.Format.gzip)
    let text = io.read_all(reader) ! return
    print(text) // first line
}

Compressor is an io.Writer that compresses into another writer; Decompressor is an io.Reader over compressed input. Closing the Compressor writes the last block and the trailer, so close it before the file underneath. See Files & streams.

Checksums#

Valk

use valk.compress

fn main() {
    println(compress.crc32("123456789").to_hex())     // CBF43926
    println(compress.adler32("Wikipedia").to_hex())   // 11E60398
}

Both take a previous result as a second argument, to checksum data that arrives in pieces.

View page source ↗