Skip to content

proposal: zstd: add DecodedSize function #647

@dsnet

Description

@dsnet

I'm trying to implement proper rate-limiting and I'm finding it difficult to adequately rate-limit a server when dealing with untrusted []byte inputs since I don't know how large the decompressed output will end up. I could convert the []byte into a io.Reader and perform rate limiting based on the number of bytes read from zstd.Decoder, but that seems somewhat unfortunate.

I propose the addition of:

// DecodedLength reports the decompressed length of src in bytes.
func (*Decoder) DecodedLength(src []byte) (int64, error)

It may or may not make sense as a method on Decoder.

IIUC, this should only need to perform work proportional to O(len(src)). The amount of memory it will occupy is primarily what's needed to initialize the ANS tables and to read each symbol. We would then count each literal symbol and the length of each LZ77 back-reference symbol.

This API also assists in determining what sized buffers to grab for the output.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions