Skip to content

Receiving (potentially large) uploads #96

@chaoran-chen

Description

@chaoran-chen

This issue is to discuss how we should implement the upload/receival of user submissions, i.e. the way of the data from the submitter's device (browser or CLI) to our (relational core) database. The main challenge is that the uploads can be huge and I think that our software should be able to accept arbitrarily large uploads. The admin can/should (of course) limit the maximum size.

Some thoughts:

  • Do we want to be able to resume after a short connection disruption? Maybe we can use tus.io or resumable.js Not relevant for MVP.
  • What are the compression formats that we should support? zip, zstd, gz, and xz?
  • Should the user also be able to upload large uncompressed files? If they do, maybe we can compress them in the browser/CLI software first? Or do we (for now) simply reject large uncompressed files (e.g., above 1 GB)?
  • We need to have protection again zip bombs.
  • The implementation needs to be memory-efficient (we shouldn't hold multiple GBs of data in RAM for one upload).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions