96 receiving potentially large uploads · Pull Request #577 · loculus-project/loculus

ghost · 2023-11-22T11:04:21Z

No description provided.

ghost · 2023-11-23T09:52:23Z

Potentially relevant:
JetBrains/Exposed#871

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt

backend/src/test/kotlin/org/pathoplexus/backend/controller/ReviseEndpointTest.kt

fengelniederhammer

Overall it looks good, but some details probably need improvement.

backend/src/main/kotlin/org/pathoplexus/backend/controller/SubmissionController.kt

backend/src/main/kotlin/org/pathoplexus/backend/model/SubmitModel.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/MetadataUploadAuxTable.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/SequenceUploadAuxTable.kt

backend/src/main/kotlin/org/pathoplexus/backend/utils/CompressionService.kt

backend/src/test/resources/logback-test.xml

backend/src/test/kotlin/org/pathoplexus/backend/controller/SubmitFiles.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt

backend/src/test/kotlin/org/pathoplexus/backend/controller/ReviseEndpointTest.kt

backend/src/main/kotlin/org/pathoplexus/backend/config/Config.kt

ghost · 2023-11-28T15:59:00Z

Follow-up tickets

loculus-project/security-issues#23
#604

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt

ghost · 2023-11-28T23:05:47Z

supported compressed file formats, tested manually:

zst
xz
gz
lzma
bzip2 ( "Note that BZipCompressorOutputStream keeps hold of some big data structures in memory. While it is recommended for any stream that you close it as soon as you no longer need it, this is even more important for BZipCompressorOutputStream." )
zip ( creates a temporary file, see loculus-project/security-issues#23 )

fengelniederhammer

Just a couple of minor things 👍

backend/src/main/kotlin/org/pathoplexus/backend/model/SubmitModel.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/DatabaseService.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt

backend/src/test/kotlin/org/pathoplexus/backend/utils/CompressionServiceTest.kt

backend/src/test/kotlin/org/pathoplexus/backend/controller/SubmitFiles.kt

backend/src/test/kotlin/org/pathoplexus/backend/controller/SubmitEndpointTest.kt

backend/src/main/kotlin/org/pathoplexus/backend/utils/CompressionService.kt

backend/src/main/kotlin/org/pathoplexus/backend/model/SubmitModel.kt

ghost · 2023-11-29T14:17:47Z

Some performance measure:

1 mio identical sequence entries zstd compressed
very small file size + local upload = negligible upload time
batch insert into aux tables with 1000 entries (100 seems also fine and can reduce RAM footprint even more, but is probably a bit slower)

Result:

(Plateau at the end is pretty much exactly 1GB)

Roughly 4 min (where validating and copying into prod table is ~ 10 sec each)

backend/src/main/kotlin/org/pathoplexus/backend/model/SubmitModel.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/CompressionService.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/DatabaseService.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/SequenceEntryDataTables.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/SequenceEntryTableProvider.kt

backend/src/main/kotlin/org/pathoplexus/backend/service/DatabaseService.kt

* introduce two auxiliary tables to efficiently validate and merge metadata and sequence data * remove singleton SequenceEntriesTable and replace it with provided and cached `Table`s to facilitate compression of sequence data * de-compress sequence strings with custom dictionary when de-serializing * support for zstd, gzip, xz, lzma, zip, bzip2

ghost linked an issue Nov 22, 2023 that may be closed by this pull request

Receiving (potentially large) uploads #96

Closed

ghost force-pushed the 96-receiving-potentially-large-uploads branch 3 times, most recently from d79b4c4 to d3608d4 Compare November 27, 2023 16:46

ghost marked this pull request as ready for review November 27, 2023 16:46

ghost commented Nov 27, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt Outdated Show resolved Hide resolved

ghost commented Nov 27, 2023

View reviewed changes

backend/src/test/kotlin/org/pathoplexus/backend/controller/ReviseEndpointTest.kt Outdated Show resolved Hide resolved

ghost marked this pull request as draft November 27, 2023 16:55

ghost force-pushed the 96-receiving-potentially-large-uploads branch from d3608d4 to bfcebde Compare November 27, 2023 16:59

ghost marked this pull request as ready for review November 27, 2023 17:40

ghost requested review from chaoran-chen and fengelniederhammer and removed request for fengelniederhammer November 27, 2023 17:40

fengelniederhammer requested changes Nov 28, 2023

View reviewed changes

fengelniederhammer reviewed Nov 28, 2023

View reviewed changes

ghost force-pushed the 96-receiving-potentially-large-uploads branch 2 times, most recently from fd68633 to 816eaef Compare November 28, 2023 15:19

ghost commented Nov 28, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/config/Config.kt Outdated Show resolved Hide resolved

ghost force-pushed the 96-receiving-potentially-large-uploads branch from 816eaef to 35a8db1 Compare November 28, 2023 15:24

ghost commented Nov 28, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/service/UploadDatabaseService.kt Outdated Show resolved Hide resolved

ghost force-pushed the 96-receiving-potentially-large-uploads branch 2 times, most recently from 6dd476c to e69e83c Compare November 28, 2023 23:05

ghost force-pushed the 96-receiving-potentially-large-uploads branch from e69e83c to 0f04ee4 Compare November 28, 2023 23:09

fengelniederhammer requested changes Nov 29, 2023

View reviewed changes

fengelniederhammer reviewed Nov 29, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/model/SubmitModel.kt Outdated Show resolved Hide resolved

ghost force-pushed the 96-receiving-potentially-large-uploads branch 2 times, most recently from be974fc to e2b1de8 Compare November 29, 2023 13:47

ghost force-pushed the 96-receiving-potentially-large-uploads branch from 8dd241b to d9e868c Compare November 29, 2023 16:19

JonasKellerer reviewed Nov 30, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/model/SubmitModel.kt Outdated Show resolved Hide resolved

ghost force-pushed the 96-receiving-potentially-large-uploads branch from d9e868c to 28dd626 Compare November 30, 2023 09:47

fengelniederhammer approved these changes Nov 30, 2023

View reviewed changes

ghost force-pushed the 96-receiving-potentially-large-uploads branch 2 times, most recently from a31e910 to c336612 Compare December 1, 2023 21:50

ghost added the preview Triggers a deployment to argocd label Dec 1, 2023

fengelniederhammer reviewed Dec 4, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/service/DatabaseService.kt Outdated Show resolved Hide resolved

fengelniederhammer reviewed Dec 4, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/service/SequenceEntryDataTables.kt Outdated Show resolved Hide resolved

ghost force-pushed the 96-receiving-potentially-large-uploads branch 2 times, most recently from e290fe3 to 4f18267 Compare December 4, 2023 17:34

ghost mentioned this pull request Dec 4, 2023

Compression of sequences in postgres DB #497

Closed

ghost requested a review from fengelniederhammer December 4, 2023 17:55

ghost force-pushed the 96-receiving-potentially-large-uploads branch from 4f18267 to fe54e89 Compare December 4, 2023 17:58

fengelniederhammer requested changes Dec 5, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/service/SequenceEntryTableProvider.kt Outdated Show resolved Hide resolved

backend/src/main/kotlin/org/pathoplexus/backend/service/DatabaseService.kt Outdated Show resolved Hide resolved

ghost force-pushed the 96-receiving-potentially-large-uploads branch from 92b0c67 to 03388c0 Compare December 5, 2023 09:35

ghost requested a review from fengelniederhammer December 5, 2023 09:44

fengelniederhammer approved these changes Dec 5, 2023

View reviewed changes

backend/src/main/kotlin/org/pathoplexus/backend/service/DatabaseService.kt Outdated Show resolved Hide resolved

ghost force-pushed the 96-receiving-potentially-large-uploads branch from 03388c0 to 483ac99 Compare December 5, 2023 12:22

ghost force-pushed the 96-receiving-potentially-large-uploads branch from 483ac99 to 8c428b1 Compare December 5, 2023 12:43

ghost merged commit 73c35eb into main Dec 5, 2023

ghost deleted the 96-receiving-potentially-large-uploads branch December 5, 2023 13:04

This pull request was closed.

Conversation

ghost commented Nov 22, 2023 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Nov 23, 2023

Uh oh!

Uh oh!

Uh oh!

fengelniederhammer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost commented Nov 28, 2023

Uh oh!

Uh oh!

ghost commented Nov 28, 2023

Uh oh!

fengelniederhammer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost commented Nov 29, 2023 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ghost commented Nov 22, 2023 •

edited by ghost

Loading

ghost commented Nov 29, 2023 •

edited by ghost

Loading