[REVERTED] Support 2GiB+ videos: Encode `Blob` as arrow `Binary` by emilk · Pull Request #10875 · rerun-io/rerun

emilk · 2025-08-12T16:23:20Z

This changes the encoding of the component column containing bobs from List<List<u8>> to List<Binary>, i.e. BinaryArray or LargeBinaryArray (both are supported!).

Old data is migrated on load.

This adds support for blobs larger than 2 GiB, e.g. large video files.

In the viewer we now support both 32-bit and 64-bit offsets for the binary data (BinaryArray vs LargeBinaryArray).
This is technically a type of datatype generics. Sort of. See #9144

The SDK code (Rust, Python, C++) always encoded as 64-bit (supporting >2GiB videos).
Legacy blobs are converted from 32-bit List to 64-bit binary

TODO

Fix python
Merge Better support binary arrow data in the viewer #10899
Merge Enable backwards compatibility checks for Python-only snippets #10898
Use 63-bit LargeBinary instead of 31-bit Binary
- Closes videos larger than 2.1GB cause rerun to crash #10929
- Closes Allow logging >2GiB binary blobs in C++ #10973
@rerun-bot full-check

github-actions · 2025-08-12T16:23:42Z

Web viewer built successfully. If applicable, you should also test it:

I have tested the web viewer

Result	Commit	Link	Manifest
✅	`527734c`	https://rerun.io/viewer/pr/10875	`+nightly` `+main`

^{Note: This comment is updated whenever you push a commit.}

github-actions · 2025-08-12T16:23:46Z

Latest documentation preview deployed successfully.

Result	Commit	Link
✅	`527734c`	https://landing-84hcg2shc-rerun.vercel.app/docs

^{Note: This comment is updated whenever you push a commit.}

emilk · 2025-08-12T16:37:50Z

crates/store/re_types/definitions/rerun/datatypes/blob.fbs

  "attr.rust.tuple_struct"
 ) {
-  data: [ubyte] (order: 100);
+  data: [ubyte] (order: 100, "attr.rerun.override_type": "binary");


This is the key change

emilk · 2025-08-13T12:12:54Z

@rerun-bot full-check

github-actions · 2025-08-13T12:13:29Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/16936854258

emilk · 2025-08-13T12:32:53Z

crates/viewer/re_ui/tests/snapshots/arrow_ui.png

The new arrow UI never shows the contents of blob, just their size

emilk · 2025-08-13T18:49:13Z

@rerun-bot full-check

github-actions · 2025-08-13T18:49:41Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/16946460940

emilk · 2025-08-14T10:55:13Z

This is a big PR. I'll split it in two.

### Related * Split out from #10875 See commit messages

emilk · 2025-08-22T07:59:37Z

@rerun-bot full-check

github-actions · 2025-08-22T08:00:11Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/17149677935

emilk · 2025-08-22T14:14:13Z

@rerun-bot full-check

github-actions · 2025-08-22T14:14:44Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/17157666898

This reverts commit 139dd83.

### Related * Reverts #10875 * Reverts #11005 * Fixes #11028 * Closes #11032 * Re-opens #10929 * Re-opens #10973 ### What Reverts the the change from `List<u8>` to `Binary` because of Rerun Cloud migration woes. Hopefully temporarily.

(cherry picking from #11039) ### Related * Reverts #10875 * Reverts #11005 * Fixes #11028 * Closes #11032 * Re-opens #10929 * Re-opens #10973 ### What Reverts the the change from `List<u8>` to `Binary` because of Rerun Cloud migration woes. Hopefully temporarily. Co-authored-by: Emil Ernerfeldt <[email protected]>

### Related * https://linear.app/rerun/project/cloud-support-for-evolution-of-data-schemas-59fb0b1a81ee/overview * Solves #11028 * rerun-io/dataplatform#1605 * Follows #10875 ### What This migrates all arrow schemas to the latest sorbet version right after decoding them off the gRPC wire

emilk added include in changelog 🔩 data model Sorbet labels Aug 12, 2025

emilk force-pushed the emilk/blob-as-binary branch from 3e42878 to 5aec85b Compare August 12, 2025 16:25

emilk commented Aug 12, 2025

View reviewed changes

emilk force-pushed the emilk/blob-as-binary branch from d18ad87 to 6a2b085 Compare August 13, 2025 11:19

emilk added the 🪵 Log & send APIs Affects the user-facing API for all languages label Aug 13, 2025

emilk force-pushed the emilk/blob-as-binary branch from 6a2b085 to ec623a3 Compare August 13, 2025 11:43

emilk commented Aug 13, 2025

View reviewed changes

crates/viewer/re_ui/tests/snapshots/arrow_ui.png

Copy link
Copy Markdown

Member Author

emilk Aug 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new arrow UI never shows the contents of blob, just their size

emilk force-pushed the emilk/blob-as-binary branch 3 times, most recently from 629b15b to 14bbb97 Compare August 13, 2025 18:45

emilk marked this pull request as ready for review August 13, 2025 19:01

emilk added the do-not-merge Do not merge this PR label Aug 13, 2025

emilk marked this pull request as draft August 14, 2025 10:55

emilk mentioned this pull request Aug 14, 2025

Better support binary arrow data in the viewer #10899

Merged

emilk force-pushed the emilk/blob-as-binary branch from 14bbb97 to 875ca13 Compare August 14, 2025 12:22

emilk changed the base branch from main to emilk/misc-binary-improvements August 14, 2025 12:22

emilk marked this pull request as ready for review August 14, 2025 12:24

emilk mentioned this pull request Aug 14, 2025

Add arrow data tree view and syntax highlighting #10777

Merged

emilk added a commit that referenced this pull request Aug 14, 2025

Better support binary arrow data in the viewer (#10899)

c6d3c78

### Related * Split out from #10875 See commit messages

Base automatically changed from emilk/misc-binary-improvements to main August 14, 2025 20:46

emilk force-pushed the emilk/blob-as-binary branch from 875ca13 to 1f47c5e Compare August 14, 2025 20:47

Codegen: Add support for encoding data as arrow binary

ad8d9c4

emilk added 2 commits August 22, 2025 09:09

Use 64 bit binary arrays in our migrator

f70b79d

Try to fix test

20e2211

emilk marked this pull request as ready for review August 22, 2025 08:30

emilk added 7 commits August 22, 2025 10:55

Switch C++ to 64-bit

2b1d7c1

Fix warning

c368db5

Fix bad error messages

0e3718b

Better error message again

a6046c2

Better error message

7f3838a

Improve chunk comparing: treat BinaryArray as LargeBinaryArray

0b711c1

Fix C++

527734c

emilk merged commit 139dd83 into main Aug 24, 2025
89 of 92 checks passed

emilk deleted the emilk/blob-as-binary branch August 24, 2025 14:39

oxkitsune mentioned this pull request Aug 26, 2025

Fix MCAP parsers to handle Blob as LargeBinary #11005

Merged

This was referenced Aug 28, 2025

Migrate arrow schemas right after decoding them from protobuff #11032

Merged

LargeBinary Migration Doesn't Update Datafusion Schema #11028

Open

emilk added a commit that referenced this pull request Aug 28, 2025

Revert "Support 2GiB+ videos: Encode Blob as arrow Binary (#10875)"

17a15e2

This reverts commit 139dd83.

emilk mentioned this pull request Aug 28, 2025

Revert BinaryArray migration #11039

Merged

andrea-reale mentioned this pull request Sep 2, 2025

Revert BinaryArray migration (#11039) #11065

Merged

emilk changed the title ~~Support 2GiB+ videos: Encode Blob as arrow Binary~~ [REVERTED] Support 2GiB+ videos: Encode Blob as arrow Binary Sep 2, 2025

emilk added exclude from changelog PRs with this won't show up in CHANGELOG.md and removed include in changelog labels Sep 2, 2025

emilk mentioned this pull request Sep 2, 2025

Support 2GiB+ videos: Encode Blob as arrow Binary #11068

Closed

1 task

Conversation

emilk commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

github-actions bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilk Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

emilk commented Aug 13, 2025

Uh oh!

github-actions bot commented Aug 13, 2025

Uh oh!

emilk Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

emilk commented Aug 13, 2025

Uh oh!

github-actions bot commented Aug 13, 2025

Uh oh!

emilk commented Aug 14, 2025

Uh oh!

emilk commented Aug 22, 2025

Uh oh!

github-actions bot commented Aug 22, 2025

Uh oh!

emilk commented Aug 22, 2025

Uh oh!

github-actions bot commented Aug 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

emilk commented Aug 12, 2025 •

edited

Loading

github-actions bot commented Aug 12, 2025 •

edited

Loading

github-actions bot commented Aug 12, 2025 •

edited

Loading