Skip to content

Conversation

@abey79
Copy link
Member

@abey79 abey79 commented Aug 18, 2024

What

This PR introduce a new, low-level chunk store browser crate re_chunk_store_ui with the following features:

  • browse both the recording and blueprint store
  • chunk store metadata display
  • chunk list view
    • info and config
    • query: all chunks/latest-at-relevent chunks/range-relevant chunks
    • filter: by entity and/or component
    • sort by: chunk id, entity, row count, any timeline
    • copy entire list to clipboard
  • chunk view
    • chunk metadata (including underlying arrow metadata)
    • sort by: row id, any timeline
    • copy chunk to clipboard

The feature is available from the rerun menu and the ctrl-shift-D shortcut.

Note: it would be very easy to drop the current re_viewer_context dependency and make a stand-alone chunk store viewer binary that only depends on re_ui and the lower-level store/type crates.

image image

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
  • If applicable, add a new check to the release checklist!
  • If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

To run all checks from main, comment on the PR with @rerun-bot full-check.

@abey79 abey79 changed the title [WIP] Datastore UI Chunk store browser Aug 24, 2024
@abey79 abey79 added ui concerns graphical user interface enhancement New feature or request include in changelog labels Aug 24, 2024
@abey79 abey79 marked this pull request as ready for review August 26, 2024 08:46
@nikolausWest
Copy link
Member

Nice!

Small nit:
Screenshot 2024-08-26 at 11 06 15
It took me a while to understand that this meant "number of rows" and not "row number"

@abey79
Copy link
Member Author

abey79 commented Aug 26, 2024

Nice!

Small nit: Screenshot 2024-08-26 at 11 06 15 It took me a while to understand that this meant "number of rows" and not "row number"

Yeah the reasoning is that the actual column content is narrow (just a number), so it's annoying to have a wide column header. Let's see if we can figure out something better with @gavrelina

@nikolausWest
Copy link
Member

Yeah the reasoning is that the actual column content is narrow (just a number), so it's annoying to have a wide column header.

# rows would have done it for me (although 2 chars longer)

@teh-cmc teh-cmc self-requested a review August 27, 2024 07:33
Copy link
Member

@teh-cmc teh-cmc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's amazing 🙇


UPDATE: I found the cause, see code comments. My recommendation is simply to add a TODO, maybe open an issue. The root of the issue that the chunk will be displated with comfyterm, which will use the width of the parent terminal as a hint.

When you click the copy-as-text button, I'm not quite sure how the implementation decides what width should each column be? The current value is annoyingly small (a RowId doesn't even fit on a single line):
image


The fact that I'm wasting all this vertical space at the top even when I've uncollapsed everything is annoying:
image


The timestamp ranges in the chunk list could use an indication of the time spanned, e.g.

#0..=#1 255 (1 255 ticks) | #103..=#2 613 (2510 ticks) | 2024-08-26 12:38:33.536540Z..=2024-08-26 12:38:33.740418Z (0.203878 seconds)`

image


they see me browsin', they hatin'

use re_chunk_store::external::re_chunk::external::arrow2::array::Utf8Array;
use re_viewer_context::UiLayout;

//TODO(ab): this is copied/modified from `re_data_ui`. Consider unifying them?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also we really need to setup a registry for lossless text representations of arrow payloads based on semantic types.

E.g. I want rerun.components.Color to display as 0xFF88AAFF, not the unintelligible integer it prints today.
And that's not specific to this view, it's everywhere we represent arrow as just lossless text (including terminal dumps).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

Alternatively, I would eventually like to have a toggle to apply the entire re_data_ui machinery for display. That could be useful eg. for interpreting enums (can those eventually be fully arrow?). That feature would have to be behind a cargo flag, such that re_chunk_store_ui can remain independent from the rerun viewer, should we want to make it its own binary.

@abey79
Copy link
Member Author

abey79 commented Aug 27, 2024

The fact that I'm wasting all this vertical space at the top even when I've uncollapsed everything is annoying:

Yes, that's was an unwanted consequence of the fancy sort arrow ui. I've reverted to a less nice UI that doesn't cause the issue. This will be further addressed later with:

Edit: I didn't read your original comment properly and answered to something else 🤦🏻. Yeah I agree this is not ideal. Not sure what's the immediate step should be here. I suggest we let it simmer for a bit to figure out what's the best way (e.g. would a top toggle table vs. metadata tree be ok or is it useful see both the metadata and the table at once?)

@abey79
Copy link
Member Author

abey79 commented Aug 27, 2024

@abey79 abey79 merged commit ae42c69 into main Aug 27, 2024
@abey79 abey79 deleted the antoine/datastore-ui branch August 27, 2024 15:26
emilk added a commit that referenced this pull request Aug 28, 2024
@emilk
Copy link
Member

emilk commented Aug 29, 2024

Remember to always review changes to Cargo.lock!

And if you have merge conflicts in Cargo.lock, just do git checkout main Cargo.lock && cargo check

teh-cmc pushed a commit that referenced this pull request Aug 29, 2024
…ons (#7308)

* This was done already in #7228
* …but accidentally reverted in
#7226 🤦
* Closes #7216

`cargo +1.80.0 build -p rerun-cli` works after this
teh-cmc pushed a commit that referenced this pull request Aug 29, 2024
…ons (#7308)

* This was done already in #7228
* …but accidentally reverted in
#7226 🤦
* Closes #7216

`cargo +1.80.0 build -p rerun-cli` works after this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request include in changelog ui concerns graphical user interface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

re_datastore: arrow table debug-view

5 participants