Skip to content

Commit ae42c69

Browse files
abey79teh-cmc
andauthored
Chunk store browser (#7226)
### What - Closes #404 This PR introduce a new, low-level chunk store browser crate `re_chunk_store_ui` with the following features: - browse both the recording and blueprint store - chunk store metadata display - chunk list view - info and config - query: all chunks/latest-at-relevent chunks/range-relevant chunks - filter: by entity and/or component - sort by: chunk id, entity, row count, any timeline - copy entire list to clipboard - chunk view - chunk metadata (including underlying arrow metadata) - sort by: row id, any timeline - copy chunk to clipboard The feature is available from the rerun menu and the `ctrl-shift-D` shortcut. Note: it would be very easy to drop the current `re_viewer_context` dependency and make a stand-alone chunk store viewer binary that only depends on `re_ui` and the lower-level store/type crates. <img width="1656" alt="image" src="https://github.com/user-attachments/assets/68c6485e-0919-4375-bb0a-772c32898305"> <img width="1656" alt="image" src="https://github.com/user-attachments/assets/15164965-abf8-4d67-b141-a95b848b2704"> ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Using examples from latest `main` build: [rerun.io/viewer](https://rerun.io/viewer/pr/7226?manifest_url=https://app.rerun.io/version/main/examples_manifest.json) * Using full set of examples from `nightly` build: [rerun.io/viewer](https://rerun.io/viewer/pr/7226?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG * [x] If applicable, add a new check to the [release checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)! * [x] If have noted any breaking changes to the log API in `CHANGELOG.md` and the migration guide - [PR Build Summary](https://build.rerun.io/pr/7226) - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html) To run all checks from `main`, comment on the PR with `@rerun-bot full-check`. --------- Co-authored-by: Clement Rey <[email protected]>
1 parent 9d321a7 commit ae42c69

File tree

22 files changed

+1467
-159
lines changed

22 files changed

+1467
-159
lines changed

ARCHITECTURE.md

Lines changed: 23 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -88,14 +88,15 @@ Of course, this will only take us so far. In the future we plan on caching queri
8888
Here is an overview of the crates included in the project:
8989

9090
<picture>
91-
<img src="https://static.rerun.io/crates/0a6b29a48f3929d5fb07f1aea889271a9eface09/full.png" alt="">
92-
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/0a6b29a48f3929d5fb07f1aea889271a9eface09/480w.png">
93-
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/0a6b29a48f3929d5fb07f1aea889271a9eface09/768w.png">
94-
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/0a6b29a48f3929d5fb07f1aea889271a9eface09/1024w.png">
95-
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/0a6b29a48f3929d5fb07f1aea889271a9eface09/1200w.png">
91+
<img src="https://static.rerun.io/crates/0973a21dff25026ed999e7d70d7c177cd2621514/full.png" alt="">
92+
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/0973a21dff25026ed999e7d70d7c177cd2621514/480w.png">
93+
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/0973a21dff25026ed999e7d70d7c177cd2621514/768w.png">
94+
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/0973a21dff25026ed999e7d70d7c177cd2621514/1024w.png">
95+
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/0973a21dff25026ed999e7d70d7c177cd2621514/1200w.png">
9696
</picture>
9797

9898

99+
99100
<!-- !!! IMPORTANT!!!
100101
101102
This image must be updated each time a crate is added/removed/updated.
@@ -125,22 +126,23 @@ Update instructions:
125126

126127
##### UI crates
127128

128-
| Crate | Description |
129-
|-----------------------------|-------------------------------------------------------------------------------------------------------------|
130-
| re_blueprint_tree | The UI for the blueprint tree in the left panel. |
131-
| re_component_ui | Provides UI editors for Rerun component data for registration with the Rerun Viewer component UI registry. |
132-
| re_selection_panel | The UI for the selection panel. |
133-
| re_space_view | Types & utilities for defining Space View classes and communicating with the Viewport. |
134-
| re_space_view_bar_chart | A Space View that shows a single bar chart. |
135-
| re_space_view_dataframe | A Space View that shows the data contained in entities in a table. |
136-
| re_space_view_spatial | Space Views that show entities in a 2D or 3D spatial relationship. |
137-
| re_space_view_tensor | A Space View dedicated to visualizing tensors with arbitrary dimensionality. |
138-
| re_space_view_text_document | A simple Space View that shows a single text box. |
139-
| re_space_view_text_log | A Space View that shows text entries in a table and scrolls with the active time. |
140-
| re_space_view_time_series | A Space View that shows plots over Rerun timelines. |
141-
| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |
142-
| re_viewer | The Rerun Viewer |
143-
| re_viewport | The central viewport panel of the Rerun viewer. |
129+
| Crate | Description |
130+
|-----------------------------|------------------------------------------------------------------------------------------------------------|
131+
| re_blueprint_tree | The UI for the blueprint tree in the left panel. |
132+
| re_chunk_store_ui | A chunk store browser UI. |
133+
| re_component_ui | Provides UI editors for Rerun component data for registration with the Rerun Viewer component UI registry. |
134+
| re_selection_panel | The UI for the selection panel. |
135+
| re_space_view | Types & utilities for defining Space View classes and communicating with the Viewport. |
136+
| re_space_view_bar_chart | A Space View that shows a single bar chart. |
137+
| re_space_view_dataframe | A Space View that shows the data contained in entities in a table. |
138+
| re_space_view_spatial | Space Views that show entities in a 2D or 3D spatial relationship. |
139+
| re_space_view_tensor | A Space View dedicated to visualizing tensors with arbitrary dimensionality. |
140+
| re_space_view_text_document | A simple Space View that shows a single text box. |
141+
| re_space_view_text_log | A Space View that shows text entries in a table and scrolls with the active time. |
142+
| re_space_view_time_series | A Space View that shows plots over Rerun timelines. |
143+
| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |
144+
| re_viewer | The Rerun Viewer |
145+
| re_viewport | The central viewport panel of the Rerun viewer. |
144146

145147

146148
##### UI support crates

Cargo.lock

Lines changed: 27 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -850,9 +850,9 @@ dependencies = [
850850

851851
[[package]]
852852
name = "cargo-manifest"
853-
version = "0.15.0"
853+
version = "0.15.1"
854854
source = "registry+https://github.com/rust-lang/crates.io-index"
855-
checksum = "de6c0a20b187645f8569845874d0589d167f86a17504a6b6c2c9e0921c906ee3"
855+
checksum = "6b29a81447dee6418e779b6d25c8ca7250d06b583fc3975e6577162518da9d80"
856856
dependencies = [
857857
"serde",
858858
"thiserror",
@@ -1472,16 +1472,6 @@ version = "2.5.0"
14721472
source = "registry+https://github.com/rust-lang/crates.io-index"
14731473
checksum = "7e962a19be5cfc3f3bf6dd8f61eb50107f356ad6270fbb3ed41476571db78be5"
14741474

1475-
[[package]]
1476-
name = "deranged"
1477-
version = "0.3.11"
1478-
source = "registry+https://github.com/rust-lang/crates.io-index"
1479-
checksum = "b42b6fa04a440b495c8b04d0e71b707c585f83cb9cb28cf8cd0d976c315e31b4"
1480-
dependencies = [
1481-
"powerfmt",
1482-
"serde",
1483-
]
1484-
14851475
[[package]]
14861476
name = "derivative"
14871477
version = "2.2.0"
@@ -3305,12 +3295,6 @@ dependencies = [
33053295
"num-traits",
33063296
]
33073297

3308-
[[package]]
3309-
name = "num-conv"
3310-
version = "0.1.0"
3311-
source = "registry+https://github.com/rust-lang/crates.io-index"
3312-
checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9"
3313-
33143298
[[package]]
33153299
name = "num-derive"
33163300
version = "0.4.1"
@@ -3875,12 +3859,6 @@ version = "1.5.1"
38753859
source = "registry+https://github.com/rust-lang/crates.io-index"
38763860
checksum = "3bccab0e7fd7cc19f820a1c8c91720af652d0c88dc9664dd72aef2614f04af3b"
38773861

3878-
[[package]]
3879-
name = "powerfmt"
3880-
version = "0.2.0"
3881-
source = "registry+https://github.com/rust-lang/crates.io-index"
3882-
checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
3883-
38843862
[[package]]
38853863
name = "ppv-lite86"
38863864
version = "0.2.17"
@@ -4376,6 +4354,24 @@ dependencies = [
43764354
"web-time",
43774355
]
43784356

4357+
[[package]]
4358+
name = "re_chunk_store_ui"
4359+
version = "0.19.0-alpha.1+dev"
4360+
dependencies = [
4361+
"egui",
4362+
"egui_extras",
4363+
"itertools 0.13.0",
4364+
"re_chunk_store",
4365+
"re_data_ui",
4366+
"re_entity_db",
4367+
"re_format",
4368+
"re_format_arrow",
4369+
"re_log_types",
4370+
"re_types",
4371+
"re_ui",
4372+
"re_viewer_context",
4373+
]
4374+
43794375
[[package]]
43804376
name = "re_component_ui"
43814377
version = "0.19.0-alpha.1+dev"
@@ -5277,6 +5273,7 @@ dependencies = [
52775273
"re_build_tools",
52785274
"re_chunk",
52795275
"re_chunk_store",
5276+
"re_chunk_store_ui",
52805277
"re_component_ui",
52815278
"re_data_loader",
52825279
"re_data_source",
@@ -6594,35 +6591,31 @@ dependencies = [
65946591

65956592
[[package]]
65966593
name = "time"
6597-
version = "0.3.36"
6594+
version = "0.3.21"
65986595
source = "registry+https://github.com/rust-lang/crates.io-index"
6599-
checksum = "5dfd88e563464686c916c7e46e623e520ddc6d79fa6641390f2e3fa86e83e885"
6596+
checksum = "8f3403384eaacbca9923fa06940178ac13e4edb725486d70e8e15881d0c836cc"
66006597
dependencies = [
6601-
"deranged",
66026598
"itoa",
66036599
"js-sys",
66046600
"libc",
6605-
"num-conv",
66066601
"num_threads",
6607-
"powerfmt",
66086602
"serde",
66096603
"time-core",
66106604
"time-macros",
66116605
]
66126606

66136607
[[package]]
66146608
name = "time-core"
6615-
version = "0.1.2"
6609+
version = "0.1.1"
66166610
source = "registry+https://github.com/rust-lang/crates.io-index"
6617-
checksum = "ef927ca75afb808a4d64dd374f00a2adf8d0fcff8e7b184af886c3c87ec4a3f3"
6611+
checksum = "7300fbefb4dadc1af235a9cef3737cea692a9d97e1b9cbcd4ebdae6f8868e6fb"
66186612

66196613
[[package]]
66206614
name = "time-macros"
6621-
version = "0.2.18"
6615+
version = "0.2.9"
66226616
source = "registry+https://github.com/rust-lang/crates.io-index"
6623-
checksum = "3f252a68540fde3a3877aeea552b832b40ab9a69e318efd078774a01ddee1ccf"
6617+
checksum = "372950940a5f07bf38dbe211d7283c9e6d7327df53794992d293e534c733d09b"
66246618
dependencies = [
6625-
"num-conv",
66266619
"time-core",
66276620
]
66286621

Cargo.toml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,9 +77,10 @@ re_tuid = { path = "crates/utils/re_tuid", version = "=0.19.0-alpha.1", default-
7777

7878
# crates/viewer:
7979
re_blueprint_tree = { path = "crates/viewer/re_blueprint_tree", version = "=0.19.0-alpha.1", default-features = false }
80+
re_component_ui = { path = "crates/viewer/re_component_ui", version = "=0.19.0-alpha.1", default-features = false }
8081
re_context_menu = { path = "crates/viewer/re_context_menu", version = "=0.19.0-alpha.1", default-features = false }
8182
re_data_ui = { path = "crates/viewer/re_data_ui", version = "=0.19.0-alpha.1", default-features = false }
82-
re_component_ui = { path = "crates/viewer/re_component_ui", version = "=0.19.0-alpha.1", default-features = false }
83+
re_chunk_store_ui = { path = "crates/viewer/re_chunk_store_ui", version = "=0.19.0-alpha.1", default-features = false }
8384
re_renderer = { path = "crates/viewer/re_renderer", version = "=0.19.0-alpha.1", default-features = false }
8485
re_renderer_examples = { path = "crates/viewer/re_renderer_examples", version = "=0.19.0-alpha.1", default-features = false }
8586
re_selection_panel = { path = "crates/viewer/re_selection_panel", version = "=0.19.0-alpha.1", default-features = false }

crates/store/re_chunk_store/src/query.rs

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,43 @@ use crate::RowId;
2323
// as both static and temporal, which is probably wrong.
2424

2525
impl ChunkStore {
26+
/// Retrieve all [`Timeline`]s in the store.
27+
pub fn all_timelines(&self) -> BTreeSet<Timeline> {
28+
self.temporal_chunk_ids_per_entity
29+
.values()
30+
.flat_map(|per_timeline| per_timeline.keys().copied())
31+
.collect()
32+
}
33+
34+
/// Retrieve all [`EntityPath`]s in the store.
35+
pub fn all_entities(&self) -> BTreeSet<EntityPath> {
36+
self.static_chunk_ids_per_entity
37+
.keys()
38+
.cloned()
39+
.chain(self.temporal_chunk_ids_per_entity.keys().cloned())
40+
.collect()
41+
}
42+
43+
/// Retrieve all [`ComponentName`]s in the store.
44+
pub fn all_components(&self) -> BTreeSet<ComponentName> {
45+
self.static_chunk_ids_per_entity
46+
.values()
47+
.flat_map(|static_chunks_per_component| static_chunks_per_component.keys())
48+
.chain(
49+
self.temporal_chunk_ids_per_entity_per_component
50+
.values()
51+
.flat_map(|temporal_chunk_ids_per_timeline| {
52+
temporal_chunk_ids_per_timeline.values().flat_map(
53+
|temporal_chunk_ids_per_component| {
54+
temporal_chunk_ids_per_component.keys()
55+
},
56+
)
57+
}),
58+
)
59+
.copied()
60+
.collect()
61+
}
62+
2663
/// Retrieve all the [`ComponentName`]s that have been written to for a given [`EntityPath`] on
2764
/// the specified [`Timeline`].
2865
///
@@ -72,7 +109,7 @@ impl ChunkStore {
72109
/// Static components are always included in the results.
73110
///
74111
/// Returns `None` if the entity has never had any data logged to it.
75-
pub fn all_components(&self, entity_path: &EntityPath) -> Option<ComponentNameSet> {
112+
pub fn all_components_for_entity(&self, entity_path: &EntityPath) -> Option<ComponentNameSet> {
76113
re_tracing::profile_function!();
77114

78115
self.query_id.fetch_add(1, Ordering::Relaxed);
@@ -349,6 +386,26 @@ impl ChunkStore {
349386

350387
Some(ResolvedTimeRange::new(*start, *end))
351388
}
389+
390+
/// Returns the min and max times at which data was logged on a specific timeline, considering
391+
/// all entities.
392+
///
393+
/// This ignores static data.
394+
pub fn time_range(&self, timeline: &Timeline) -> Option<ResolvedTimeRange> {
395+
re_tracing::profile_function!();
396+
397+
self.query_id.fetch_add(1, Ordering::Relaxed);
398+
399+
self.temporal_chunk_ids_per_entity
400+
.values()
401+
.filter_map(|temporal_chunk_ids_per_timeline| {
402+
let per_time = temporal_chunk_ids_per_timeline.get(timeline)?;
403+
let start = per_time.per_start_time.first_key_value()?.0;
404+
let end = per_time.per_end_time.last_key_value()?.0;
405+
Some(ResolvedTimeRange::new(*start, *end))
406+
})
407+
.reduce(|r1, r2| r1.union(r2))
408+
}
352409
}
353410

354411
// LatestAt

crates/store/re_chunk_store/src/store.rs

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -482,6 +482,18 @@ impl ChunkStore {
482482
self.chunks_per_chunk_id.values()
483483
}
484484

485+
/// Get a chunk based on its ID.
486+
#[inline]
487+
pub fn chunk(&self, id: &ChunkId) -> Option<&Arc<Chunk>> {
488+
self.chunks_per_chunk_id.get(id)
489+
}
490+
491+
/// Get the number of chunks.
492+
#[inline]
493+
pub fn num_chunks(&self) -> usize {
494+
self.chunks_per_chunk_id.len()
495+
}
496+
485497
/// Lookup the _latest_ arrow [`ArrowDataType`] used by a specific [`re_types_core::Component`].
486498
#[inline]
487499
pub fn lookup_datatype(&self, component_name: &ComponentName) -> Option<&ArrowDataType> {
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
[package]
2+
authors.workspace = true
3+
description = "Display the contents of chunk stores."
4+
edition.workspace = true
5+
homepage.workspace = true
6+
license.workspace = true
7+
name = "re_chunk_store_ui"
8+
publish = true
9+
readme = "README.md"
10+
repository.workspace = true
11+
rust-version.workspace = true
12+
version.workspace = true
13+
include = ["../../LICENSE-APACHE", "../../LICENSE-MIT", "**/*.rs", "Cargo.toml"]
14+
15+
[lints]
16+
workspace = true
17+
18+
[package.metadata.docs.rs]
19+
all-features = true
20+
21+
[dependencies]
22+
re_chunk_store.workspace = true
23+
re_data_ui.workspace = true
24+
re_entity_db.workspace = true
25+
re_format.workspace = true
26+
re_format_arrow.workspace = true
27+
re_log_types.workspace = true
28+
re_types.workspace = true
29+
re_ui.workspace = true
30+
re_viewer_context.workspace = true
31+
32+
egui.workspace = true
33+
egui_extras.workspace = true
34+
itertools.workspace = true
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# re_chunk_store_ui
2+
3+
Part of the [`rerun`](https://github.com/rerun-io/rerun) family of crates.
4+
5+
[![Latest version](https://img.shields.io/crates/v/re_chunk_store_ui.svg?speculative-link)](https://crates.io/crates/re_data_ui)
6+
[![Documentation](https://docs.rs/re_chunk_store_ui/badge.svg?speculative-link)](https://docs.rs/re_data_ui)
7+
![MIT](https://img.shields.io/badge/license-MIT-blue.svg)
8+
![Apache](https://img.shields.io/badge/license-Apache-blue.svg)
9+
10+
Provides a UI for the datastore.

0 commit comments

Comments
 (0)