Skip to content

[1/21] feat(storage): add datasets resource-type prefix to logical paths - #1

Closed
tanishqgandhi1908 wants to merge 11 commits into
stack/00-basefrom
stack/01-datasets-prefix
Closed

tanishqgandhi1908 wants to merge 11 commits into
stack/00-basefrom
stack/01-datasets-prefix

Conversation

@tanishqgandhi1908

@tanishqgandhi1908 tanishqgandhi1908 commented Aug 7, 2026 •

Copy link
Copy Markdown
Owner

Part 1 of the ML models stack. Base: stack/00-base.

  • consolidate the datasets literal
  • fix(file-service): prefix dataset cover-image paths with datasets/
  • fix(test): align dataset-selection-modal spec to prefixed path; ruff-format python test
  • Merge remote-tracking branch 'upstream/main' into feat/dataset-logical-path-prefix
  • feat(storage): require the datasets prefix on dataset logical paths
  • test(frontend): use datasets-prefixed path in cover-image spec
  • Merge remote-tracking branch 'upstream/main' into feat/dataset-logical-path-prefix
  • refactor: remove unused frontend path parser; reword prefix docs
  • test+fix: extend datasets prefix to python UDF path + add frontend parse tests
  • fix(file-service): correct dataset owner email, harden file tree, add tests
  • feat: datasets prefix

Presentation stack on the fork — see the PR plan for the intended upstream sequence.


Stack (bottom to top)

Each PR's base is the branch below it, so the diff shown is only that step. Merging bottom-up auto-retargets the one above.

aicam and others added 11 commits July 1, 2026 13:57
… tests

Review fixes on apache#5911: use dataset owner email in retrieveLatestDatasetVersion,
replace brittle Option.get/.head with headOption, strip datasets prefix on
selection-modal reopen, clarify FileResolver docs; add FileResolver/DatasetFileNode
and frontend path-helper tests.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
The datasets logical-path prefix strips four leading segments
(datasets/owner/dataset/version); update the cover-image test's input
path to include the prefix so the extracted relative path is the file
name, not an empty string.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Now that models are coming as a separate resource/table, the leading
resource-type segment is what selects the backing table, so an unprefixed
path can no longer be routed unambiguously. Make the "datasets" prefix
required instead of a tolerated fallback, and migrate existing data.

- FileResolver: a dataset path must start with the "datasets" segment;
  unprefixed paths are no longer treated as dataset paths.
- pytexera DatasetFileDocument: same rule, mirroring the backend.
- FileListerSourceOpExec: parse the now-prefixed datasetVersionPath
  (skip the "datasets" segment); extracted into a testable helper.
- Migration (sql/updates/29.sql): prepend "datasets/" to legacy paths
  stored in workflow.content and workflow_version.content, covering both
  the fileName (scan sources) and datasetVersionPath (file lister)
  operator properties. Only values whose first two segments match an
  existing (user.email, dataset.name) are rewritten, so local paths and
  URLs are left untouched; email format is irrelevant (owner may be a
  username without "@"). Uses create_missing=false and is idempotent.
- Example workflows: use datasets-prefixed paths.

Tests: FileResolverSpec and WorkflowExecutionsResourceSpec updated;
test_dataset_file_document.py updated; new FileListerSourceOpExecSpec.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
…l-path-prefix

# Conflicts:
#	frontend/src/app/common/type/datasetVersionFileTree.spec.ts
…format python test

- dataset-selection-modal.component.spec.ts: expect the datasets-prefixed
  selectedPath in version (non-file) mode, matching the emitted path.
- test_dataset_file_document.py: apply ruff format (wrap an over-length line)
  so `ruff format --check` passes.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
The cover-image endpoints resolve a dataset path built as
{owner}/{name}/{coverImage}. With the prefix now required by FileResolver,
these must carry the datasets/ segment; add it in the set-cover,
get-cover redirect, and cover-url handlers so cover images resolve again.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

👋 Thanks for opening this pull request, @tanishqgandhi1908!

It looks like the pull request description doesn't quite follow our template yet:

  • The What changes were proposed in this PR? section is missing; please keep the template's headings.
  • The How was this PR tested? section is missing; please keep the template's headings.
  • The Was this PR authored or co-authored using generative AI tooling? section is missing; please keep the template's headings.

Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed.

You can find the template prompts by editing the description, or see CONTRIBUTING.md for the full contribution flow.

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Yicong-Huang, @Ma77Ball, @kunwp1
    You can notify them by mentioning @Yicong-Huang, @Ma77Ball, @kunwp1 in a comment.

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

⚠️ Benchmark changes need a look

🟢 5 better · 🔴 1 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main 0467c76 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🟢 bs=10 sw=10 sl=64 356 0.217 26,799/38,426/38,426 us 🟢 -53.2% / ⚪ within ±5%
🔴 bs=100 sw=10 sl=64 752 0.459 134,422/144,672/144,672 us 🔴 +6.3% / ⚪ within ±5%
⚪ bs=1000 sw=10 sl=64 859 0.524 1,163,779/1,217,286/1,217,286 us ⚪ within ±5% / ⚪ within ±5%
Baseline details

Latest main 0467c76 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 356 tuples/sec 298 tuples/sec n/a +19.5% n/a
bs=10 sw=10 sl=64 MB/s 0.217 MB/s 0.182 MB/s n/a +19.2% n/a
bs=10 sw=10 sl=64 p50 26,799 us 32,160 us n/a -16.7% n/a
bs=10 sw=10 sl=64 p95 38,426 us 82,061 us n/a -53.2% n/a
bs=10 sw=10 sl=64 p99 38,426 us 82,061 us n/a -53.2% n/a
bs=100 sw=10 sl=64 throughput 752 tuples/sec 781 tuples/sec n/a -3.7% n/a
bs=100 sw=10 sl=64 MB/s 0.459 MB/s 0.477 MB/s n/a -3.8% n/a
bs=100 sw=10 sl=64 p50 134,422 us 126,483 us n/a +6.3% n/a
bs=100 sw=10 sl=64 p95 144,672 us 148,417 us n/a -2.5% n/a
bs=100 sw=10 sl=64 p99 144,672 us 148,417 us n/a -2.5% n/a
bs=1000 sw=10 sl=64 throughput 859 tuples/sec 854 tuples/sec n/a +0.6% n/a
bs=1000 sw=10 sl=64 MB/s 0.524 MB/s 0.521 MB/s n/a +0.6% n/a
bs=1000 sw=10 sl=64 p50 1,163,779 us 1,172,531 us n/a -0.7% n/a
bs=1000 sw=10 sl=64 p95 1,217,286 us 1,202,005 us n/a +1.3% n/a
bs=1000 sw=10 sl=64 p99 1,217,286 us 1,202,005 us n/a +1.3% n/a
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,561.76,200,128000,356,0.217,26799.13,38425.63,38425.63
1,100,10,64,20,2660.46,2000,1280000,752,0.459,134421.97,144671.55,144671.55
2,1000,10,64,20,23277.64,20000,12800000,859,0.524,1163778.72,1217286.08,1217286.08

tanishqgandhi1908 pushed a commit that referenced this pull request Aug 27, 2026
…jsdom (apache#7999)

### What changes were proposed in this PR?

`report-generation.service.spec.ts` drives the real html2canvas, which
clones the whole document
rather than the element it is pointed at. The unit-test builder runs
spec files with
`isolate: false`, so one jsdom document is shared by every spec file in
a worker and the clone
drags in whatever DOM the files before it left behind.

On the macOS runner that clone was measured at 12–37s against a 20s test
timeout, failing
`fails when the editor cannot be rendered` while ubuntu and windows
passed on the same commit.

Two changes, both test-only:

- **`report-generation.service.spec.ts`** — the snapshot suite parks the
document's existing body
nodes for the duration of the file and restores them after, so the
render's cost depends only on
the DOM these tests build. It spans the file rather than each test
because the renders outlive
the tests that start them (visible in the failing log: clone #1 finished
during test #3).

- **`jsdom-svg-polyfill.ts`** — stops jsdom announcing
`getComputedStyle(elt, pseudoElt)` and
`scrollTo` on the virtual console, which vitest forwards to
`console.error` as a full stack trace
per call. The failing job carried 5,691 of them for six renders.
Dropping `pseudoElt` changes no
behaviour: jsdom complains and then ignores the argument, returning the
element's own declaration
either way. `scrollTo` has nothing to move — jsdom has no layout, and
the one html2canvas reaches
for belongs to the throwaway clone iframe, so it is neutered as each
`contentWindow` is handed out.

### Any related issues, documentation, discussions?

Closes apache#8001.

This continues apache#7886 (closing apache#7887), which stopped the four
image-inlining tests in this file
waiting on the render. The flake moved to the two tests that still await
one.

Scope note for reviewers: this bounds the render's cost, it does not
remove the dependency on a
real render. The two remaining tests still await html2canvas, so a
sufficiently loaded runner
could in principle still be slow — just not by the two orders of
magnitude measured here. Taking
apache#7886's logic to its conclusion (no test in this file waiting on a real
render) would need a
stub html2canvas that survives `isolate: false`, which is a larger
change to what those two tests
cover; happy to do that instead if reviewers prefer it.

Unrelated pre-existing noise left alone: a full run still emits ~827
jsdom
`HTMLCanvasElement.prototype.getContext` traces from
`JointUIService.getMeasureContext`
(`joint-ui.service.ts:219`). Silencing it would change what
`measureText` returns for the
joint-ui specs, so it is out of scope here.

### How was this PR tested?

`ng test --watch=false` on this branch (base `444fc58284`), four full
runs: 201 files,
5131 passed / 1 skipped each time, no `Not implemented:
window.getComputedStyle` traces.
Clone time across the file's six renders was 79–353 ms on three warm
runs; the test that was
timing out renders in 85–114 ms. The first run on a cold checkout was
slower — up to 3.9s for a
clone — which is a reminder that machine load, not document size, is
what is left.

The effect of the console noise was measured directly by seeding the
document with 500 nodes:
clone time across the six renders went 18.8s → 13.1s with the polyfill
change alone (~30%; the
share is larger where console forwarding is slower, as on CI).

`yarn --cwd frontend format:ci` is clean.

I cannot demonstrate the macOS leg green from here — that needs CI on
this branch.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5 [1M context])

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants