Repository navigation
[1/21] feat(storage): add datasets resource-type prefix to logical paths - #1
tanishqgandhi1908 wants to merge 11 commits into
Conversation
… tests Review fixes on apache#5911: use dataset owner email in retrieveLatestDatasetVersion, replace brittle Option.get/.head with headOption, strip datasets prefix on selection-modal reopen, clarify FileResolver docs; add FileResolver/DatasetFileNode and frontend path-helper tests. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The datasets logical-path prefix strips four leading segments (datasets/owner/dataset/version); update the cover-image test's input path to include the prefix so the extracted relative path is the file name, not an empty string. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Now that models are coming as a separate resource/table, the leading resource-type segment is what selects the backing table, so an unprefixed path can no longer be routed unambiguously. Make the "datasets" prefix required instead of a tolerated fallback, and migrate existing data. - FileResolver: a dataset path must start with the "datasets" segment; unprefixed paths are no longer treated as dataset paths. - pytexera DatasetFileDocument: same rule, mirroring the backend. - FileListerSourceOpExec: parse the now-prefixed datasetVersionPath (skip the "datasets" segment); extracted into a testable helper. - Migration (sql/updates/29.sql): prepend "datasets/" to legacy paths stored in workflow.content and workflow_version.content, covering both the fileName (scan sources) and datasetVersionPath (file lister) operator properties. Only values whose first two segments match an existing (user.email, dataset.name) are rewritten, so local paths and URLs are left untouched; email format is irrelevant (owner may be a username without "@"). Uses create_missing=false and is idempotent. - Example workflows: use datasets-prefixed paths. Tests: FileResolverSpec and WorkflowExecutionsResourceSpec updated; test_dataset_file_document.py updated; new FileListerSourceOpExecSpec. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…l-path-prefix # Conflicts: # frontend/src/app/common/type/datasetVersionFileTree.spec.ts
…format python test - dataset-selection-modal.component.spec.ts: expect the datasets-prefixed selectedPath in version (non-file) mode, matching the emitted path. - test_dataset_file_document.py: apply ruff format (wrap an over-length line) so `ruff format --check` passes. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The cover-image endpoints resolve a dataset path built as
{owner}/{name}/{coverImage}. With the prefix now required by FileResolver,
these must carry the datasets/ segment; add it in the set-cover,
get-cover redirect, and cover-url handlers so cover images resolve again.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
👋 Thanks for opening this pull request, @tanishqgandhi1908! It looks like the pull request description doesn't quite follow our template yet:
Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed. You can find the template prompts by editing the description, or see CONTRIBUTING.md for the full contribution flow. |
Automated Reviewer SuggestionsBased on the
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 356 | 0.217 | 26,799/38,426/38,426 us | 🟢 -53.2% / ⚪ within ±5% |
| 🔴 | bs=100 sw=10 sl=64 | 752 | 0.459 | 134,422/144,672/144,672 us | 🔴 +6.3% / ⚪ within ±5% |
| ⚪ | bs=1000 sw=10 sl=64 | 859 | 0.524 | 1,163,779/1,217,286/1,217,286 us | ⚪ within ±5% / ⚪ within ±5% |
Baseline details
Latest main 0467c76 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 356 tuples/sec | 298 tuples/sec | n/a | +19.5% | n/a |
| bs=10 sw=10 sl=64 | MB/s | 0.217 MB/s | 0.182 MB/s | n/a | +19.2% | n/a |
| bs=10 sw=10 sl=64 | p50 | 26,799 us | 32,160 us | n/a | -16.7% | n/a |
| bs=10 sw=10 sl=64 | p95 | 38,426 us | 82,061 us | n/a | -53.2% | n/a |
| bs=10 sw=10 sl=64 | p99 | 38,426 us | 82,061 us | n/a | -53.2% | n/a |
| bs=100 sw=10 sl=64 | throughput | 752 tuples/sec | 781 tuples/sec | n/a | -3.7% | n/a |
| bs=100 sw=10 sl=64 | MB/s | 0.459 MB/s | 0.477 MB/s | n/a | -3.8% | n/a |
| bs=100 sw=10 sl=64 | p50 | 134,422 us | 126,483 us | n/a | +6.3% | n/a |
| bs=100 sw=10 sl=64 | p95 | 144,672 us | 148,417 us | n/a | -2.5% | n/a |
| bs=100 sw=10 sl=64 | p99 | 144,672 us | 148,417 us | n/a | -2.5% | n/a |
| bs=1000 sw=10 sl=64 | throughput | 859 tuples/sec | 854 tuples/sec | n/a | +0.6% | n/a |
| bs=1000 sw=10 sl=64 | MB/s | 0.524 MB/s | 0.521 MB/s | n/a | +0.6% | n/a |
| bs=1000 sw=10 sl=64 | p50 | 1,163,779 us | 1,172,531 us | n/a | -0.7% | n/a |
| bs=1000 sw=10 sl=64 | p95 | 1,217,286 us | 1,202,005 us | n/a | +1.3% | n/a |
| bs=1000 sw=10 sl=64 | p99 | 1,217,286 us | 1,202,005 us | n/a | +1.3% | n/a |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,561.76,200,128000,356,0.217,26799.13,38425.63,38425.63
1,100,10,64,20,2660.46,2000,1280000,752,0.459,134421.97,144671.55,144671.55
2,1000,10,64,20,23277.64,20000,12800000,859,0.524,1163778.72,1217286.08,1217286.08…jsdom (apache#7999) ### What changes were proposed in this PR? `report-generation.service.spec.ts` drives the real html2canvas, which clones the whole document rather than the element it is pointed at. The unit-test builder runs spec files with `isolate: false`, so one jsdom document is shared by every spec file in a worker and the clone drags in whatever DOM the files before it left behind. On the macOS runner that clone was measured at 12–37s against a 20s test timeout, failing `fails when the editor cannot be rendered` while ubuntu and windows passed on the same commit. Two changes, both test-only: - **`report-generation.service.spec.ts`** — the snapshot suite parks the document's existing body nodes for the duration of the file and restores them after, so the render's cost depends only on the DOM these tests build. It spans the file rather than each test because the renders outlive the tests that start them (visible in the failing log: clone #1 finished during test #3). - **`jsdom-svg-polyfill.ts`** — stops jsdom announcing `getComputedStyle(elt, pseudoElt)` and `scrollTo` on the virtual console, which vitest forwards to `console.error` as a full stack trace per call. The failing job carried 5,691 of them for six renders. Dropping `pseudoElt` changes no behaviour: jsdom complains and then ignores the argument, returning the element's own declaration either way. `scrollTo` has nothing to move — jsdom has no layout, and the one html2canvas reaches for belongs to the throwaway clone iframe, so it is neutered as each `contentWindow` is handed out. ### Any related issues, documentation, discussions? Closes apache#8001. This continues apache#7886 (closing apache#7887), which stopped the four image-inlining tests in this file waiting on the render. The flake moved to the two tests that still await one. Scope note for reviewers: this bounds the render's cost, it does not remove the dependency on a real render. The two remaining tests still await html2canvas, so a sufficiently loaded runner could in principle still be slow — just not by the two orders of magnitude measured here. Taking apache#7886's logic to its conclusion (no test in this file waiting on a real render) would need a stub html2canvas that survives `isolate: false`, which is a larger change to what those two tests cover; happy to do that instead if reviewers prefer it. Unrelated pre-existing noise left alone: a full run still emits ~827 jsdom `HTMLCanvasElement.prototype.getContext` traces from `JointUIService.getMeasureContext` (`joint-ui.service.ts:219`). Silencing it would change what `measureText` returns for the joint-ui specs, so it is out of scope here. ### How was this PR tested? `ng test --watch=false` on this branch (base `444fc58284`), four full runs: 201 files, 5131 passed / 1 skipped each time, no `Not implemented: window.getComputedStyle` traces. Clone time across the file's six renders was 79–353 ms on three warm runs; the test that was timing out renders in 85–114 ms. The first run on a cold checkout was slower — up to 3.9s for a clone — which is a reminder that machine load, not document size, is what is left. The effect of the console noise was measured directly by seeding the document with 500 nodes: clone time across the six renders went 18.8s → 13.1s with the polyfill change alone (~30%; the share is larger where console forwarding is slower, as on CI). `yarn --cwd frontend format:ci` is clean. I cannot demonstrate the macOS leg green from here — that needs CI on this branch. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5 [1M context]) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Part 1 of the ML models stack. Base:
stack/00-base.datasetsprefixPresentation stack on the fork — see the PR plan for the intended upstream sequence.
Stack (bottom to top)
Each PR's base is the branch below it, so the diff shown is only that step. Merging bottom-up auto-retargets the one above.