test: add the aged-corpus benchmark for the day planner#3572
Merged
Conversation
Everything shipped in this epic so far has been reasoned improvement, not measured improvement. This seeds a synthetic corpus at 1, 6 and 12 simulated months and reports what the operations behind a user action actually cost, so the claim can be checked instead of argued. The corpus is shaped like a real install — a small pending head over a large terminal ledger — because leaving every job pending measures a backlog nobody has and hides the property under test. With all jobs pending, claimNext grew 127 to 317 microseconds across the sweep, which is cost tracking outstanding work exactly as designed; with a realistic ledger it is flat. Baseline (median of 9, microseconds), 360 to 4380 agent entities: outbox.claimNext 157 / 94 / 127 dayView.captures 224 / 144 / 136 dayView.statusEvents 313 / 344 / 159 dayView.plannerOwnsDay 122 / 113 / 86 planEditor.pendingDiffs 80 / 56 / 53 planWriter.lookback 666 / 445 / 262 Flat or lower at twelve months than at one, across a 12x increase in stored history. Opt-in via --dart-define=LOTTI_BENCHMARK=1; a smoke test runs always so the harness cannot rot. Does not cover wake prompt bytes, token counts or digest duration — those need the agent workflow, not the storage layer.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
💤 Files with no reviewable changes (1)
📝 WalkthroughWalkthroughThe changes add an opt-in Daily OS benchmark that seeds synthetic history, measures bounded-cost operations across simulated corpus ages, validates metric coverage, and documents usage. The AI settings file also repositions a part directive without changing behavior. ChangesDaily OS benchmark
AI settings import adjustment
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant BenchmarkTest
participant DayPlannerCorpus
participant AgentDatabase
participant DayProcessingDb
BenchmarkTest->>DayPlannerCorpus: Seed synthetic corpus
DayPlannerCorpus->>AgentDatabase: Store daily entities
DayPlannerCorpus->>DayProcessingDb: Enqueue and drain jobs
BenchmarkTest->>DayPlannerCorpus: Measure operations
DayPlannerCorpus->>AgentDatabase: Query entity metrics
DayPlannerCorpus->>DayProcessingDb: Query outbox metrics
DayPlannerCorpus-->>BenchmarkTest: Return counts and timings
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3572 +/- ##
=======================================
Coverage 99.22% 99.22%
=======================================
Files 1781 1781
Lines 130359 130359
=======================================
Hits 129345 129345
Misses 1014 1014
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything shipped in this epic so far (#3564, #3566, #3569, #3571) has been
reasoned improvement — the arguments were sound and tests pinned the
behaviour, but nobody had measured a single number. This makes the claim
checkable.
What it does
Seeds a synthetic Daily OS corpus at 1, 6 and 12 simulated months under the
coordinator (the agent that actually accumulates — a
day_agent:<dayId>holdsone day and goes cold), then reports the cost of the operations a real user
action triggers.
Opt-in, because seeding twelve months writes thousands of rows:
fvm flutter test --dart-define=LOTTI_BENCHMARK=1 \ test/features/daily_os_next/benchmark/A smoke test runs unconditionally and asserts the full metric set, so a
renamed repository method breaks the build rather than silently dropping a
column from a report nobody reads closely.
Baseline
Median of 9, microseconds. Absolute values are machine-specific; the slope is
the point.
outbox.claimNextdayView.capturesdayView.statusEventsdayView.plannerOwnsDayplanEditor.pendingDiffsplanWriter.lookbackCorpus: 360 / 2,184 / 4,380 agent entities and 90 / 546 / 1,095 processing
jobs. Every metric is flat or lower at twelve months than at one, across a 12×
increase in stored history. Values drifting down is noise and cache warming,
not a real speedup.
One finding worth reading
My first corpus left every processing job
pending, andclaimNextgrew127 → 189 → 317 µs across the sweep. That is not a defect — it is cost
tracking outstanding work, exactly as designed — but it measures a 1,095-job
backlog no real install has, and it hides the property actually under test.
Reshaping the corpus the way a real outbox looks (a small pending head over a
large terminal ledger) makes
claimNextflat at 157 / 94 / 127 while theledger grows 12×. That is ADR 0044's partial-index claim demonstrated rather
than asserted: the retained ledger stays off the drain path.
It also means the benchmark is only as honest as its corpus, which is why the
shape is documented in the harness rather than buried.
Scope
Deliberately storage-layer only. It does not cover wake prompt bytes,
token counts, or digest wake duration — those need the full agent workflow
rather than the repositories, and are a separate piece of work. Called out in
the README so the gap is visible rather than assumed covered.
Analyzer and formatter clean; no CHANGELOG entry (test-only).
Summary by CodeRabbit
Documentation
Tests