|
| 1 | +# O1 selected and observed capture handoff |
| 2 | + |
| 3 | +Base: `583c2217d2524e8f2bc4c8a77dfa336309d741fc`. |
| 4 | +Rows: R03 and R04. This slice is groundwork plus capture-time metric freezing; |
| 5 | +it does **not** establish working SUMMARY_ONLY execution or close R03/R04. |
| 6 | +No remote fail-before, build, unit, corpus or charged-memory result has been |
| 7 | +obtained. No command requiring a local compiler or script was run. |
| 8 | + |
| 9 | +## Runtime changes |
| 10 | + |
| 11 | +At the base, `capture_presentation` copies selected M+1 or ordinary observed |
| 12 | +Bs points into its independently owned report unless a sink supplies scratch |
| 13 | +storage. `fill_selected_report` computes metrics again at each read, using |
| 14 | +the captured points but current engine metric inputs. Its sink-off array |
| 15 | +copy covers max(published count, metric count). |
| 16 | + |
| 17 | +The capture now computes the complete existing `pf_metrics_t` after the one |
| 18 | +presentation hook and before publishing the capture. The trade blocks use |
| 19 | +the same closed-row prefix and final open-row order. Selected sink points |
| 20 | +are validated before metric computation, with the same typed errors and |
| 21 | +precedence as the existing final selected-sink scan. A report read copies the |
| 22 | +frozen metric block, with no metric source, trade fold or timezone walk. |
| 23 | +Removing a sink still explicitly requests the legacy full array; its |
| 24 | +materialization remains available and uses the independent captured source. |
| 25 | + |
| 26 | +`SelectedCaptureLatches` separates final-value validation from the writer. |
| 27 | +Feed it final execution points in their recorded order. For a host that |
| 28 | +rewrites arbitrary earlier points, scan its final array at capture; do not |
| 29 | +latch provisional values. Its copyable scalar state preserves trade, |
| 30 | +open-lot, account, contiguity, then count refusal order. The existing selected |
| 31 | +sink validates its edited metric observations too, including a suffix hidden |
| 32 | +from publication; the anchor is excluded from observation validation. |
| 33 | + |
| 34 | +`PresentationCaptureWindow` owns two points, two observation/exposure flag |
| 35 | +pairs, a logical count and a held count. `append_final` releases the oldest |
| 36 | +point directly to the existing scratch writer before reusing its slot. |
| 37 | +`append_metric_point` supplies the same release interface with the original |
| 38 | +observation and exposure flags for the forthcoming exact metric passes. |
| 39 | +`emit_metric_tail` visits every edited held point, including hidden points. |
| 40 | +`published_count` subtracts only the number hidden by the hook. None of these |
| 41 | +operations retains the released prefix, allocates a growing collection or |
| 42 | +introduces a population cap. |
| 43 | + |
| 44 | +The resident state of these helpers is fixed by their types, independent of |
| 45 | +bars, observations, trades or keys. The frozen metric block reuses the |
| 46 | +existing 768-byte field in the capture. Existing scratch writers/readers each |
| 47 | +have a fixed 64 KiB window. FULL capture and FULL report arrays are still |
| 48 | +O(M) / O(Bs); scratch-backed STREAMED output remains a per-point external |
| 49 | +sequence. Neither qualifies as SUMMARY_ONLY. |
| 50 | + |
| 51 | +Batch-B already supplied finalizable sink recording, L17a selected validation |
| 52 | +latches, the two-point selected presentation, L9s sink assembly and bounded |
| 53 | +scratch readers. Those pieces were extended rather than replaced. The |
| 54 | +range-end held-point rewrite and extrema prefix restore were already present; |
| 55 | +this lane did not change equity recording, sealing, digests or their headers. |
| 56 | + |
| 57 | +## Exact metric integration boundary |
| 58 | + |
| 59 | +The metrics lane's `metrics_fold.hpp` and `tests/test_metrics_fold.cpp` WIP |
| 60 | +specify the calls below. The new capture test includes that header and |
| 61 | +exercises both exact passes through the held window, comparing all 768 bytes |
| 62 | +to the frozen array oracle for 96 additional curves. That unit requires the |
| 63 | +supervisor to consume the metrics lane's header and implementation first; |
| 64 | +neither has been copied into this lane. |
| 65 | + |
| 66 | +The expected runtime integration calls, already exercised by the new unit |
| 67 | +using that WIP's exact spellings, are: |
| 68 | + |
| 69 | +1. Construct `metrics::EquityMetricsFold first` before execution. |
| 70 | +2. Insert the selected anchor with |
| 71 | + `window.append_metric_point(anchor, false, false, emit_first)`. |
| 72 | + Insert each final selected observation with its actual exposure flag; |
| 73 | + ordinary observed capture has no synthetic anchor. |
| 74 | +3. In `emit_first(point, observation, in_market)`, call |
| 75 | + `first.fold(point, metrics::equity_month_key(point.time_ms, chart_timezone), |
| 76 | + observation, in_market)`. The held terminal value is supplied only after |
| 77 | + execution rewrites, and the held presentation window only after its edit. |
| 78 | + Finish that window with `window.emit_metric_tail(emit_first)`. |
| 79 | +4. Take `const auto pass_a = first.first_pass()`. Bind |
| 80 | + `metrics::EquityMetricInputs{initial_capital, first_open, last_close, |
| 81 | + net_profit}` with the selected first/last input prices and M denominator. |
| 82 | + The anchor's observation flag is false, including M=0 and M=1. |
| 83 | +5. Construct `metrics::EquityMetricsSecondPass second(pass_a)` and walk one |
| 84 | + exact deterministic replay. Its visitor calls |
| 85 | + `second.fold(point, metrics::equity_month_key(point.time_ms, chart_timezone), |
| 86 | + observation, in_market)` over the same final presented points. |
| 87 | +6. Obtain `const auto ratios = second.finish()` and require |
| 88 | + `ratios.matches(pass_a)`. This is additional metric evidence; it does not |
| 89 | + replace the contract's ordered trade and final execution/source witnesses. |
| 90 | +7. Freeze `metrics::finalize_metrics(pass_a, inputs, all, longs, shorts, |
| 91 | + &ratios)` into the capture's existing metric block. A repeated report read |
| 92 | + performs none of these calls. |
| 93 | + |
| 94 | +Do not infer exposure from nonzero open profit. Do not hold a non-recursive |
| 95 | +timezone guard across dispatch or replay. The recording lane must preserve |
| 96 | +the two still-editable points and their flags until presentation is final; |
| 97 | +folding the penultimate selected point before its possible edit is incorrect. |
| 98 | + |
| 99 | +Still needed from the runtime integration owner: the stored/admitted |
| 100 | +SUMMARY_ONLY mode; its selected/observed capture branch and frozen logical |
| 101 | +counts; the pre-bar-zero point/replay provider; execution and trade witness |
| 102 | +gates; exact exposure flags; and output suppression. These require engine or |
| 103 | +consumer declarations outside this lane's assigned function ranges. No new |
| 104 | +per-run global registry, alternate mode API or implicit fallback was added. |
| 105 | +The integration owner must also update the read-time metric descriptions in |
| 106 | +`src/native_execution_consumer.hpp` and `include/pineforge/metrics_stream.hpp`; |
| 107 | +this lane did not edit those shared headers. |
| 108 | + |
| 109 | +## Presentation hooks and compatibility |
| 110 | + |
| 111 | +The inventory below comes from `git grep` for `present_report(` across |
| 112 | +include, src, tests and examples. C callback hook forwarding does not expose |
| 113 | +an additional whole-array presentation hook. |
| 114 | + |
| 115 | +| Hook | Reads or edits | |
| 116 | +| --- | --- | |
| 117 | +| `BacktestEngine::present_report` in `include/pineforge/engine.hpp` | Default no-op. | |
| 118 | +| `PineStrategyHost::present_report` in `src/source/pine_strategy_host.cpp` | Final equity point; presented open-at-end trade rows in lot order; `total_trades`, `trades_len`, `net_profit` and closed-row PnL. It re-marks terminal rows, updates net PnL and rewrites the final point. It does not inspect equity history. | |
| 119 | +| `tests/test_report_sink_equity.cpp` fixture | Counts the presented length, delegates to Pine, edits the final point or shrinks the curve. | |
| 120 | +| `tests/test_report_sink_scratch_curve.cpp` fixture | Edits only the final point. | |
| 121 | +| `tests/test_report_sink_selected_l17a.cpp` fixture | Counts calls, reads/edits the last point and cuts one point; can throw. | |
| 122 | +| `tests/test_selected_metrics_l17b.cpp` `Host` | Reads presented trades and the window; edits its last two points. Its explicit legacy branch reads/edits the full curve's middle, last point and published length. It also snapshots engine history as a test oracle. | |
| 123 | +| Same file `LiveReportHost` | Counts calls and reads the scratch descriptor; no equity-array read. | |
| 124 | +| Same file `PineHost` | Counts calls and delegates to Pine's terminal presentation. | |
| 125 | +| Same file `InvalidWindowHost` | Deliberately replaces storage, grows/negates the length or throws after replacement. | |
| 126 | +| Same file `StorageHost` | Reads the presented first and last points and length. | |
| 127 | +| Same file `TradeStorageHost` | Reads/edits open-at-end trade rows, net PnL and final equity point. | |
| 128 | +| Same file `RandomCaptureHost` added here | Edits the last two points and hides zero, one or two points. | |
| 129 | +| `tests/test_selected_window_report_integration.cpp` fixture | Counts calls, throws or sets the published length to one. | |
| 130 | +| `tests/test_selected_production_routes.cpp` and `tests/test_selected_window_routes.cpp` fixtures | Record the presentation event; no array reads. | |
| 131 | + |
| 132 | +FULL continues to invoke the full-array hook on the full-array route, |
| 133 | +including arbitrary middle edits and shrink/hide behavior. Existing sink |
| 134 | +hooks retain their bounded view. Summary mode must require an explicit |
| 135 | +bounded-presentation declaration and an exact replay provider before calling |
| 136 | +any custom hook. The conservative default for an arbitrary override is |
| 137 | +whole-array access; C++ cannot discover which indexes an override will read. |
| 138 | + |
| 139 | +For a whole-array-only presenter, summary negotiation must return |
| 140 | +`PF_REPORT_MODE_REPLAY_UNAVAILABLE` before execution, as specified by the |
| 141 | +contract lane's `docs/design/o1-report-contract.md`, “Exact second replay for |
| 142 | +SUMMARY_ONLY.” |
| 143 | +The provider/transport must surface the requested mode's documented error |
| 144 | +rather than invoke the hook with a shorter array. This declaration/error |
| 145 | +boundary and the required successful-summary-versus-FULL hook test remain |
| 146 | +unimplemented here; a documentation entry is not runtime coverage. |
| 147 | + |
| 148 | +## Verification and remaining acceptance |
| 149 | + |
| 150 | +No tests in this slice have been executed. Expected fail-before behavior: |
| 151 | + |
| 152 | +| Test | Expected failure on the base | |
| 153 | +| --- | --- | |
| 154 | +| `test_selected_metrics_l17b` / `captured_metrics_are_frozen` | Repeated reads recompute percentages/exposure/CAGR from deliberately changed execution metric inputs, changing the raw metric block. | |
| 155 | +| Same executable / `frozen_sink_reads_need_no_metric_storage` | A selected read allocates a `SpillCurve` reader and trade folds; an observed read allocates a scratch reader and trade folds. The allocation-refusal interval detects both. | |
| 156 | +| `test_selected_capture_latches` | The frozen previous-epoch header closure lacks the new latch/window and metric-fold headers. The test also compares all failure combinations and 2,048 random arrays against the independent original scan. No remote first diagnostic is recorded yet. | |
| 157 | + |
| 158 | +The added random engine captures assert actual window-script counts, compare |
| 159 | +all 768 metric bytes to frozen array reference functions across 72 curves |
| 160 | +including M=0/M=1 and three timezones, and compare every published sink point. |
| 161 | +The helper tests assert 100,000 and 400,000 consumed points before checking |
| 162 | +allocation counts. These helper populations are **not** the requested real |
| 163 | +script-bar SUMMARY_ONLY growth test. The inherited selected sink fixture |
| 164 | +asserts real M at 1,000 and 1,000,000 bars; it is already bounded in Batch-B. |
| 165 | + |
| 166 | +Outstanding: real SUMMARY_ONLY capture at M=100,000 and M=400,000 with a |
| 167 | +growing-before/fixed-after receipt; whole-array presenter negotiation in |
| 168 | +SUMMARY_ONLY and successful FULL behavior; complete B1 replay/witness |
| 169 | +integration; and all remote verification. Do not promote R03 or R04 to a |
| 170 | +complete O(1) verdict from this slice. |
| 171 | + |
| 172 | +Lab CTest rows to run are `test_selected_capture_latches`, |
| 173 | +`test_selected_metrics_l17b`, `test_report_sink_selected_l17a`, |
| 174 | +`test_selected_window_report_integration`, `test_report_sink_equity`, |
| 175 | +`test_report_sink_scratch_curve` and `test_selected_production_routes`. |
| 176 | +The supervisor must also run preflight, complete all-target release/debug |
| 177 | +verification and the full corpus on the final integrated tree. |
| 178 | + |
| 179 | +No ABI exports, declaration counts, existing document anchors or hash leaves |
| 180 | +were changed. The new latches/window are local report state, not new durable |
| 181 | +execution state. The existing report-only metric field remains under its |
| 182 | +existing output-state waiver. Other lanes must preserve the bodies of |
| 183 | +`capture_presentation` and `fill_selected_report`, the two new internal |
| 184 | +headers, this document, the new test unit/registration and the added capture |
| 185 | +tests when merging. The only edits outside the owned function bodies are |
| 186 | +their two include directives; no other lane's function or header was edited. |
0 commit comments