Skip to content

Commit 70ddaa3

Browse files
luisleo526claude
andauthored
Add opt-in recorded outputs and validate output manifests (#356)
* Recorded outputs: the recorder, C ABI group pf_outputs and run_json --outputs A module can record what it computes on each bar besides its trades: doubles in named slots per bar, doubles kept once per run, and events with an optional message, one row per bar the module publishes. - BacktestEngine gains the recorder (engine.hpp tail block, engine_report.cpp): declare_outputs, output_run_begin, output_bar, output_value, output_event, output_constant and the outputs_enabled_ flag for the host, readers for the C side. The host states the shape, each run's boundary and each row's identity (its open time); a row restarts at an equal open and a lower one fails the run. State sits outside both hashed regions; nothing a run computes reads it. - pineforge.h gains group pf_outputs: nine runtime exports (strategy_outputs_set_enabled and the readers, the size-prefixed pf_output_event_v1_t) and three per-module declarations (strategy_outputs_api_version, strategy_outputs_manifest, strategy_signal_safety_receipt). PF_ABI_VERSION stays 4; pf_report_t and every existing symbol are unchanged. - native_c_api.h lists the six writers as C++-only base-class seams at zero line shift; check_c_abi_runtime.py counts 73 runtime implementations and 93 declarations. - docker/run_json.py --outputs writes the record as the report's "outputs" block. - Tests: test_outputs_recorder (hand values, every refusal), test_outputs_stream_equivalence (stream == batch, the stale carried callback, negative controls), test_outputs_truncation (a cut batch records the prefix, negative control), test_outputs_run_reuse, test_outputs_off_identity (trades, equity and hashes equal with recording undeclared, off and on), test_outputs_c_api_c99, scripts/test_run_json_outputs.py and scripts/test_report_outputs_keys.py; the ci_verify row floors rise by those rows. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs: recorded outputs (lane SM-E1) docs/outputs.md describes the record (rows, values, events, run constants, colours as an encoding), the C++ host API and its row rules, the C readers, batches and streams, run_json --outputs, and the costs. ADR-0001 gains the recorder's codegen-ABI row and the C-surface row; the report-schema, abi-stability and docker pages describe the outputs block, the new exports and the epoch ruling; the stated PF_API counts (73 runtime, 93 in pineforge.h, 136 across both headers) follow the tree; CHANGELOG gains the Unreleased entry. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Recorded outputs: twin pf_output_phase_t to NativeRunPhase; restate counts check_native_c_api_surface.py requires every C enumeration to be twinned to its kernel enumeration or ruled C-only: pf_output_phase_t mirrors NativeRunPhase value for value (src/c_abi.cpp asserts the same), so it joins pf_native_run_phase_e as a twin. test_ci_verify.py pins the check_c_abi_runtime.py counts (93 declarations, 73 implementations), and the two remaining pages that state them follow the tree. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Recorded outputs: the truncation test carries a colour slot beside its run constant Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Recorded outputs: run-failure codes for every refusal; review fixes On the stable run-failure codes (#351): - strategy_outputs_set_enabled records its refusals through the failure record, code outputs_rejected (reason not_declared or run_in_progress), and clears it when it succeeds, so strategy_get_last_error_code reads "" after a switch that worked. - The recorder's refusals carry their codes: a broken precondition is engine_invariant (still a std::logic_error), the event queue or the rows reaching INT_MAX are pine_runtime_limit (limit output_events or output_rows; the row cap is new and keeps bar_index an int32_t), a bar recalculated after a caller cleared its events is outputs_rejected (reason recalculated_after_clear). - docker/run_failure_codes.json gains outputs_rejected, the two limits, strategy_library_incompatible's reasons outputs_api_mismatch and outputs_manifest_invalid, and the recorder's texts; the registry, the catalog diff and the code tables of run-failure-codes.md are regenerated. - run_json: OutputsError is a RunFailure, written as the coded failure line. --outputs requires the manifest and every reader, checks the outputs API version, refuses a manifest that is not a JSON object, names a slot the library does not record or lacks an output an event names, and copies each event's message when it reads the event. Tests: test_outputs_recorder checks each refusal's code and the getters, and a stream ended on a forming bar records confirmed 0; run_json_codes_test.py and test_report_outputs_keys.py cover the new refusals; test_run_json_outputs.py asserts the coded lines and that a colour slot is written as JSON integers. Test comments say what they test instead of naming internal design records. Docs: the switch's codes and that a call changing nothing answers 0 (pineforge.h, docs/outputs.md); a message holding a NUL reads to its first NUL while message_hash64 covers every recorded byte; call output_bar first in every calculation that writes; the ADR row names OutputEvent and the readers. Also restores the endif() of the per-entry-exit fixture block in tests/CMakeLists.txt, which the rebase onto main left out. Revises four sentences of the lane SM-E1 docs commit: the ADR row's first cell (`declare_outputs`, `output_run_begin`, `output_bar`, `output_value`, `output_event`, `output_constant`, `outputs_enabled_`) now also names the readers and OutputEvent; docs/outputs.md's "Fails (throws `std::logic_error`) when" list and its "`open_ms` is below the last row's" rule now give each refusal's code; report-schema.md's "`--outputs` on a module that records nothing" line now shows the coded failure line. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Recorded outputs: a neutral outputs_limit code; the switch keeps the failure record The recorder is a generic kernel unit, so its two INT_MAX caps no longer raise pine_runtime_limit. The catalog gains outputs_limit (class strategy_limit, "A recorded-outputs capacity limit was reached during the run.") with reason too_many_events or too_many_rows and max; both caps raise it. pine_runtime_limit is main's entry again (limits map_pairs, max_bars_back, matrix_elements, udt_objects), so its row of run-failure-codes.md reads 4 values again. The enum, the registry, the catalog diff and the page's tables are regenerated. test_outputs_recorder checks the caps' exception as their sites throw it, as test_run_failure_codes does for recalc_cap, and that a reason outside the list reads engine_invariant. strategy_outputs_set_enabled now follows strategy_set_trace_enabled and the checked settings setters, which never clear a failure record: a success leaves the record as it was, and a refusal keeps a failed run's own text and code, as the C boundary keeps them for any refused call. Tests cover a success after a refusal, and a success and a refusal after a failed run. docs/outputs.md and engine.hpp say which exception type each failure throws (std::runtime_error for the caps and a recalculation after a clear), and pineforge.h and docs/outputs.md state the switch's record rule. Revised sentences: docs/outputs.md's "The event queue and the rows reaching `INT_MAX` are `pine_runtime_limit`", "A recalculation of a bar after a caller cleared that bar's events is `outputs_rejected`", the `strategy_outputs_set_enabled(s, on)` row ("0 with no failure left behind") and the costs line "The event queue holds at most `INT_MAX` events between clears"; the pine_runtime_limit row of run-failure-codes.md. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test: cover malformed output manifest entries at harness boundary * fix: validate consumed output manifest values before lookup * test: bind recorded outputs to typed provenance at the harness boundary * docs: distinguish planned recorded outputs from the 1.4.0 release * docs: mark recorded outputs as unreleased for 1.5.0 * docs: clarify unreleased output failure reasons --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 31a40a1 commit 70ddaa3

39 files changed

Lines changed: 5268 additions & 154 deletions

‎CHANGELOG.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,33 @@ README's *Releases* section and on the GitHub releases page. From 1.0.0 the
55
version number follows semantic versioning over the surfaces the
66
[public contract](docs/pages/public-contract.md) lists.
77

8+
## Unreleased
9+
10+
- **Recorded outputs.** A module can record what it computes on each bar
11+
besides its trades: doubles in named slots per bar, doubles kept once per
12+
run, and events with an optional message, one row per bar the module
13+
publishes (`docs/outputs.md`). A generated module or a C++ host writes them
14+
through new protected `BacktestEngine` members (`declare_outputs`,
15+
`output_run_begin`, `output_bar`, `output_value`, `output_event`,
16+
`output_constant`); a caller reads them through nine new runtime exports of
17+
`pineforge.h`, group `pf_outputs` (`strategy_outputs_set_enabled`, the
18+
`strategy_outputs_*` readers, the size-prefixed `pf_output_event_v1_t`),
19+
present in every module built on this engine, and a recording module adds
20+
`strategy_outputs_api_version`, `strategy_outputs_manifest` and
21+
`strategy_signal_safety_receipt`. `docker/run_json.py --outputs` writes the
22+
record as the report's `outputs` block. Every refusal carries a run-failure
23+
code: the new `outputs_rejected` (reasons `not_declared`, `run_in_progress`,
24+
`recalculated_after_clear`), the new `outputs_limit` (reasons
25+
`too_many_events`, `too_many_rows`), `strategy_library_incompatible` with the
26+
new reasons `outputs_api_mismatch` and `outputs_manifest_invalid`, and
27+
`engine_invariant` for a broken recorder precondition (all additions to
28+
`docker/run_failure_codes.json`). Recording is off until a caller
29+
turns it on, and nothing a run computes depends on it: trades, reports,
30+
metrics and every hash are byte-identical, `pf_report_t` and every existing
31+
symbol are unchanged, and `PF_ABI_VERSION` stays 4. The new members enter
32+
`engine_script_run_v19` without an epoch; a C++ object compiled against an
33+
earlier v19 header is rebuilt with the library.
34+
835
## 1.4.0 — 2026-10-08
936

1037
**[1.4.0](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.4.0)**,

‎README.md‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,10 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3333
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) is source-available under the PineForge Source License 1.2 from its release 1.3.0 (release 1.2.0 keeps the PineForge Source License 1.1, and releases up to 1.1.0 keep the PolyForm Noncommercial terms they shipped with); personal trading is free, and commercial use requires a separate license; the capital in an account a proprietary-trading firm or funded-trader program provides or allocates, including a challenge, evaluation or simulated account, is investment capital. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
3434
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time, measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3535
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists. Same on Linux and macOS.
36-
- **Yours to embed.** 124 `PF_API` declarations across two headers — `pineforge.h`'s 81 (64 runtime exports and up to 17 per-strategy functions a generated module defines) and `native_c_api.h`'s 43 native-host declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
36+
- **Yours to embed.** 136 `PF_API` declarations across two headers — `pineforge.h`'s 93 (73 runtime exports and up to 20 per-strategy functions a generated module defines) and `native_c_api.h`'s 43 native-host declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
37+
38+
Recorded outputs and their additional API declarations described here are
39+
unreleased work targeted for **1.5.0**; they are not part of 1.4.0.
3740

3841
**[1.4.0](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.4.0)
3942
was released on 2026-10-08**, paired with
@@ -501,11 +504,13 @@ API, the script ABI epoch and the pairing with codegen — is the
501504
[public contract](docs/pages/public-contract.md); what a 0.x user must act on
502505
is in [CHANGELOG.md](CHANGELOG.md).
503506

504-
A newly generated strategy `.so` exposes 80 compiled-strategy `PF_API` declarations
505-
(64 runtime implementations plus sixteen generated exports) plus 43 native-host
506-
declarations: 123 `PF_API` exports in total; a script that declares
507-
`use_bar_magnifier = true` also exports `strategy_declares_bar_magnifier`, bringing
508-
the inventories to 81 and 124. Older modules lack the six opt-in
507+
The counts below include recorded outputs and describe the unreleased API targeted for 1.5.0; recorded outputs are not part of 1.4.0.
508+
509+
A newly generated strategy `.so` exposes 89 compiled-strategy `PF_API` declarations
510+
(73 runtime implementations plus sixteen generated exports) plus 43 native-host
511+
declarations: 132 `PF_API` exports in total; `use_bar_magnifier = true` adds
512+
`strategy_declares_bar_magnifier` and a module that records [outputs](docs/outputs.md) its
513+
three outputs exports, bringing the inventories to at most 93 and 136. Older modules lack the six opt-in
509514
[checked-settings exports](docs/checked-settings.md). Older modules can also lack
510515
the two opt-in [execution-capability exports](docs/strategy-capabilities.md).
511516
In an optimized build `nm -gU` also shows libc++'s

‎docker/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,33 @@ docker run --rm \
266266
or an index naming no symbol: the key is absent and the report is what it was
267267
before this variable existed, apart from `elapsed_seconds`.
268268

269+
### Recorded outputs (the harness's `--outputs`)
270+
271+
This section describes unreleased support targeted for **1.5.0**. Released
272+
1.4.0 images do not provide recorded outputs.
273+
274+
A library compiled to record outputs keeps, besides its trades, values per bar,
275+
values per run and events with an optional message (`docs/outputs.md` in the
276+
engine repository). `run_json.py --outputs` turns recording on and writes the
277+
record as the report's `outputs` key, just before `fingerprint`, with
278+
`"outputs": true` in `applied_runtime`: the bars' times, each listed series,
279+
the run constants, each horizontal level's price and every event (sequence,
280+
output, bar, phase, value, message). Its keys are listed in the engine's
281+
report-schema page. Without the flag the report is unchanged. On a library that
282+
records no outputs the run fails before it starts: one line
283+
`{"engine":"pineforge","error":"--outputs: this library records no outputs (compile the script as an indicator, or with outputs on)","code":"outputs_rejected","args":{"reason":"not_declared"}}`
284+
on stdout, harness exit 1. With `--bench` the timed runs record too.
285+
286+
Recording preserves the typed provenance rules described below. The runtime
287+
fact `"outputs": true` remains a boolean in both `applied_runtime` and the
288+
fingerprinted provenance; input and override normalization runs before the
289+
fingerprint is built. Recording does not certify a foreign source or replace
290+
the native settings receipt. Without `--outputs`, no output API is called and
291+
no `"outputs": false` runtime field is added. A provenance normalization
292+
failure keeps the existing `fingerprint: null` behavior, including a recorded
293+
output block when recording succeeded; an invalid output manifest instead
294+
produces the structured failure line and no partial report.
295+
269296
The engine catches run errors (TF mismatch, unsupported emulation flags,
270297
unknown-input-TF, etc.) into `strategy_get_last_error()` and, with the
271298
1.4.0 runtime, its code/args getters. The failure line below describes both

‎docker/run_failure_codes.json‎

Lines changed: 72 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,19 @@
249249
"null admission history event",
250250
"null strategy",
251251
"null strategy, key or value",
252+
"outputs: bar opened at {open_ms} is before the last recorded bar",
253+
"outputs: declare_outputs repeated with other counts",
254+
"outputs: declare_outputs while a run is in progress",
255+
"outputs: declared counts must not be negative",
256+
"outputs: event sequence overflow",
257+
"outputs: output {output} is out of range",
258+
"outputs: output_bar before any output_run_begin",
259+
"outputs: output_constant before any output_run_begin",
260+
"outputs: output_event with no open bar",
261+
"outputs: output_value with no open bar",
262+
"outputs: run constant {index} changed within a run",
263+
"outputs: run constant {index} is out of range",
264+
"outputs: series slot {slot} is out of range",
252265
"params_json is reserved; use checked setters",
253266
"quantity fraction requires a positive finite denominator",
254267
"quantity grid requires a finite positive step",
@@ -453,6 +466,51 @@
453466
"retryable": true,
454467
"since": "1.4.0"
455468
},
469+
"outputs_limit": {
470+
"args": {
471+
"max": {
472+
"kind": "integer"
473+
},
474+
"reason": {
475+
"kind": "vocab",
476+
"values": [
477+
"too_many_events",
478+
"too_many_rows"
479+
]
480+
}
481+
},
482+
"class": "strategy_limit",
483+
"description": "A recorded-outputs capacity limit was reached during the run.",
484+
"english": [
485+
"outputs: the event queue is full",
486+
"outputs: the row count is at its limit"
487+
],
488+
"retryable": false,
489+
"since": "1.5.0"
490+
},
491+
"outputs_rejected": {
492+
"args": {
493+
"reason": {
494+
"kind": "vocab",
495+
"values": [
496+
"not_declared",
497+
"run_in_progress",
498+
"recalculated_after_clear"
499+
]
500+
}
501+
},
502+
"class": "input",
503+
"description": "Recorded outputs were asked of a strategy library that records none, or the recording cannot be honoured as asked.",
504+
"english": [
505+
"--outputs: this library records no outputs (compile the script as an indicator, or with outputs on)",
506+
"--outputs: {message}",
507+
"outputs: bar {bar} was recalculated after its events were cleared",
508+
"outputs: recording cannot change during a run",
509+
"outputs: this module declares no outputs"
510+
],
511+
"retryable": false,
512+
"since": "1.5.0"
513+
},
456514
"pine_array_error": {
457515
"args": {
458516
"collection": {
@@ -1117,13 +1175,26 @@
11171175
"abi_mismatch",
11181176
"symbol_missing",
11191177
"setter_missing",
1120-
"settings_api_mismatch"
1178+
"settings_api_mismatch",
1179+
"outputs_api_mismatch",
1180+
"outputs_manifest_invalid"
11211181
]
11221182
}
11231183
},
11241184
"class": "engine_fault",
11251185
"description": "The compiled strategy library does not match the harness.",
11261186
"english": [
1187+
"--outputs: event {index} could not be read",
1188+
"--outputs: series slot {slot} could not be read",
1189+
"--outputs: the bar times could not be read",
1190+
"--outputs: the library lacks {export}, which recording needs; rebuild.",
1191+
"--outputs: the library returned no outputs manifest: {error}",
1192+
"--outputs: the library's outputs API version is {version}, the harness reads 1; rebuild.",
1193+
"--outputs: the outputs manifest is not JSON: {error}",
1194+
"--outputs: the outputs manifest is not a JSON object",
1195+
"--outputs: the outputs manifest lists no output {index}, which an event names",
1196+
"--outputs: the outputs manifest names series slot {slot}, which the library does not record",
1197+
"--outputs: the outputs manifest's {key} is not a list of entries with {fields}",
11271198
"cannot load the strategy library: {error}",
11281199
"checked settings API mismatch: the strategy library must export strategy_settings_api_version() == 1, strategy_create_checked, strategy_set_input_checked and strategy_set_override_checked, or none of them; rebuild.",
11291200
"pineforge ABI mismatch: .so reports {abi}, harness expects 4; rebuild.",

0 commit comments

Comments
 (0)