Skip to content

Commit 78f369e

Browse files
authored
Make the recorded output series limit an operator parameter, default no limit (#677)
The built-in refusal of more than 64 recorded output series is removed. A host may set a maximum before the strategy is created (a checked creation-time setting, run_json --limit-output-series N; 0 or absent means unlimited); only a declaration above that maximum is refused, with the same typed outputs_limit failure carrying the operator's positive value. The failure line of a creation refused by the operator limit carries a documented cause member. The run_json artifact-window worst case is computed from the declared series count instead of a fixed number, and the report-window cost per declared series and the report sink's wire bounds are documented. The engine keeps the limit as one optional member, so the pinned size of PineStrategyHost and the offset of its adapter move by 16 bytes in the layout probe, and the stream engine layout version advances to 2. The member is not const, so the engine stays copy-assignable. This replaces the series-count ceiling, the 10,551,296-byte worst case and the exact-64 statements that f891354 introduced in docs/outputs.md, docs/pages/memory-management.md and docs/pages/run-failure-codes.md, rewrites the final-artifact window rows of the memory page that 976d85a introduced, and replaces the row-cost bullet "A row costs 8 bytes per slot and is kept for the whole run or stream." that 70ddaa3 introduced. Unset or 0 means no limit; a positive value is the limit; anything else is a usage error. The previous fixed value (64) is available only through the opt-in TradingView-compatibility settings. The engine ABI layout version moves from 1 to 2 because the backtest engine gained a field. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- Commits: * afded56 Make the recorded output series limit an operator parameter, default no limit The built-in refusal of more than 64 recorded output series is removed. A host may set a maximum before the strategy is created (a checked creation-time setting, run_json --limit-output-series N; 0 or absent means unlimited); only a declaration above that maximum is refused, with the same typed outputs_limit failure carrying the operator's positive value. The failure line of a creation refused by the operator limit carries a documented cause member. The run_json artifact-window worst case is computed from the declared series count instead of a fixed number, and the report-window cost per declared series and the report sink's wire bounds are documented. The engine keeps the limit as one optional member, so the pinned size of PineStrategyHost and the offset of its adapter move by 16 bytes in the layout probe, and the stream engine layout version advances to 2. The member is not const, so the engine stays copy-assignable. This replaces the series-count ceiling, the 10,551,296-byte worst case and the exact-64 statements that f891354 introduced in docs/outputs.md, docs/pages/memory-management.md and docs/pages/run-failure-codes.md, rewrites the final-artifact window rows of the memory page that 976d85a introduced, and replaces the row-cost bullet "A row costs 8 bytes per slot and is kept for the whole run or stream." that 70ddaa3 introduced. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * dd01133 Export-list test: follow the two new runtime exports The generated ABI export list now holds 120 declared symbols, 100 of them implemented in the runtime, after the two output series limit exports. The test's pinned counts follow. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * 5ad98a9 Merge main (#671) into the output series limit change # Conflicts: # CHANGELOG.md * 6070469 Merge main (#676) into the output series limit change # Conflicts: # tests/CMakeLists.txt --- Merge note (L6 super top): train T2 (CR-556). Merge gate on head 6070469: pineforge/parity success from sweep exp-engine-pr677-6070469d-20261031 (134/134 cases, 7989 probes, tree e4f6138; cov lost 0, band1 +0/-0, band2 +0/-0, hard reg 0, leavers 0; neutral, accepted by merge-rule/v2; receipt a0f51a241a3c), plus pineforge/verify from CORE's full 8 on the same head. GKE PR CI green on this head.
1 parent 1d7ea83 commit 78f369e

27 files changed

Lines changed: 700 additions & 136 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ version number follows semantic versioning over the surfaces the
77

88
## Unreleased
99

10+
- Recorded output series are unlimited by default; 0 or omission of `--limit-output-series` means no limit, while a positive N is the operator's cap and any refusal reports that positive N as `max`. Report-window memory follows the declared count: 6 MiB + 64 KiB × S + a 64 KiB reader with the default I/O sizes.
1011
- Reading a map is cheaper: each map now keeps a fixed read cache of 8 slots
1112
for the keys and values it has just read, so a repeated `get` or `contains`
1213
of the same key no longer reads the Store again. The cache adds a fixed 528

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3333
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) is source-available under the PineForge Source License 1.2 from its release 1.3.0 (release 1.2.0 keeps the PineForge Source License 1.1, and releases up to 1.1.0 keep the PolyForm Noncommercial terms they shipped with); personal trading is free, and commercial use requires a separate license; the capital in an account a proprietary-trading firm or funded-trader program provides or allocates, including a challenge, evaluation or simulated account, is investment capital. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
3434
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies), measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3535
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists.
36-
- **Yours to embed.** 163 `PF_API` declarations across six installed headers — `pineforge.h`'s 107 (87 of the 98 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
36+
- **Yours to embed.** 165 `PF_API` declarations across six installed headers — `pineforge.h`'s 109 (89 of the 100 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3737

3838
Recorded outputs and their additional API declarations described here are
3939
part of **1.5.0**.
@@ -543,7 +543,7 @@ is in [CHANGELOG.md](CHANGELOG.md).
543543
The counts below describe the current API, including recorded outputs, the report sink and scratch-table selection; recorded outputs are not part of 1.4.0.
544544

545545
A newly generated strategy `.so` exposes 114 compiled-strategy `PF_API` declarations
546-
(98 runtime implementations plus sixteen generated exports) plus 43 native-host
546+
(100 runtime implementations plus sixteen generated exports) plus 43 native-host
547547
declarations: 157 `PF_API` exports in total; `use_bar_magnifier = true` adds
548548
`strategy_declares_bar_magnifier` and a module that records [outputs](docs/outputs.md) its
549549
three outputs exports, bringing the inventories to at most 118 and 161. Older modules lack the six opt-in

‎docker/run_json.py‎

Lines changed: 113 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3222,7 +3222,7 @@ def uses_checked_settings(lib) -> bool:
32223222
return True
32233223

32243224

3225-
def create_strategy(lib, checked: bool):
3225+
def _create_strategy(lib, checked: bool):
32263226
"""A new strategy handle, through strategy_create_checked when checked. A
32273227
creation that fails (a NULL handle, or a checked status other than OK) is
32283228
strategy_create_failed, whatever the cause: run_json reads no code from the
@@ -4981,7 +4981,7 @@ def _make_state():
49814981
"""A fresh, fully configured strategy state (the ordinary run's setup without the
49824982
legacy trade-start gate, which selected mode refuses); freed again if it fails."""
49834983
nonlocal syminfo_applied
4984-
st = create_strategy(lib, checked)
4984+
st = create_strategy(lib, checked, getattr(args, "max_output_series", None))
49854985
sink = None
49864986
try:
49874987
apply_settings(lib, st, inputs, overrides, checked)
@@ -5265,7 +5265,7 @@ def main(argv=None) -> int:
52655265
except RunFailure as failure:
52665266
if not abort.started and abort.requested():
52675267
failure = abort.failure()
5268-
write_failure(str(failure), failure.code, failure.code_args)
5268+
_write_run_failure(failure)
52695269
return failure.exit_status
52705270
except Exception as error:
52715271
if not abort.started and abort.requested():
@@ -5450,7 +5450,7 @@ def _main(argv, binary_inputs) -> int:
54505450
argv = _legacy_abbreviations(argv, pre_metrics_long, all_long)
54515451
if not argv:
54525452
ap.error("the following arguments are required: --so, --ohlcv")
5453-
args = ap.parse_args(argv)
5453+
args = _parse_output_series_limit(ap, argv)
54545454
if args.profile not in ("unjailed", "jailed"):
54555455
raise _request_invalid("PINEFORGE_RUN_JSON_PROFILE must be unjailed or jailed", "arguments")
54565456
missing = []
@@ -5559,7 +5559,7 @@ def _make_state(*, body=False, metrics_legacy=False):
55595559
Match the validation harness's setup and free a state whose
55605560
configuration fails before propagating its failure."""
55615561
nonlocal syminfo_applied
5562-
st = create_strategy(lib, checked)
5562+
st = create_strategy(lib, checked, getattr(args, "max_output_series", None))
55635563
sink = None
55645564
try:
55655565
apply_settings(lib, st, inputs, overrides, checked)
@@ -6525,7 +6525,7 @@ def __init__(self, sink, lib, state):
65256525
for entry in _manifest_entries(
65266526
self.manifest, "constants", ("index",))
65276527
}
6528-
resources = sink._spool_resources
6528+
resources = _output_series_resources(sink, self.slots)
65296529
directory = sink.directory
65306530
self.opens = _JSONColumn(resources, directory)
65316531
self.closes = _JSONColumn(resources, directory)
@@ -7434,9 +7434,9 @@ def _open_uncached_sink(path, *, raw=False):
74347434
# Six large windows for fixed files with outputs: report, equity/trade, open/close/event.
74357435
_ARTIFACT_WINDOW_BUDGET_BYTES = 6 * 1024 * 1024
74367436
_ARTIFACT_LARGE_WINDOW = 1024 * 1024
7437-
# TradingView Pine Script, Writing scripts > Limitations > Chart visuals > Plot limits:
7438-
# "A maximum of 64 plot counts are allowed per script." The engine refuses more series.
7439-
_OUTPUT_SERIES_LIMIT = 64
7437+
# These are default I/O sizes, not admission limits. After the fixed files,
7438+
# each declared output series adds one small window; the sink reader adds one.
7439+
# The per-run accountant receives the actual declaration, with no built-in cap.
74407440
# Indexed decoding bounds copied slices; sequential reads use the full file window.
74417441
_JSON_COLUMN_LOOKUP_READ_SIZE = 64 * 1024
74427442

@@ -7450,6 +7450,7 @@ class _ArtifactWindowBudget:
74507450

74517451
def __init__(self):
74527452
self.remaining = _ARTIFACT_WINDOW_BUDGET_BYTES
7453+
self.declared_series = 0
74537454
self.report_window = self.take()
74547455

74557456
def take(self):
@@ -7458,6 +7459,16 @@ def take(self):
74587459
return _ARTIFACT_LARGE_WINDOW
74597460
return _BoundedJSONFile.window
74607461

7462+
def declare_series(self, count):
7463+
self.declared_series = max(self.declared_series, count)
7464+
7465+
@property
7466+
def worst_case_bytes(self):
7467+
"""Default artifact windows plus the independently bounded sink reader."""
7468+
return (_ARTIFACT_WINDOW_BUDGET_BYTES
7469+
+ _BoundedJSONFile.window * self.declared_series
7470+
+ _UncachedSinkFile.window)
7471+
74617472

74627473
def _artifact_windows(owner):
74637474
"""Share one accountant through a sink and its report resource scopes."""
@@ -7869,8 +7880,8 @@ def _prepare_report_json(document, resources, args, sink):
78697880
raise sink._spool_failure
78707881
# Keep the final artifact provisional until auxiliary files have closed.
78717882
with ExitStack() as prepared_files:
7872-
prepared_files._window_budget = _artifact_windows(
7873-
sink if sink is not None else resources)
7883+
prepared_files._window_budget = _declared_artifact_windows(
7884+
sink if sink is not None else resources, _report_series_count(document))
78747885
prepared = _prepare_stream_report(
78757886
document, prepared_files,
78767887
sink.directory if sink is not None else args.spill_dir,
@@ -8256,7 +8267,7 @@ def _sink_outputs_block(lib, state, sink, resources, *, summary_only=False):
82568267
_manifest_entries(block["manifest"], "outputs", ("index", "id"))}
82578268
# After the report reservation: equity/trade spools, then open/close/event
82588269
# columns in creation order, then recorded series in slot order.
8259-
resources._window_budget = _artifact_windows(sink)
8270+
resources._window_budget = _declared_artifact_windows(sink, slots)
82608271
opens, closes, events = (_JSONColumn(resources, sink.directory) for _ in range(3))
82618272
columns = [_JSONColumn(resources, sink.directory) for _ in entries]
82628273
expected_row = 0
@@ -9246,5 +9257,95 @@ def execute(self, run, make_state, settings_receipt, abort):
92469257
return self.result(recovered_settings)
92479258

92489259

9260+
def _output_series_count(text):
9261+
if not re.fullmatch(r"[0-9]+", text) or int(text) > (1 << 63) - 1:
9262+
raise argparse.ArgumentTypeError("expected a nonnegative int64 series count")
9263+
return int(text)
9264+
9265+
9266+
def _parse_output_series_limit(parser, argv):
9267+
parser.add_argument(
9268+
"--limit-output-series", dest="max_output_series", type=_output_series_count,
9269+
default=None, metavar="N",
9270+
help="Maximum declared output series (N >= 0); 0 or omitted means no limit.")
9271+
return parser.parse_args(argv)
9272+
9273+
9274+
def _output_series_limit_functions(lib, maximum):
9275+
names = ("pf_output_series_limit_set_checked", "pf_output_series_limit_failure")
9276+
present = [hasattr(lib, name) for name in names]
9277+
if not any(present) and maximum is None:
9278+
return None
9279+
for name, found in zip(names, present):
9280+
if not found:
9281+
raise StrategyLibraryError(
9282+
f"the library lacks {name}; rebuild to configure output series",
9283+
{"reason": "symbol_missing", "missing": name})
9284+
setter, failure = (getattr(lib, name) for name in names)
9285+
setter.argtypes = [ctypes.POINTER(ctypes.c_int64), ctypes.c_char_p, ctypes.c_size_t]
9286+
setter.restype = ctypes.c_int
9287+
failure.argtypes = [ctypes.POINTER(ctypes.c_int64)]
9288+
failure.restype = ctypes.c_int
9289+
return setter, failure
9290+
9291+
9292+
def create_strategy(lib, checked: bool, max_output_series=None):
9293+
"""Apply the checked option before the generated constructor declares."""
9294+
if max_output_series == 0:
9295+
max_output_series = None
9296+
functions = _output_series_limit_functions(lib, max_output_series)
9297+
if functions is None:
9298+
return _create_strategy(lib, checked)
9299+
setter, read_failure = functions
9300+
maximum = None if max_output_series is None else ctypes.c_int64(max_output_series)
9301+
error = ctypes.create_string_buffer(_SETTINGS_ERROR_CAPACITY)
9302+
if setter(None if maximum is None else ctypes.byref(maximum),
9303+
error, len(error)) != PF_SETTINGS_OK:
9304+
raise _request_invalid(
9305+
"error: --limit-output-series: " + _c_text(error.value), "arguments")
9306+
try:
9307+
return _create_strategy(lib, checked)
9308+
except RunFailure as failure:
9309+
refused_max = ctypes.c_int64()
9310+
if (failure.code == "strategy_create_failed"
9311+
and read_failure(ctypes.byref(refused_max)) == 1):
9312+
failure.cause = {
9313+
"code": "outputs_limit",
9314+
"args": {"max": refused_max.value, "reason": "too_many_series"}}
9315+
raise
9316+
finally:
9317+
# Each handle has already snapshotted the option. Keep later creations
9318+
# in this interpreter unlimited unless their caller supplies a limit.
9319+
setter(None, error, len(error))
9320+
9321+
9322+
def _write_run_failure(failure):
9323+
cause = getattr(failure, "cause", None)
9324+
if cause is None:
9325+
write_failure(str(failure), failure.code, failure.code_args)
9326+
return
9327+
# This path has the fixed empty strategy_create_failed args and the
9328+
# runtime's scalar series-limit cause; no engine message is parsed.
9329+
document = json.loads(failure_line(str(failure), failure.code, failure.code_args))
9330+
document["cause"] = cause
9331+
_publish_failure(_dump_line(document) + "\n")
9332+
9333+
9334+
def _declared_artifact_windows(owner, series):
9335+
budget = _artifact_windows(owner)
9336+
budget.declare_series(series)
9337+
return budget
9338+
9339+
9340+
def _output_series_resources(sink, series):
9341+
resources = sink._spool_resources
9342+
resources._window_budget = _declared_artifact_windows(sink, series)
9343+
return resources
9344+
9345+
9346+
def _report_series_count(document):
9347+
return len((document.get("outputs") or {}).get("series", ()))
9348+
9349+
92499350
if __name__ == "__main__":
92509351
sys.exit(_host_main())

‎docker/run_json_codes_test.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1515,8 +1515,12 @@ def test_a_failed_checked_create_is_strategy_create_failed(harness, create, text
15151515
"same native tuple on the path/array route"),
15161516
("_csv2bin_main", "failure_line(str(failure), failure.code, failure.code_args)", 1,
15171517
"RunFailure or pf_bars.BarsError; BarsError code choices are scanned below"),
1518-
("main", "write_failure(str(failure), failure.code, failure.code_args)", 2,
1518+
("main", "write_failure(str(failure), failure.code, failure.code_args)", 1,
15191519
"caught RunFailure or _SigtermAbort.failure (literal run_aborted)"),
1520+
("_write_run_failure", "write_failure(str(failure), failure.code, failure.code_args)", 1,
1521+
"caught RunFailure without a creation-time series-limit cause"),
1522+
("_write_run_failure", "failure_line(str(failure), failure.code, failure.code_args)", 1,
1523+
"caught strategy_create_failed retains the runtime's typed series-limit cause"),
15201524
("main", "write_failure(str(typed), typed.code, typed.code_args)", 1,
15211525
"_typed_failure forwards only the helper classes whose producers are scanned"),
15221526
("_main", "RunFailure(*_post_boundary_engine_failure(failure, phases))", 1,

0 commit comments

Comments
 (0)