Skip to content

Commit d220bd7

Browse files
luisleo526claude
andcommitted
cr150-q7: cooperative SIGTERM abort at a completed input-bar boundary
Flag-only native SIGTERM handler (volatile sig_atomic_t store, no calls in the handler). The pump stops after a completed input bar, the sink writer flushes the whole-record prefix and closes fd 4 without a trailer, and the invocation emits one typed run_aborted line (args.bars_count, args.strategy_started, bars.count and strategy_started mirrors) before exiting 1. Python blocks SIGTERM during startup so a pre-start signal is reported instead of lost, and never installs a Python signal callback. Catalog: run_aborted gains the two optional arguments and a boolean kind. Class ruling (CORE-TOP, lane commit 8d16e20, applied by hand here): no cancelled class, no retry field, fd 6 unused; its wording change in docker/run_failure_codes.json is part of this commit. (cherry picked from commit ffa4535 of lane core-cr150-batchc; applied on the O(1) integration line by the supervisor with the 8 conflicts resolved by lane o1w-merge0 round 10: README, run_json.py (Q7 fd ownership combined with the jailed fd/path/admission rules and selected named-sink support), ABI inventory union counted from the merged files (91 runtime definitions, 111 runtime-header declarations, 156 over six headers), c_abi.cpp, native_execution_consumer.hpp, tests/CMakeLists.txt with Q7's two registrations appended.) KNOWN OPEN: docker/run_failure_codes_diff.json, src/run_failure_registry.inc and docs/pages/run-failure-codes.md are STALE against the edited docker/run_failure_codes.json (Q7 run_aborted arguments and wording, the scratch_io_failed split); regenerate off the Mac and commit on top. NOT VERIFIED: nothing was compiled or run on this commit. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 3478350 commit d220bd7

35 files changed

Lines changed: 1439 additions & 252 deletions

‎README.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3333
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) is source-available under the PineForge Source License 1.2 from its release 1.3.0 (release 1.2.0 keeps the PineForge Source License 1.1, and releases up to 1.1.0 keep the PolyForm Noncommercial terms they shipped with); personal trading is free, and commercial use requires a separate license; the capital in an account a proprietary-trading firm or funded-trader program provides or allocates, including a challenge, evaluation or simulated account, is investment capital. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
3434
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies), measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3535
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists. Same on Linux and macOS.
36-
- **Yours to embed.** 153 `PF_API` declarations across six installed headers — `pineforge.h`'s 97 (77 of the 88 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
36+
- **Yours to embed.** 156 `PF_API` declarations across six installed headers — `pineforge.h`'s 100 (80 of the 91 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3737

3838
Recorded outputs and their additional API declarations described here are
3939
part of **1.5.0**.
@@ -519,11 +519,11 @@ is in [CHANGELOG.md](CHANGELOG.md).
519519

520520
The counts below describe the current API, including recorded outputs, the report sink and scratch-table selection; recorded outputs are not part of 1.4.0.
521521

522-
A newly generated strategy `.so` exposes 104 compiled-strategy `PF_API` declarations
523-
(88 runtime implementations plus sixteen generated exports) plus 43 native-host
524-
declarations: 147 `PF_API` exports in total; `use_bar_magnifier = true` adds
522+
A newly generated strategy `.so` exposes 107 compiled-strategy `PF_API` declarations
523+
(91 runtime implementations plus sixteen generated exports) plus 43 native-host
524+
declarations: 150 `PF_API` exports in total; `use_bar_magnifier = true` adds
525525
`strategy_declares_bar_magnifier` and a module that records [outputs](docs/outputs.md) its
526-
three outputs exports, bringing the inventories to at most 108 and 151. Older modules lack the six opt-in
526+
three outputs exports, bringing the inventories to at most 111 and 154. Older modules lack the six opt-in
527527
[checked-settings exports](docs/checked-settings.md). Older modules can also lack
528528
the two opt-in [execution-capability exports](docs/strategy-capabilities.md).
529529
In an optimized build `nm -gU` also shows libc++'s
@@ -557,6 +557,7 @@ inventories are pinned by `scripts/check_c_abi_runtime.py`:
557557
| `strategy_execution_contract` / `strategy_configure_native_v1` / `strategy_configure_native_fx_curve_v1` / `_fx_curve_ext_v1` | Query Legacy vs NativeMarketV1; apply the versioned native run specification (an invalid or refused base call returns `-1` and leaves the handle Failed; a cooperative `Aborted` handle may be reused with the same key and a higher run number); stage or clear an immutable native FX curve, `_fx_curve_ext_v1` also writing the typed refusal (`pf_native_fx_curve_error_e`) and the offending point's index |
558558
| `strategy_request_abort` / `strategy_last_run_status` | Cooperative abort of a run in progress; `0`=completed, `1`=aborted |
559559
| `strategy_set_scratch_tables_v1` | Copy disk-scratch or bounded-memory table policy and its eleven admission maxima while idle; NULL restores disk defaults (since 1.7.0) |
560+
| `pf_run_abort_install_sigterm_v1` / `pf_run_abort_requested_v1` / `pf_run_abort_restore_sigterm_v1` | Opt-in child invocation SIGTERM flag; stop after a completed input bar, retaining typed progress and an incomplete sink; install/restore with SIGTERM blocked |
560561
| `strategy_set_realtime_tail` | Live-runtime surface (ABI v4): the array's last bar is a still-forming tail — `barstate.islast=false`, `last_bar_index`/`last_bar_time` frozen at the horizon bar, no range-end row |
561562
| `strategy_set_probe_suppress_tail_logic` | ABI v4: the last bar runs only the broker's pre-`on_bar` steps (pending-order settlement, intraday-cap/loss checks) and returns — no `on_bar`, no margin-call / `process_orders_on_close` second pass / bracket-reissue processing (the range-end row is `strategy_set_realtime_tail`'s to skip; the flags are independent) |
562563
| `strategy_set_path_order` / `strategy_last_bar_dual_entry_path` | ABI v4: force the intrabar O→H/L→C leg order (`AUTO`/`HIGH_FIRST`/`LOW_FIRST`) for path-dependent fill probing; read which side won a same-bar dual-entry-stop arbitration |

‎docker/run_failure_codes.json‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1107,9 +1107,18 @@
11071107
"since": "1.5.0"
11081108
},
11091109
"run_aborted": {
1110-
"args": {},
1110+
"args": {
1111+
"bars_count": {
1112+
"kind": "integer",
1113+
"optional": true
1114+
},
1115+
"strategy_started": {
1116+
"kind": "boolean",
1117+
"optional": true
1118+
}
1119+
},
11111120
"class": "engine_fault",
1112-
"description": "The run was aborted on request.",
1121+
"description": "The run was aborted on request. Cooperative SIGTERM stops after the in-flight input bar completes, leaves no sink trailer, and exits 1. bars_count is the nonnegative number of fully completed input bars, mirrored as top-level bars.count, matching the runner's bars_processed population; strategy_started is a boolean mirrored at the top level. Both are optional for legacy abort raisers. There is no cancelled class: the abort path has exactly two internal callers, the orchestrator's hand-off at the service limit (the run is restarted with the same run id) and an operator stop. The abort line itself carries no retry semantics: it has no retryable field, and a retry is APP's decision from which caller stopped the run. On the abort path bars_count is diagnostic only and is never billed. No live-counter descriptor is used (fd 6 is not used).",
11131122
"english": [
11141123
"native run aborted"
11151124
],
@@ -2098,6 +2107,7 @@
20982107
}
20992108
},
21002109
"kinds": {
2110+
"boolean": "A JSON boolean.",
21012111
"identifier": "A name the compiled strategy spells as a literal (a built-in, an argument).",
21022112
"integer": "A JSON integer.",
21032113
"keyword": "A fixed keyword.",

‎docker/run_json.py‎

Lines changed: 134 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,9 @@
122122
is read only from strategy_get_last_error_code and its args from
123123
strategy_get_last_error_args. For report_sink_failed, reason, high_water_bytes
124124
and arena_bytes are also copied from args to the top level whenever present.
125+
For run_aborted, args.bars_count is mirrored as bars.count and the boolean
126+
args.strategy_started is mirrored as strategy_started. SIGTERM is observed
127+
after a completed input bar; fd 4 closes without a trailer before this line.
125128
A run failure from a library without the code
126129
getter prints a line without "code" and "args": the earlier
127130
{"engine":"pineforge","error":"<text>"} line byte for byte, or, for a run status
@@ -172,6 +175,7 @@
172175
import mmap
173176
import os
174177
import re
178+
import signal
175179
import stat
176180
import struct
177181
import sys
@@ -2187,6 +2191,74 @@ def __init__(self, text, code_args):
21872191
super().__init__(text, "strategy_library_incompatible", code_args)
21882192

21892193

2194+
class _SigtermAbort:
2195+
"""Block startup SIGTERM until the native flag-only handler can own it.
2196+
2197+
Python callbacks cannot interrupt a ctypes run at a bar boundary. No
2198+
Python signal handler is installed: a pending startup signal stays in the
2199+
kernel until attach(), or is reported before the strategy starts.
2200+
"""
2201+
2202+
def __init__(self):
2203+
self.lib = None
2204+
self.started = False
2205+
self.mask = (signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGTERM})
2206+
if hasattr(signal, "pthread_sigmask") else None)
2207+
2208+
def requested(self):
2209+
if self.lib is not None:
2210+
return bool(self.lib.pf_run_abort_requested_v1())
2211+
return self.mask is not None and signal.SIGTERM in signal.sigpending()
2212+
2213+
@staticmethod
2214+
def failure():
2215+
return RunFailure("native run aborted", "run_aborted",
2216+
{"bars_count": 0, "strategy_started": False})
2217+
2218+
def check_before_start(self):
2219+
if self.requested():
2220+
raise self.failure()
2221+
2222+
def attach(self, lib):
2223+
self.check_before_start()
2224+
names = ("pf_run_abort_install_sigterm_v1", "pf_run_abort_requested_v1",
2225+
"pf_run_abort_restore_sigterm_v1")
2226+
present = [hasattr(lib, name) for name in names]
2227+
if not any(present):
2228+
# Libraries predating this opt-in API retain their signal behavior.
2229+
if self.mask is not None:
2230+
signal.pthread_sigmask(signal.SIG_SETMASK, self.mask)
2231+
return
2232+
if not all(present) or self.mask is None:
2233+
raise StrategyLibraryError("the strategy library has an incomplete SIGTERM API",
2234+
{"reason": "symbol_missing"})
2235+
for name in names:
2236+
function = getattr(lib, name)
2237+
function.argtypes, function.restype = [], ctypes.c_int
2238+
if lib.pf_run_abort_install_sigterm_v1() != 0:
2239+
raise _engine_invariant("the invocation SIGTERM handler could not be installed")
2240+
self.lib = lib
2241+
signal.pthread_sigmask(signal.SIG_UNBLOCK, {signal.SIGTERM})
2242+
self.check_before_start()
2243+
2244+
def begin_call(self):
2245+
self.check_before_start()
2246+
self.started = True
2247+
2248+
def close(self):
2249+
if self.mask is None:
2250+
return
2251+
# Keep the native handler through cleanup AND the single stdout line.
2252+
# Consume blocked repeats before restoring an embedding caller's mask.
2253+
signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGTERM})
2254+
if self.lib is not None and self.lib.pf_run_abort_restore_sigterm_v1() != 0:
2255+
raise RuntimeError("the invocation SIGTERM handler could not be restored")
2256+
while signal.SIGTERM in signal.sigpending():
2257+
signal.sigwait({signal.SIGTERM})
2258+
signal.pthread_sigmask(signal.SIG_SETMASK, self.mask)
2259+
self.mask = None
2260+
2261+
21902262
def _cut_utf8(text: str, limit: int) -> str:
21912263
"""text cut to at most limit bytes of UTF-8, on a character boundary (a lone
21922264
surrogate counts as its three bytes and survives)."""
@@ -2229,12 +2301,20 @@ def failure_line(text, code=None, code_args=None) -> str:
22292301
if isinstance(value, str) else value)
22302302
for name, value in (code_args or {}).items()}
22312303
doc.update({name: doc["args"][name] for name in mirrors if name in doc["args"]})
2304+
if code == "run_aborted":
2305+
if "bars_count" in doc["args"]:
2306+
doc["bars"] = {"count": doc["args"]["bars_count"]}
2307+
if "strategy_started" in doc["args"]:
2308+
doc["strategy_started"] = doc["args"]["strategy_started"]
22322309
line = _dump_line(doc)
22332310
if len(line) > ERROR_LINE_MAX:
22342311
if code is not None and len(_dump_line({**doc, "error": ""})) > ERROR_LINE_MAX:
22352312
doc["args"] = {} # arguments no registry could hold: the code stays
22362313
for name in mirrors:
22372314
doc.pop(name, None)
2315+
if code == "run_aborted":
2316+
doc.pop("bars", None)
2317+
doc.pop("strategy_started", None)
22382318
low, high = 0, len(doc["error"].encode("utf-8", "surrogatepass"))
22392319
while low < high: # the longest cut whose line fits
22402320
mid = (low + high + 1) // 2
@@ -4602,6 +4682,9 @@ def _selected_run(args, inputs, overrides, input_tf, script_tf, bar_magnifier,
46024682
# 5. Execution begins before the strategy library loads.
46034683
_advance(phases, "execution")
46044684
lib = load_strategy(args.so)
4685+
abort = getattr(binary_inputs, "_sigterm_abort", None)
4686+
if abort is not None:
4687+
abort.attach(lib)
46054688
checked = uses_checked_settings(lib)
46064689
if args.outputs:
46074690
require_outputs(lib)
@@ -4650,6 +4733,8 @@ def _make_state():
46504733
abi.configure(state, request.start_ms, request.end_ms)
46514734
binding = _observer_binding(lib, state, phases.writer)
46524735
binding.attach()
4736+
if abort is not None:
4737+
abort.begin_call()
46534738
lib.run_backtest_full(
46544739
state, retained, fed_count,
46554740
input_tf, script_tf,
@@ -4880,13 +4965,21 @@ def main(argv=None) -> int:
48804965
tokens = sys.argv[1:] if argv is None else argv
48814966
if tokens and tokens[0] == "csv2bin":
48824967
return _csv2bin_main(tokens[1:])
4968+
abort = _SigtermAbort()
48834969
try:
48844970
with ExitStack() as binary_inputs:
4971+
binary_inputs._sigterm_abort = abort
48854972
return _main(argv, binary_inputs)
48864973
except RunFailure as failure:
4974+
if not abort.started and abort.requested():
4975+
failure = abort.failure()
48874976
write_failure(str(failure), failure.code, failure.code_args)
48884977
return failure.exit_status
48894978
except Exception as error:
4979+
if not abort.started and abort.requested():
4980+
failure = abort.failure()
4981+
write_failure(str(failure), failure.code, failure.code_args)
4982+
return 1
48904983
typed = _typed_failure(error)
48914984
if typed is not None:
48924985
write_failure(str(typed), typed.code, typed.code_args)
@@ -4895,6 +4988,8 @@ def main(argv=None) -> int:
48954988
write_failure(f"harness internal error: {type(error).__name__}: {error}",
48964989
"harness_internal_error")
48974990
return 1
4991+
finally:
4992+
abort.close()
48984993

48994994

49004995
def _legacy_abbreviations(argv, legacy_long, all_long):
@@ -5074,6 +5169,18 @@ def _main(argv, binary_inputs) -> int:
50745169
if args.ohlcv is not None and args.ohlcv_bin is not None:
50755170
raise _request_invalid("error: --ohlcv and --ohlcv-bin are mutually exclusive", "ohlcv_bin")
50765171
_validate_sink_options(args)
5172+
abort = getattr(binary_inputs, "_sigterm_abort", None)
5173+
if args.report_sink_fd is not None:
5174+
args._sink_fd_owner = _InheritedSinkFd(args.report_sink_fd)
5175+
def close_startup_sink():
5176+
# Jailed admission refusals retain caller ownership until the
5177+
# sink is installed. A startup SIGTERM still gives fd 4 EOF
5178+
# before the abort line, even before library/strategy creation.
5179+
if args.profile != "jailed" or (abort is not None and abort.requested()):
5180+
args._sink_fd_owner.close()
5181+
binary_inputs.callback(close_startup_sink)
5182+
if abort is not None:
5183+
abort.check_before_start()
50775184
selected = _selected_requested(args)
50785185
if selected and args.ohlcv_bin is not None:
50795186
raise _request_invalid("error: selected-window mode requires a CSV chart", "ohlcv_bin")
@@ -5116,6 +5223,8 @@ def _main(argv, binary_inputs) -> int:
51165223

51175224
_advance(phases, "execution")
51185225
lib = load_strategy(args.so)
5226+
if abort is not None:
5227+
abort.attach(lib)
51195228
checked = uses_checked_settings(lib)
51205229
if args.outputs:
51215230
require_outputs(lib)
@@ -5168,6 +5277,8 @@ def _make_state(*, body=False):
51685277
return st, sink
51695278

51705279
def _run(st, rep):
5280+
if abort is not None:
5281+
abort.begin_call()
51715282
lib.run_backtest_full(
51725283
st, chart["bars"], n,
51735284
input_tf, script_tf,
@@ -5502,7 +5613,8 @@ def _new_report_sink(lib, state, args, *, body=True):
55025613
if int(lib.pf_report_sink_version()) != 1:
55035614
raise _engine_invariant("the strategy library has an incompatible report sink ABI")
55045615
return _FdReportSink(lib, state, fd=args.report_sink_fd,
5505-
run_id=getattr(args, "run_id", ""))
5616+
run_id=getattr(args, "run_id", ""),
5617+
owner=getattr(args, "_sink_fd_owner", None))
55065618
if jailed:
55075619
return None
55085620
names = ("pf_report_sink_version", "strategy_set_report_sink_v1",
@@ -5522,13 +5634,31 @@ def _new_report_sink(lib, state, args, *, body=True):
55225634
path=getattr(args, "report_sink", None) if body else None)
55235635

55245636

5637+
class _InheritedSinkFd:
5638+
"""One close attempt shared by startup and native-state cleanup."""
5639+
5640+
def __init__(self, fd):
5641+
self.fd = fd
5642+
5643+
def close(self):
5644+
if self.fd is not None:
5645+
fd, self.fd = self.fd, None
5646+
try:
5647+
os.close(fd)
5648+
except OSError as error:
5649+
# Never retry a descriptor that the OS may already have reused.
5650+
if error.errno != errno.EBADF:
5651+
raise
5652+
5653+
55255654
class _FdReportSink:
55265655
"""Own the inherited endpoint; never create a path or consume its stream."""
55275656

55285657
fd_mode, named = True, True
55295658

5530-
def __init__(self, lib, state, *, fd, run_id):
5659+
def __init__(self, lib, state, *, fd, run_id, owner=None):
55315660
self.fd = fd
5661+
self._owner = owner if owner is not None else _InheritedSinkFd(fd)
55325662
self.run_id = run_id
55335663
self.counts = {}
55345664
self.summary = None
@@ -5574,13 +5704,8 @@ def verify(self, lib, state, report):
55745704

55755705
def close(self):
55765706
if self.fd is not None:
5577-
fd, self.fd = self.fd, None
5578-
try:
5579-
os.close(fd)
5580-
except OSError as error:
5581-
# No retry: a reused descriptor must never be closed a second time.
5582-
if error.errno != errno.EBADF:
5583-
raise
5707+
self.fd = None
5708+
self._owner.close()
55845709

55855710

55865711
class _ReportSink:

0 commit comments

Comments
 (0)