Skip to content

Commit 2688775

Browse files
luisleo526claude
andcommitted
sink-fd: fd report transport diagnostics, jailed arena admission, jailed serialization, explicit report-mode refusal, jail write test (C15, C16, C22, stage 0)
TX-SINKFD (SINK-SPLIT stages 0+A+B, engine side). One commit for the whole engine-side seam because the hunks of docker/run_json.py and the C runtime interleave; see the lane report for the split by class. C15 (test_run_json_sink_fd_l22v, test_run_json_sink_fd_native_l22v, test_report_sink_summary): the fd transport returned its document before the annualization block and took fd ownership before the required API symbols were checked. The writer now retains the two emitted E endpoints and the T-row entry_bar_index <= 0 predicate, published only after the trailer, and a new additive runtime query strategy_report_sink_diagnostics_v1 (struct pf_report_sink_diagnostics_v1, 72 bytes) hands them to run_json. No wire byte changes, no callback is replayed, the existing receipt layouts are unchanged. Stage 0 admission: a new additive runtime export strategy_require_report_scratch_arena_v1 attaches the inherited fd-5 arena and refuses an absent arena with report_sink_failed/arena_backing, never creating a backing file or falling back to TMPDIR. run_json calls it on every jailed handle before sink installation. The guard also refuses --report-sink-fd 5 for every profile, and jailed sink paths and --spill-dir before any library is loaded. C16 (test_run_json_jailed_profile): the jailed profile now serializes the report fully into an owned StringIO through the existing lazy-array serializer instead of creating a spool; the test's CDLL mock forwards the libc load to the real CDLL. C22 (test_o1_report_mode_unsupported): --report-mode FULL|SUMMARY_ONLY| STREAMED is parsed and every explicit request is refused with the typed report_mode_unsupported before feeds, library loading or handle creation; omitting the option keeps the legacy route. No provider bridge exists at this base, so no positive envelope is claimed. New test_report_sink_jail_write: a native host runs the same callback with and without RLIMIT_FSIZE 0; the unlimited run must write, the limited run must get EFBIG with the file empty, both complete the v1 fd sink with identical bytes and the work directory stays empty. NOT in this commit: C1 (ruling S) and the unjailed native fd fixture (ruling F) are unresolved by design; stage B's Node caller is in the APP repo. Public/runtime export counts become 113/93 (were 111/91); the inventory, README table, docs and the checker self-test are updated together. Nothing was built or run when this was committed; the proof is the lab release+debug job. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> (cherry picked from commit 03d3d10)
1 parent 8d2f2d3 commit 2688775

19 files changed

Lines changed: 810 additions & 52 deletions

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -567,6 +567,8 @@ inventories are pinned by `scripts/check_c_abi_runtime.py`:
567567
| `strategy_get_last_error` | The latest runtime error |
568568
| `strategy_get_last_error_code` / `strategy_get_last_error_args` | The latest failure's stable code from the closed [run-failure vocabulary](docs/pages/run-failure-codes.md) (`""` when the run succeeded) and its typed arguments as canonical JSON; behind `PINEFORGE_HAS_RUN_FAILURE_CODES_V1` |
569569
| `pf_report_scratch_arena_stats` | Process-wide scratch-arena peak claimed bytes and exact supplied size; query after finalization and before detach for stdout diagnostics, outside sink bytes |
570+
| `strategy_require_report_scratch_arena_v1` | Jailed startup admission: attach/validate the inherited fd-5 arena, or refuse with typed `report_sink_failed/arena_backing`; no temporary-file fallback |
571+
| `strategy_report_sink_diagnostics_v1` | Fixed-size presented equity endpoints and first-bar trade provenance after a completed sink; stdout annualization needs no sink read-back |
570572
| `pf_version_get` / `pf_version_string` / `pf_abi_version` | Runtime version, version string, struct-layout version (`PF_ABI_VERSION == 4`) |
571573
| `strategy_execution_contract` / `strategy_configure_native_v1` / `strategy_configure_native_fx_curve_v1` / `_fx_curve_ext_v1` | Query Legacy vs NativeMarketV1; apply the versioned native run specification (an invalid or refused base call returns `-1` and leaves the handle Failed; a cooperative `Aborted` handle may be reused with the same key and a higher run number); stage or clear an immutable native FX curve, `_fx_curve_ext_v1` also writing the typed refusal (`pf_native_fx_curve_error_e`) and the offending point's index |
572574
| `strategy_request_abort` / `strategy_last_run_status` | Cooperative abort of a run in progress; `0`=completed, `1`=aborted |

‎docker/run_json.py‎

Lines changed: 106 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -157,9 +157,17 @@
157157
startup) installs bounded-memory scratch tables on every fresh handle and
158158
keeps the report sink OFF unless --report-sink-fd explicitly selects a stream.
159159
Path sinks (--report-sink) are refused in this profile. Scratch-table admission
160-
is required for fd sinks too. It requires strategy_set_scratch_tables_v1; an
161-
older library is a typed incompatibility. The default profile is unjailed.
160+
is required for fd sinks too. Every jailed handle also requires an attached
161+
scratch arena (PF_SCRATCH_ARENA_FD=5); absent/invalid backing is a typed refusal
162+
before execution, never a temporary-file fallback. --spill-dir is refused,
163+
and --report-sink-fd 5 is reserved for the arena in every profile.
164+
An older library missing either admission API is a typed incompatibility.
165+
The default profile is unjailed.
162166
See docs/pages/scratch-tables.md for the policy tier and remaining jail work.
167+
168+
--report-mode FULL|SUMMARY_ONLY|STREAMED requests the opt-in report envelope.
169+
This ctypes host has no envelope provider bridge and refuses explicit requests
170+
as report_mode_unsupported before execution. Omitting it keeps the legacy route.
163171
"""
164172
from __future__ import annotations
165173

@@ -4834,6 +4842,7 @@ def _make_state():
48344842
enable_outputs(lib, st)
48354843
if args.profile == "jailed":
48364844
apply_scratch_tables(lib, st, default_scratch_manifest(fed_count))
4845+
_require_jailed_arena(lib, st)
48374846
sink = _new_report_sink(lib, st, args)
48384847
except BaseException:
48394848
try:
@@ -5001,15 +5010,7 @@ def _make_state():
50015010
# 6. Serialization begins; the timing is known now and is outside the provenance.
50025011
_advance(phases, "serialization")
50035012
out["diagnostics"]["phase_timing"] = phases.writer.phase_timing()
5004-
if sink is not None and getattr(sink, "fd_mode", False):
5005-
buffer = io.StringIO()
5006-
json.dump(out, buffer, separators=(",", ":"), allow_nan=False)
5007-
buffer.write("\n")
5008-
buffer.seek(0)
5009-
else:
5010-
buffer = _prepare_stream_report(
5011-
out, binary_inputs, sink.directory if sink is not None else args.spill_dir,
5012-
sink_errors=sink is not None)
5013+
buffer = _prepare_report_json(out, binary_inputs, args, sink)
50135014
if output_files is not None:
50145015
output_files.close() # surface column close failures before stdout
50155016
except Exception as error:
@@ -5264,8 +5265,11 @@ def _main(argv, binary_inputs) -> int:
52645265
pre_profile_long = tuple(s for s in ap._option_string_actions if s.startswith("--"))
52655266
ap.add_argument("--profile", choices=("unjailed", "jailed"), default=environment_profile,
52665267
help="Host profile (default: PINEFORGE_RUN_JSON_PROFILE or unjailed). "
5267-
"jailed requires bounded scratch tables; only an explicit "
5268+
"jailed requires bounded scratch tables and the inherited arena; only an explicit "
52685269
"--report-sink-fd enables a sink, and --report-sink paths are refused.")
5270+
ap.add_argument("--report-mode", choices=("FULL", "SUMMARY_ONLY", "STREAMED"), default=None,
5271+
help="Request a pineforge-report/v1 envelope; requires a provider "
5272+
"bridge for this host. Absent keeps the legacy report route.")
52695273
all_long = tuple(s for s in ap._option_string_actions if s.startswith("--"))
52705274
argv = _legacy_abbreviations(argv, legacy_long, all_long)
52715275
argv = _legacy_abbreviations(argv, pre_profile_long, all_long)
@@ -5288,6 +5292,17 @@ def _main(argv, binary_inputs) -> int:
52885292
missing.append("--ohlcv")
52895293
if missing:
52905294
ap.error("the following arguments are required: " + ", ".join(missing))
5295+
if args.report_mode is not None:
5296+
# report_mode.h declares data, not a C provider bridge. This ctypes
5297+
# host has only the legacy run/report API; neither ABI 4 nor the v1
5298+
# sink is an envelope capability (o1-report-contract.md, transport).
5299+
# Refuse every explicit envelope request before feeds, handles or
5300+
# callbacks. A future bridge must land its implementation and
5301+
# negotiation together, including explicit FULL envelope reads.
5302+
raise RunFailure(f"unsupported report mode: {args.report_mode} "
5303+
"(no pineforge-report/v1 provider bridge for this host)",
5304+
"report_mode_unsupported",
5305+
{"mode": args.report_mode, "contract": "pineforge-report/v1"})
52915306
_refuse_jailed_path_sink(args)
52925307
if args.capabilities_json:
52935308
if args.report_sink_fd is not None:
@@ -5299,13 +5314,15 @@ def _main(argv, binary_inputs) -> int:
52995314
_validate_sink_options(args)
53005315
abort = getattr(binary_inputs, "_sigterm_abort", None)
53015316
if args.report_sink_fd is not None:
5302-
args._sink_fd_owner = _InheritedSinkFd(args.report_sink_fd)
5317+
args._sink_fd_owner = None
53035318
def close_startup_sink():
5304-
# Jailed admission refusals retain caller ownership until the
5305-
# sink is installed. A startup SIGTERM still gives fd 4 EOF
5306-
# before the abort line, even before library/strategy creation.
5307-
if args.profile != "jailed" or (abort is not None and abort.requested()):
5319+
# Admission refusals retain caller ownership until the fd API
5320+
# has been checked. Startup SIGTERM still delivers EOF before
5321+
# the abort line, including before library/strategy creation.
5322+
if args._sink_fd_owner is not None:
53085323
args._sink_fd_owner.close()
5324+
elif abort is not None and abort.requested():
5325+
_InheritedSinkFd(args.report_sink_fd).close()
53095326
binary_inputs.callback(close_startup_sink)
53105327
if abort is not None:
53115328
abort.check_before_start()
@@ -5394,6 +5411,7 @@ def _make_state(*, body=False):
53945411
enable_outputs(lib, st)
53955412
if args.profile == "jailed":
53965413
apply_scratch_tables(lib, st, default_scratch_manifest(n))
5414+
_require_jailed_arena(lib, st)
53975415
sink = _new_report_sink(lib, st, args, body=body)
53985416
except BaseException:
53995417
try:
@@ -5566,15 +5584,7 @@ def _run(st, rep):
55665584
# Serialized whole before any byte is written (the same JSON encoder), so
55675585
# a failure while it is built never follows half a report.
55685586
try:
5569-
if sink is not None and getattr(sink, "fd_mode", False):
5570-
buffer = io.StringIO()
5571-
json.dump(out, buffer, separators=(",", ":"), allow_nan=False)
5572-
buffer.write("\n")
5573-
buffer.seek(0)
5574-
else:
5575-
buffer = _prepare_stream_report(
5576-
out, binary_inputs, sink.directory if sink is not None else args.spill_dir,
5577-
sink_errors=sink is not None)
5587+
buffer = _prepare_report_json(out, binary_inputs, args, sink)
55785588
except ValueError as error:
55795589
# Match only the encoder's allow_nan=False refusal, including the
55805590
# versions that append repr(value); other ValueErrors keep their code.
@@ -5651,11 +5661,19 @@ class ReportSinkSummaryC(ctypes.Structure):
56515661
("nonfinite_t_field", ctypes.c_int32), ("complete", ctypes.c_uint32)]
56525662

56535663

5664+
class ReportSinkDiagnosticsC(ctypes.Structure):
5665+
"""Independent 72-byte endpoint/provenance query; existing receipts stay fixed."""
5666+
_fields_ = [("struct_size", ctypes.c_uint32), ("version", ctypes.c_uint32),
5667+
("count_e", ctypes.c_int64), ("first", EquityPointC), ("last", EquityPointC),
5668+
("trade_at_start", ctypes.c_uint32), ("complete", ctypes.c_uint32)]
5669+
5670+
56545671
def _bind_report_sink(lib) -> None:
56555672
signatures = {
56565673
"pf_report_sink_version": (ctypes.c_uint32, []),
56575674
"pf_report_scratch_arena_stats": (ctypes.c_int, [
56585675
ctypes.POINTER(ctypes.c_uint64), ctypes.POINTER(ctypes.c_uint64)]),
5676+
"strategy_require_report_scratch_arena_v1": (ctypes.c_int, [ctypes.c_void_p]),
56595677
"strategy_set_report_sink_v1": (ctypes.c_int, [
56605678
ctypes.c_void_p, ctypes.POINTER(ReportSinkC)]),
56615679
"strategy_set_report_sink_fd_v1": (ctypes.c_int, [
@@ -5664,6 +5682,8 @@ def _bind_report_sink(lib) -> None:
56645682
ctypes.c_void_p, ctypes.POINTER(ReportSinkReceiptC)]),
56655683
"strategy_report_sink_summary_v1": (ctypes.c_int, [
56665684
ctypes.c_void_p, ctypes.POINTER(ReportSinkSummaryC)]),
5685+
"strategy_report_sink_diagnostics_v1": (ctypes.c_int, [
5686+
ctypes.c_void_p, ctypes.POINTER(ReportSinkDiagnosticsC)]),
56675687
}
56685688
for name, (restype, argtypes) in signatures.items():
56695689
if hasattr(lib, name):
@@ -5689,6 +5709,24 @@ def _refuse_jailed_path_sink(args):
56895709
and getattr(args, "report_sink", None) is not None):
56905710
raise _request_invalid("error: --profile jailed refuses --report-sink paths; "
56915711
"use --report-sink-fd", "arguments")
5712+
if (getattr(args, "profile", "unjailed") == "jailed"
5713+
and getattr(args, "spill_dir", None) is not None):
5714+
raise _request_invalid("error: --profile jailed refuses --spill-dir; "
5715+
"supply the inherited scratch arena", "arguments")
5716+
5717+
5718+
def _require_jailed_arena(lib, state):
5719+
admission = getattr(lib, "strategy_require_report_scratch_arena_v1", None)
5720+
if admission is None:
5721+
raise StrategyLibraryError("the jailed profile requires "
5722+
"strategy_require_report_scratch_arena_v1",
5723+
{"reason": "symbol_missing"})
5724+
if admission(state) != 0:
5725+
failure = run_failure(lib, state)
5726+
if failure is not None and failure[1]:
5727+
raise RunFailure(*failure)
5728+
raise RunFailure("the jailed profile requires an attached scratch arena",
5729+
"report_sink_failed", {"reason": "arena_backing"})
56925730

56935731

56945732
def _validate_sink_options(args):
@@ -5699,6 +5737,11 @@ def _validate_sink_options(args):
56995737
if fd < 3 or fd > (1 << (ctypes.sizeof(ctypes.c_int) * 8 - 1)) - 1:
57005738
raise _request_invalid("error: --report-sink-fd must be a descriptor number "
57015739
"from 3 through INT_MAX", "arguments", exit_status=2)
5740+
# ScratchArena::kInheritedFd is fixed by the engine, unlike the
5741+
# wrapper's conventional sink/phase descriptor numbers.
5742+
if fd == 5:
5743+
raise _request_invalid("error: --report-sink-fd 5 collides with "
5744+
"PF_SCRATCH_ARENA_FD", "arguments", exit_status=2)
57025745
if args.report_sink is not None:
57035746
raise _request_invalid("error: --report-sink-fd and --report-sink are mutually "
57045747
"exclusive", "arguments", exit_status=2)
@@ -5737,15 +5780,17 @@ def _new_report_sink(lib, state, args, *, body=True):
57375780
jailed = getattr(args, "profile", "unjailed") == "jailed"
57385781
if getattr(args, "report_sink_fd", None) is not None:
57395782
names = ("pf_report_sink_version", "strategy_set_report_sink_fd_v1",
5740-
"strategy_report_sink_receipt_v1", "strategy_report_sink_summary_v1")
5783+
"strategy_report_sink_receipt_v1", "strategy_report_sink_summary_v1",
5784+
"strategy_report_sink_diagnostics_v1")
57415785
if not all(hasattr(lib, name) for name in names):
57425786
raise StrategyLibraryError("the strategy library has no fd report sink API",
57435787
{"reason": "symbol_missing"})
57445788
if int(lib.pf_report_sink_version()) != 1:
57455789
raise _engine_invariant("the strategy library has an incompatible report sink ABI")
5790+
args._sink_fd_owner = _InheritedSinkFd(args.report_sink_fd)
57465791
return _FdReportSink(lib, state, fd=args.report_sink_fd,
57475792
run_id=getattr(args, "run_id", ""),
5748-
owner=getattr(args, "_sink_fd_owner", None))
5793+
owner=args._sink_fd_owner)
57495794
if jailed:
57505795
return None
57515796
names = ("pf_report_sink_version", "strategy_set_report_sink_v1",
@@ -5793,6 +5838,7 @@ def __init__(self, lib, state, *, fd, run_id, owner=None):
57935838
self.run_id = run_id
57945839
self.counts = {}
57955840
self.summary = None
5841+
self.diagnostics = None
57965842
try:
57975843
descriptor = ReportSinkFdC(ctypes.sizeof(ReportSinkFdC), 1, fd,
57985844
run_id.encode("utf-8"))
@@ -5831,7 +5877,14 @@ def verify(self, lib, state, report):
58315877
or summary.wins < 0 or summary.losses < 0
58325878
or summary.wins + summary.losses > counts["T"]):
58335879
raise _engine_invariant("the report sink summary differs from its engine receipt")
5834-
self.counts, self.summary = counts, summary
5880+
diagnostics = ReportSinkDiagnosticsC()
5881+
diagnostics.struct_size, diagnostics.version = ctypes.sizeof(diagnostics), 1
5882+
if (lib.strategy_report_sink_diagnostics_v1(state, ctypes.byref(diagnostics)) != 0
5883+
or diagnostics.complete != 1 or diagnostics.count_e != counts["E"]
5884+
or diagnostics.trade_at_start not in (0, 1)
5885+
or (diagnostics.count_e and diagnostics.last.time_ms < diagnostics.first.time_ms)):
5886+
raise _engine_invariant("the report sink diagnostics differ from its engine receipt")
5887+
self.counts, self.summary, self.diagnostics = counts, summary, diagnostics
58355888

58365889
def close(self):
58375890
if self.fd is not None:
@@ -6320,6 +6373,21 @@ def _prepare_stream_report(document, resources, directory, *, sink_errors=True):
63206373
return file
63216374

63226375

6376+
def _prepare_report_json(document, resources, args, sink):
6377+
# The jailed array route is intentionally resident, as are its native
6378+
# arrays. The fd route contains only the bounded summary. Neither may
6379+
# create a report file, even when no sink is installed.
6380+
if args.profile == "jailed" or getattr(sink, "fd_mode", False):
6381+
buffer = resources.enter_context(io.StringIO())
6382+
_stream_json(document, buffer)
6383+
buffer.write("\n")
6384+
buffer.seek(0)
6385+
return buffer
6386+
return _prepare_stream_report(
6387+
document, resources, sink.directory if sink is not None else args.spill_dir,
6388+
sink_errors=sink is not None)
6389+
6390+
63236391
def _sink_trade(record):
63246392
# The first 16 fields of fixed v1 T are exactly the existing JSON trade.
63256393
# Close cause and ids remain in the sink; they are not report additions.
@@ -6408,6 +6476,15 @@ def _build_sink_report(report, ohlcv_path, n_bars, first_ts, last_ts, elapsed,
64086476
out["summary"] = _sink_summary(report, sink)
64096477
if getattr(sink, "fd_mode", False):
64106478
_validate_report_result(out) # the engine's summary checked every T
6479+
diagnostics = sink.diagnostics
6480+
endpoints = [
6481+
{"time_ms": int(point.time_ms), "equity": float(point.equity),
6482+
"open_profit": float(point.open_profit)}
6483+
for point in (diagnostics.first, diagnostics.last)
6484+
] if diagnostics.count_e else ()
6485+
first_bar_trade = ({"entry_bar_index": 0},) if diagnostics.trade_at_start else ()
6486+
out["diagnostics"].update(
6487+
_annualization_block(endpoints, out["metrics"], first_bar_trade, window_anchor))
64116488
# stdout.json has the small report only. The wrapper/finalizer owns
64126489
# the stream's artifact and receipt; no child-local path names it.
64136490
out.pop("trades")

‎docs/pages/abi-stability.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,23 @@ process-wide signal handling.
185185

186186
## Report sink summary receipt
187187

188+
The additive `strategy_report_sink_diagnostics_v1` query supplies annualization
189+
provenance without reading the sink back. Its descriptor in `report_sink.h`
190+
is 72 bytes: `struct_size`/`version` at 0/4, `count_e` at 8, first and last
191+
`pf_equity_point_t` at 16/40, `trade_at_start` at 64 and `complete` at 68.
192+
The points are the emitted E endpoints; `trade_at_start` folds whether any
193+
emitted T has `entry_bar_index <= 0`. It publishes only after the trailer.
194+
Missing, failed, aborted or unstarted generations return zeros/incomplete.
195+
Initialize size/version=1; a bad handle, pointer, size or version returns -1
196+
without changing the output. This separate query preserves both existing
197+
receipt layouts, wire bytes and broker hashes.
198+
199+
Jailed hosts additionally call `strategy_require_report_scratch_arena_v1`
200+
before execution. The engine validates/attaches the inherited fixed-fd-5
201+
arena or returns the handle's typed `report_sink_failed`/`arena_backing`.
202+
It never creates backing. `run_json.py` refuses sink fd 5 and jailed spill
203+
directories, and takes sink-fd ownership only after the fd API is admitted.
204+
188205
`strategy_report_sink_summary_v1` is an additive query in
189206
`<pineforge/report_sink.h>` for both sink transports. It avoids a record
190207
read-back when a jailed child publishes its summary. The existing

0 commit comments

Comments
 (0)