Skip to content

run_json: reserve the report channel, so strategy output on fd 1 cannot corrupt a report - #608

Closed
luisleo526 wants to merge 5 commits into
mainfrom
report/stdout-channel
Closed

luisleo526 wants to merge 5 commits into
mainfrom
report/stdout-channel

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

run_json reserves the report channel, so strategy output written to fd 1 can no longer corrupt a report.

  • The channel is reserved only in the host process. Library calls keep using sys.stdout.
  • Descriptor numbers are validated before the collision check. An out-of-range number gets the typed refusal and exit 2.
  • A fault after publication is reported on stderr. Nothing else is written after the report.

Lab: all 8 legs pass, and every old harness test case passes unchanged. The paired parity sweep completed 134/134.

Draft until review corrections are folded and main is merged in.

🤖 Generated with Claude Code

luisleo526 and others added 4 commits October 28, 2026 08:15
…ot corrupt a report

The harness printed the report or a typed failure on fd 1, but the strategy
library it loads runs in the same process and could write to fd 1 as well: one
native printf corrupted a successful report.

- The host saves the caller's stdout as a private report descriptor before it
  parses arguments or loads any strategy, redirects native fd 1 to stderr, and
  writes the report and every typed failure through the saved descriptor.
  Failed-report discard now discards that descriptor.
- A publication guard shared by the main handler and the cleanup keeps a fault
  after a published result or failure from printing a second document; the
  exit status stays nonzero and the late diagnostic goes to stderr.
- FD sink mode keeps the native NDJSON stream on its caller-supplied stream. A
  sink or phase descriptor, or the arena slot, that collides with the report
  channel is refused with the existing run_request_invalid failure.
- docker/entrypoint.sh sends ordinary transpile, compile and bind subprocess
  output to diagnostics; the explicit transpile-only mode keeps stdout.
- The documentation states the limit: the loaded library inherits the saved
  descriptor, so this isolates ordinary native writes and is not a security
  boundary.
- tests/test_run_json_stdout.py with tests/test_run_json_stdout_strategy.cpp
  loads a stub strategy that writes a marker to fd 1 through the real harness:
  clean runs print exactly the expected report with the marker only on stderr,
  a failing run prints exactly one failure document, a fault after publication
  prints no second document, and colliding descriptors are refused.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…and fix two cases of its test

A fault after a published result or failure kept the exit status and printed no
second document, but the late exception's message was swallowed by the
traceback helper and never reached stderr; it is now written as one diagnostic
line to the real stderr. In the test, the stub strategy's printing variant
changed one reported input, so the two reports differed in content; both runs
now use the same inputs and are still compared byte for byte, and the
descriptor-collision case builds its fake owner with the type the parser
produces. Every check prints its operands on failure.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…brary calls on sys.stdout

The report owner redirected native fd 1 and wrote the report through a saved
descriptor in every call, so in-process callers that replace sys.stdout (the
harness tests call run_json.main directly) saw an empty output and a missing
owner. The fd reservation now happens only in the process entry, _host_main;
a library call without an installed owner writes to sys.stdout, looked up at call
time, exactly as before, so the existing tests pass unchanged. The publication
guard is kept per run in the owner instead of one process-wide latch. A new case
calls the library entry with a replaced sys.stdout and checks the old behaviour.
The documentation citations of docker/run_json.py that the earlier edit moved
are re-pointed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…sion check, and satisfy two CI guards

The descriptor-collision guard of the report channel called os.path.sameopenfile
on the caller's --report-sink-fd and --run-phase-fd values before the existing
range validation, so an out-of-range number died with an OverflowError and exit
status 1 instead of the typed refusal with exit status 2. The collision check now
runs only on values that passed the range validation; an out-of-range value keeps
its old message and status, and a new case covers a huge descriptor number. The
stdout-contract test module now carries the sanitizer and coverage guard its
neighbours have, and the private-fixture evidence line numbers that the CMake
additions moved are updated.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Comment thread docker/run_json.py
abort.close()
finally:
abort.close()
return 1 # A fault after library publication was diagnosed by the run guard.
Comment thread docker/run_json.py
sys.stderr.flush()
return 1
_REPORT_OUTPUT = output
status = 1
patch.setenv("PINEFORGE_STDOUT_TEST_MODE", mode or "")
try:
status = run_json._host_main(argv)
except BaseException as error:
Comment thread docker/run_json.py
except BaseException:
try:
os.close(fd)
except OSError:
Comment thread docker/run_json.py
os.dup2(devnull, self.fd, inheritable=False)
finally:
os.close(devnull)
except OSError:
finally:
try:
os.close(sink_write)
except OSError:
…ut, and tighten the channel tests

When the host's stdout and stderr are the same open file (a 2>&1 wrapper, a
terminal), routing native fd-1 writes to stderr cannot keep them out of the
report stream. The host now notices this before redirecting, writes one stderr
line, "report channel not isolated: stdout and stderr are the same file", and
keeps running: only the isolation claim is lost, so a run that worked before
still works. The docs and the changelog say the isolation holds only when the
two streams stay separate.

argparse help printed through sys.stdout, which in host mode is now fd 1 and
therefore stderr; help for run_json.py and its csv2bin form now goes to the
original stdout again.

A rejected second report publication in library mode no longer points the
caller's stdout at the null device. The wording about how the library reaches
the report descriptor now says it can write it because it runs in this
process, and that the descriptor is non-inheritable so exec'd children do not
get it. Reports are ASCII, so the UTF-8 encoding of the channel changes no
bytes. New cases: merged streams, help on stdout, descriptor relocation with 3
and 4 occupied, non-ASCII failure text on the channel, and the second library
publication. The doc citations that the source shift moved are re-pointed.

Two sentences introduced by 67ade9d ("run_json: reserve the report channel, so
strategy output on fd 1 cannot corrupt a report") are replaced on purpose.
In docs/pages/memory-management.md, "Native code runs in the same process and
inherits the saved descriptor, so code that deliberately writes through that
inherited file description can s..." is replaced by the sentence that the
library can write the descriptor because it runs in this process and that the
descriptor is non-inheritable. In docs/pages/scratch-tables.md, "The saved
report descriptor is inherited by native code in this process, so this
separation is transport isolation rather than a security boundary." is
replaced by the same corrected wording; the limit it states (not a security
boundary) is kept.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 28, 2026
…aling, UDT graphs, metrics replay and the CI admission fix (#616)

This lands twelve engine pull requests as one change, verified and measured once on the combined tree:

- #593 scratch: a fixed write-through read cache for the adapter's id tables
- #594 live runner: a delta report stream by default, and report-path work that does not rescan closed trades
- #597 live runner: typed pre-growth stop for the ledger budget and flat pending counts
- #608 run_json: reserve the report channel, so strategy output on fd 1 cannot corrupt a report
- #586 Pine UDT object graphs on the paged object Store
- #578 native order: dormant working book, flat per-point cost for never-filled orders
- #579 adapter: skip roster passes and empty-family scans that cannot change anything
- #599 adapter: read projected pending orders through one cached duplicate index
- #563 adapter: gate the reversal exit cancel on candidate owners instead of every historical id
- #591 report sink: retire sealed closed-trade rows on sink routes when the module opts in
- #596 metrics: stream the bar Sharpe and Sortino through an ordered, packed return replay
- #615 ci: accept release.yml's matrix runner under its fetched path in PR admission

Fixes found while testing the combined tree are in the final commits: the retiring ledger's cache sizing, a residency witness that bounds sink-run heap by the declared cache cap and checks zero growth at 100k, 200k and 1M trades, a test fix for a descriptor reuse race, and a standing-orders timing grid whose 20,000-order point is tracked in #621. The member pull requests are closed as landed here.
@luisleo526

Copy link
Copy Markdown
Collaborator Author

Landed in #616 (squash a991b4e on main), which carries this change with one verify and one parity run on the combined tree.

@luisleo526 luisleo526 closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant