You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9edaaa7
Browse filesBrowse the repository at this point in the historyBrowse files
Feed projections: honour the report-sink scratch directory, and keep a one-page projection out of scratch
The calendar day partition (trade day, week open and month open columns) and the raw-array recorded series built their projections with the default scratch directory, so a configured directory was ignored and every nonempty projection opened a backing file even when it was far below one transfer page. The engine now hands the report-sink scratch directory to those builders.
FeedProjectionBuilder gains an explicit resident mode used only by those production sites. A projection that never passes its one fixed transfer page (4096 / sizeof(Row) rows) is published from an in-memory reader of exactly that page and opens no file. The next append past the page spills through an unlinked ScratchFile in the configured directory; a directory that cannot hold the file raises the existing auxiliary scratch I/O error, with no fallback to another directory. The default builder is unchanged and stays file-backed, so the tests that prove cold reads and cache behaviour through the scratch provider keep their meaning, and a reader supplied by a caller keeps the directory it was built with.
BacktestEngine gains a protected const accessor for the directory, defined inline so no source line that the documentation anchors cite moves, and build_native_day_partition takes the directory as a trailing defaulted argument.
The new test watches the configured and the default scratch locations for projection backing files: four backings in the configured directory and none in the default one for projections one row past their page, none for one-row projections of a default run, and a typed ENOENT with nothing created elsewhere for an unusable configured directory. Row values are read back bit for bit.
docs/feed-projections.md and docs/pages/memory-management.md describe the resident page and the spill directory. Retire the sentences introduced by a279666 in docs/feed-projections.md: "The default producer writes complete rows sequentially to an unlinked ScratchFile, with one mandatory transfer page per producer", "Publication explicitly finishes the final partial page", and "Destruction performs no flush"; the production builders' one-page resident rule replaces them.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: docs/pages/memory-management.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -334,7 +334,7 @@ to EOF got exit status 0 and a complete verified trailer.
334
334
| `--profile jailed` or `PINEFORGE_RUN_JSON_PROFILE=jailed` | `docker/run_json.py` flag or environment (`docker/run_json.py:5340,5457`); the release entrypoint forwards the environment form only | `unjailed` | Selects the bounded-memory table backend and the inherited arena for the Python host. It is the scratch profile of the engine run, not a read-only container: the entrypoint itself still compiles in `${TMPDIR:-/tmp}`. Every executed success report carries `diagnostics.run_profile` (`"jailed"` or `"unjailed"`) and `diagnostics.scratch_arena_jailed` (true only when the jailed profile ran and the arena admission succeeded; `_add_scratch_arena_diagnostics` at `docker/run_json.py:5895-5900`); the marker records the checked profile and the admission, not an independent check of operating-system confinement |
335
335
| `PINEFORGE_REPORT_SINK_FD` | environment, read by the release entrypoint (`PINEFORGE_REPORT_SINK_FD` at `docker/entrypoint.sh:296`) | unset | When set, the entrypoint passes `--report-sink-fd <value>` to `run_json.py`, so the report records stream to that inherited descriptor. `run_json.py` validates the value: a descriptor number from 3 through `INT_MAX`, not 5, not combined with `--report-sink`, not in a benchmark run (`PINEFORGE_BENCH`), not in a validate-only run and not the same number as `--run-phase-fd` (`_validate_sink_options` at `docker/run_json.py:5943-5986`); a refusal is a `run_request_invalid` failure with exit status 2. Unset, nothing changes |
336
336
| `pf_cache_policy_v1` through `strategy_set_cache_policy_v1` | optional bytes per supported cache kind, per handle | internal defaults (next section) | Chooses optional resident cache sizes. Omitted kinds use defaults; explicit zero bypasses caching. Nonzero ledger bytes must convert to 1–1,048,576 decoded entries; other kinds bypass below one line. A cold read uses backing. Settings freeze at run admission; an idle replacement affects the next run. A standalone C++ ledger also accepts an optional row budget through its constructor or `TradeLedger::set_cache_rows`, including zero to bypass. The engine defaults to 65,536 ledger entries with or without a report sink (`TradeLedger::set_cache_rows`, `src/native_execution_consumer.cpp:3207-3208`) |
337
-
| Calendar projection and recorded-tape pages | C++ overloads | 256 KiB per calendar column and per recorded tape | Selects optional resident payload and complete shared read-through-cache metadata, allocated lazily; zero, sub-page sizes and optional allocation failures read from the provider through bypass. Each active read has a separate fixed transfer page. Logical rows and raw-array ownership are unchanged; see [Feed projection readers](../feed-projections.md) |
337
+
| Calendar projection and recorded-tape pages | C++ overloads | 256 KiB per calendar column and per recorded tape | Selects optional resident payload and complete shared read-through-cache metadata, allocated lazily; zero, sub-page sizes and optional allocation failures read from the provider through bypass. Production projections up to one transfer page stay in memory; larger projections use the report-sink scratch directory or its default. Each active read has a separate fixed transfer page. Logical rows and raw-array ownership are unchanged; see [Feed projection readers](../feed-projections.md) |
338
338
| Resident event and row limits of the command journal and order book | implementation constants | fixed constants (next section) | The optional cache policy does not resize these fixed workspaces or cap retained history. The journal's resident-event threshold and the book's resident-row limit remain fixed |
339
339
| `PF_SCRATCH_ARENA_JAILED` | environment, exactly `1` or `0` | unset | Read with the arena variables by `ScratchArena::attach_from_environment` (declared at `include/pineforge/detail/scratch_arena.hpp:142`, defined in `src/scratch_arena.cpp`). `1` additionally requires the attached arena to be a sealed memfd at attachment, else the attach fails with `EINVAL`; `0` or unset adds no requirement (a direct attach may then use ordinary files); any other value fails with `EINVAL`. The separate admission call `strategy_require_report_scratch_arena_v1` requires a sealed memfd on Linux whatever this variable says (stated in the comment on the `ScratchArena` class in `include/pineforge/detail/scratch_arena.hpp`; see [Scratch tables](@ref scratch_tables)) |
0 commit comments