Skip to content

Commit 9edaaa7

Browse files
luisleo526claude
andcommitted
Feed projections: honour the report-sink scratch directory, and keep a one-page projection out of scratch
The calendar day partition (trade day, week open and month open columns) and the raw-array recorded series built their projections with the default scratch directory, so a configured directory was ignored and every nonempty projection opened a backing file even when it was far below one transfer page. The engine now hands the report-sink scratch directory to those builders. FeedProjectionBuilder gains an explicit resident mode used only by those production sites. A projection that never passes its one fixed transfer page (4096 / sizeof(Row) rows) is published from an in-memory reader of exactly that page and opens no file. The next append past the page spills through an unlinked ScratchFile in the configured directory; a directory that cannot hold the file raises the existing auxiliary scratch I/O error, with no fallback to another directory. The default builder is unchanged and stays file-backed, so the tests that prove cold reads and cache behaviour through the scratch provider keep their meaning, and a reader supplied by a caller keeps the directory it was built with. BacktestEngine gains a protected const accessor for the directory, defined inline so no source line that the documentation anchors cite moves, and build_native_day_partition takes the directory as a trailing defaulted argument. The new test watches the configured and the default scratch locations for projection backing files: four backings in the configured directory and none in the default one for projections one row past their page, none for one-row projections of a default run, and a typed ENOENT with nothing created elsewhere for an unusable configured directory. Row values are read back bit for bit. docs/feed-projections.md and docs/pages/memory-management.md describe the resident page and the spill directory. Retire the sentences introduced by a279666 in docs/feed-projections.md: "The default producer writes complete rows sequentially to an unlinked ScratchFile, with one mandatory transfer page per producer", "Publication explicitly finishes the final partial page", and "Destruction performs no flush"; the production builders' one-page resident rule replaces them. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 89d78da commit 9edaaa7

9 files changed

Lines changed: 107 additions & 25 deletions

File tree

‎docs/feed-projections.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,9 +60,10 @@ selection. An idle policy change takes effect on inheriting tapes at the next
6060
preparation without resizing their previous caches. No policy bytes are added
6161
to a per-tape budget, and page tags are charged exactly once.
6262

63-
The default producer writes complete rows sequentially to an unlinked
64-
`ScratchFile`, with one mandatory transfer page per producer. Publication
65-
explicitly finishes the final partial page. Destruction performs no flush.
63+
Production builders keep projections of at most one fixed transfer page in an
64+
in-memory reader and spill larger projections through an unlinked `ScratchFile`
65+
in the report-sink scratch directory (or the default when empty); publication
66+
finishes tails and destruction never flushes.
6667
Ordinary file failures retain `scratch_io_failed` and errno. An attached
6768
arena retains its existing typed backing and exhaustion failures; arena
6869
backing is still memory. Selecting a smaller cache cannot increase the

‎docs/pages/memory-management.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -334,7 +334,7 @@ to EOF got exit status 0 and a complete verified trailer.
334334
| `--profile jailed` or `PINEFORGE_RUN_JSON_PROFILE=jailed` | `docker/run_json.py` flag or environment (`docker/run_json.py:5340,5457`); the release entrypoint forwards the environment form only | `unjailed` | Selects the bounded-memory table backend and the inherited arena for the Python host. It is the scratch profile of the engine run, not a read-only container: the entrypoint itself still compiles in `${TMPDIR:-/tmp}`. Every executed success report carries `diagnostics.run_profile` (`"jailed"` or `"unjailed"`) and `diagnostics.scratch_arena_jailed` (true only when the jailed profile ran and the arena admission succeeded; `_add_scratch_arena_diagnostics` at `docker/run_json.py:5895-5900`); the marker records the checked profile and the admission, not an independent check of operating-system confinement |
335335
| `PINEFORGE_REPORT_SINK_FD` | environment, read by the release entrypoint (`PINEFORGE_REPORT_SINK_FD` at `docker/entrypoint.sh:296`) | unset | When set, the entrypoint passes `--report-sink-fd <value>` to `run_json.py`, so the report records stream to that inherited descriptor. `run_json.py` validates the value: a descriptor number from 3 through `INT_MAX`, not 5, not combined with `--report-sink`, not in a benchmark run (`PINEFORGE_BENCH`), not in a validate-only run and not the same number as `--run-phase-fd` (`_validate_sink_options` at `docker/run_json.py:5943-5986`); a refusal is a `run_request_invalid` failure with exit status 2. Unset, nothing changes |
336336
| `pf_cache_policy_v1` through `strategy_set_cache_policy_v1` | optional bytes per supported cache kind, per handle | internal defaults (next section) | Chooses optional resident cache sizes. Omitted kinds use defaults; explicit zero bypasses caching. Nonzero ledger bytes must convert to 1–1,048,576 decoded entries; other kinds bypass below one line. A cold read uses backing. Settings freeze at run admission; an idle replacement affects the next run. A standalone C++ ledger also accepts an optional row budget through its constructor or `TradeLedger::set_cache_rows`, including zero to bypass. The engine defaults to 65,536 ledger entries with or without a report sink (`TradeLedger::set_cache_rows`, `src/native_execution_consumer.cpp:3207-3208`) |
337-
| Calendar projection and recorded-tape pages | C++ overloads | 256 KiB per calendar column and per recorded tape | Selects optional resident payload and complete shared read-through-cache metadata, allocated lazily; zero, sub-page sizes and optional allocation failures read from the provider through bypass. Each active read has a separate fixed transfer page. Logical rows and raw-array ownership are unchanged; see [Feed projection readers](../feed-projections.md) |
337+
| Calendar projection and recorded-tape pages | C++ overloads | 256 KiB per calendar column and per recorded tape | Selects optional resident payload and complete shared read-through-cache metadata, allocated lazily; zero, sub-page sizes and optional allocation failures read from the provider through bypass. Production projections up to one transfer page stay in memory; larger projections use the report-sink scratch directory or its default. Each active read has a separate fixed transfer page. Logical rows and raw-array ownership are unchanged; see [Feed projection readers](../feed-projections.md) |
338338
| Resident event and row limits of the command journal and order book | implementation constants | fixed constants (next section) | The optional cache policy does not resize these fixed workspaces or cap retained history. The journal's resident-event threshold and the book's resident-row limit remain fixed |
339339
| `PF_SCRATCH_ARENA_JAILED` | environment, exactly `1` or `0` | unset | Read with the arena variables by `ScratchArena::attach_from_environment` (declared at `include/pineforge/detail/scratch_arena.hpp:142`, defined in `src/scratch_arena.cpp`). `1` additionally requires the attached arena to be a sealed memfd at attachment, else the attach fails with `EINVAL`; `0` or unset adds no requirement (a direct attach may then use ordinary files); any other value fails with `EINVAL`. The separate admission call `strategy_require_report_scratch_arena_v1` requires a sealed memfd on Linux whatever this variable says (stated in the comment on the `ScratchArena` class in `include/pineforge/detail/scratch_arena.hpp`; see [Scratch tables](@ref scratch_tables)) |
340340

‎include/pineforge/detail/feed_projection.hpp‎

Lines changed: 70 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -276,37 +276,92 @@ class ScratchFeedReader final : public FeedReader<Row> {
276276
mutable std::uint64_t read_bytes_ = 0;
277277
};
278278

279+
// Immutable provider for a production projection that fits in its one
280+
// transfer page. The owner is bounded by Row and never grows with input.
281+
template<class Row>
282+
class ResidentFeedReader final : public FeedReader<Row> {
283+
static_assert(std::is_trivially_copyable_v<Row>);
284+
static_assert(std::is_nothrow_default_constructible_v<Row>);
285+
static_assert(sizeof(Row) <= 4096);
286+
static constexpr std::size_t kRows = 4096 / sizeof(Row);
287+
public:
288+
ResidentFeedReader(const Row* rows, std::size_t count) : count_(count) {
289+
if (count > kRows)
290+
throw std::invalid_argument("resident feed projection extent");
291+
if (count != 0)
292+
std::memcpy(rows_.data(), rows, count * sizeof(Row));
293+
}
294+
std::size_t size() const noexcept override { return count_; }
295+
void read(std::size_t first, Row* destination,
296+
std::size_t count) const override {
297+
if (first > count_ || count > count_ - first)
298+
throw std::out_of_range("resident feed range");
299+
if (count && !destination)
300+
throw std::invalid_argument("resident feed destination");
301+
if (count == 0) return;
302+
std::memcpy(destination, rows_.data() + first, count * sizeof(Row));
303+
}
304+
private:
305+
std::array<Row, kRows> rows_{};
306+
std::size_t count_ = 0;
307+
};
308+
279309
// Sequential production uses one mandatory transfer page, independent of
280-
// input length and optional read capacity. finish() flushes before publishing
281-
// the immutable reader; destruction never flushes or commits.
310+
// input length and optional read capacity. Resident mode publishes that page
311+
// in memory until a later append requires scratch; finish() flushes any
312+
// pending scratch page, and destruction never flushes or commits.
282313
template<class Row>
283314
class FeedProjectionBuilder {
284315
static_assert(std::is_trivially_copyable_v<Row>);
285316
static_assert(sizeof(Row) <= 4096);
286317
static constexpr std::size_t kRows = 4096 / sizeof(Row);
287318
public:
288-
explicit FeedProjectionBuilder(const std::string& directory = {})
289-
: reader_(std::make_shared<ScratchFeedReader<Row>>(directory)) {}
319+
explicit FeedProjectionBuilder(const std::string& directory = {},
320+
bool resident_if_single_page = false)
321+
: directory_(directory),
322+
resident_if_single_page_(resident_if_single_page) {
323+
if (!resident_if_single_page_)
324+
reader_ = std::make_shared<ScratchFeedReader<Row>>(directory_);
325+
}
290326
FeedProjectionBuilder(const FeedProjectionBuilder&) = delete;
291327
FeedProjectionBuilder& operator=(const FeedProjectionBuilder&) = delete;
292328

293329
void append(const Row& row) {
294-
if (!reader_) throw std::logic_error("feed projection already published");
330+
if (published_) throw std::logic_error("feed projection already published");
331+
if (resident_if_single_page_ && !reader_ && buffered_ == kRows)
332+
spill_to_scratch();
295333
if (buffered_ == kRows) flush();
296-
if (reader_->count_ > std::numeric_limits<std::size_t>::max() - buffered_ - 1)
297-
throw AuxiliaryScratchIoError(EFBIG, "feed projection extent");
298-
const auto offset = ScratchFeedReader<Row>::byte_offset(reader_->count_ + buffered_);
299-
ScratchFile::check_extent(offset, sizeof(Row), false, ScratchFile::IoRaiser::auxiliary);
334+
if (reader_) {
335+
if (reader_->count_ > std::numeric_limits<std::size_t>::max()
336+
- buffered_ - 1)
337+
throw AuxiliaryScratchIoError(EFBIG, "feed projection extent");
338+
const auto offset =
339+
ScratchFeedReader<Row>::byte_offset(reader_->count_ + buffered_);
340+
ScratchFile::check_extent(offset, sizeof(Row), false,
341+
ScratchFile::IoRaiser::auxiliary);
342+
}
300343
std::memcpy(&buffer_[buffered_++], &row, sizeof(Row));
301344
}
302345
FeedProjection<Row> finish(std::size_t capacity_bytes = kFeedProjectionCacheBytes) {
303-
if (!reader_) throw std::logic_error("feed projection already published");
304-
flush();
305-
FeedProjection<Row> result(reader_, capacity_bytes);
346+
if (published_) throw std::logic_error("feed projection already published");
347+
std::shared_ptr<const FeedReader<Row>> provider;
348+
if (reader_) {
349+
flush();
350+
provider = reader_;
351+
} else {
352+
provider = std::make_shared<ResidentFeedReader<Row>>(buffer_.data(), buffered_);
353+
}
354+
FeedProjection<Row> result(std::move(provider), capacity_bytes);
355+
published_ = true;
306356
reader_.reset();
307357
return result;
308358
}
309359
private:
360+
void spill_to_scratch() {
361+
reader_ = std::make_shared<ScratchFeedReader<Row>>(directory_);
362+
flush();
363+
}
364+
310365
void flush() {
311366
if (buffered_ == 0) return;
312367
reader_->file_.open("pineforge-feed");
@@ -316,6 +371,9 @@ class FeedProjectionBuilder {
316371
reader_->count_ += buffered_;
317372
buffered_ = 0;
318373
}
374+
std::string directory_;
375+
bool resident_if_single_page_ = false;
376+
bool published_ = false;
319377
std::shared_ptr<ScratchFeedReader<Row>> reader_;
320378
std::array<Row, kRows> buffer_;
321379
std::size_t buffered_ = 0;

‎include/pineforge/engine.hpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1984,10 +1984,10 @@ class BacktestEngine {
19841984
// replaced by its native sibling.
19851985
void prepare_native_security_feeds(const Bar* input_bars, int n_input);
19861986
void prepare_chart_day_partition(const Bar* input_bars, int n_input);
1987+
const std::string& report_scratch_directory() const noexcept { return report_sink_state_.scratch_dir; }
19871988
bool substitute_native_security_bar(SecurityEvalState& state, Bar& bar,
19881989
bool count_miss = true);
19891990

1990-
19911991
#ifdef PINEFORGE_HAS_AUX_SECURITY_FEED_V1
19921992
virtual bool source_aux_security_feed_enabled() const;
19931993
virtual void source_aux_security_input_view(const Bar*& bars, int& n) const;

‎include/pineforge/timeframe.hpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -283,7 +283,7 @@ bool build_native_day_partition(NativeDayPartition& out,
283283
const std::string& tz,
284284
const std::string& session,
285285
const std::vector<int64_t>& stamps,
286-
const Bar* input_bars, int n_input);
286+
const Bar* input_bars, int n_input, const std::string& scratch_directory = {});
287287

288288
/// As above, with an inclusive optional cache budget for EACH of the three
289289
/// projection columns. Zero/sub-page budgets read directly from backing.
@@ -293,7 +293,7 @@ bool build_native_day_partition(NativeDayPartition& out,
293293
const std::string& session,
294294
const std::vector<int64_t>& stamps,
295295
const Bar* input_bars, int n_input,
296-
std::size_t projection_cache_bytes);
296+
std::size_t projection_cache_bytes, const std::string& scratch_directory = {});
297297

298298
/// Index of the native period holding `ms` under `p`: -1 before the first
299299
/// stamp and at / after the last stamp's nominal session-day close.

‎src/engine_aux_security.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -368,7 +368,7 @@ void BacktestEngine::prepare_chart_day_partition(const Bar* input_bars,
368368
for (const Bar& bar : feed.bars) stamps.push_back(bar.timestamp);
369369
build_native_day_partition(chart_day_partition_, syminfo_.timezone,
370370
syminfo_.session, stamps, input_bars,
371-
n_input, cache_policy_for_run().feed_calendar_pages.capacity_bytes);
371+
n_input, cache_policy_for_run().feed_calendar_pages.capacity_bytes, report_scratch_directory());
372372
return;
373373
}
374374
}

‎src/source/pine_security_eval.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1551,7 +1551,7 @@ bool source::PineStrategyHost::set_recorded_series_impl(
15511551
}
15521552
// n == 0 is a request that returned na on every chart bar (a tape with a
15531553
// header only): installed, so every read is na rather than a failure.
1554-
pineforge::detail::FeedProjectionBuilder<RecordedSeriesRow> builder;
1554+
pineforge::detail::FeedProjectionBuilder<RecordedSeriesRow> builder(report_scratch_directory(), true);
15551555
for (int i = 0; i < n; ++i)
15561556
builder.append(RecordedSeriesRow{chart_open_ms[i], values[i]});
15571557
RecordedSeries series{builder.finish(cache_capacity_bytes.value_or(

‎src/timeframe.cpp‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -803,17 +803,17 @@ bool build_native_day_partition(NativeDayPartition& out,
803803
const std::string& tz,
804804
const std::string& session,
805805
const std::vector<int64_t>& stamps,
806-
const Bar* input_bars, int n_input) {
806+
const Bar* input_bars, int n_input, const std::string& scratch_directory) {
807807
return build_native_day_partition(out, tz, session, stamps, input_bars,
808-
n_input, detail::kFeedProjectionCacheBytes);
808+
n_input, detail::kFeedProjectionCacheBytes, scratch_directory);
809809
}
810810

811811
bool build_native_day_partition(NativeDayPartition& out,
812812
const std::string& tz,
813813
const std::string& session,
814814
const std::vector<int64_t>& stamps,
815815
const Bar* input_bars, int n_input,
816-
std::size_t projection_cache_bytes) {
816+
std::size_t projection_cache_bytes, const std::string& scratch_directory) {
817817
out = NativeDayPartition{};
818818
if (stamps.empty()) return false;
819819
for (std::size_t k = 1; k < stamps.size(); ++k) {
@@ -830,7 +830,7 @@ bool build_native_day_partition(NativeDayPartition& out,
830830
p.last_bound = session_day_close_real_ms(
831831
session_day_index_nominal(stamps.back(), tz, session), tz, session);
832832
const std::size_t n = stamps.size();
833-
detail::FeedProjectionBuilder<int64_t> trade_days, week_opens, month_opens;
833+
detail::FeedProjectionBuilder<int64_t> trade_days(scratch_directory, true), week_opens(scratch_directory, true), month_opens(scratch_directory, true);
834834
// Trade instants in one merge pass: the last input bar before the next
835835
// stamp (before the last bound for the last stamp), the stamp itself
836836
// when no input bar lies in the period.

‎tests/test_feed_projection_scratch_directory.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,28 @@ void configured_directory_case() {
272272
check_calendar(host, input);
273273
}
274274

275+
void configured_spill_failure_case() {
276+
Directory default_directory("pineforge-feed-default-failure");
277+
Environment temporary("TMPDIR", default_directory.path);
278+
char unavailable[] = "/tmp/pineforge-feed-reader-XXXXXX";
279+
require(::mkdtemp(unavailable) != nullptr, "cannot create unavailable directory");
280+
require(::rmdir(unavailable) == 0, "cannot remove unavailable directory");
281+
CreationWatch watch(default_directory.path, default_directory.path);
282+
283+
detail::FeedProjectionBuilder<std::int64_t> builder(unavailable, true);
284+
for (std::size_t i = 0; i < Calendar::rows_per_page(); ++i)
285+
builder.append(static_cast<std::int64_t>(i));
286+
bool typed_failure = false;
287+
try {
288+
builder.append(static_cast<std::int64_t>(Calendar::rows_per_page()));
289+
} catch (const detail::AuxiliaryScratchIoError& error) {
290+
typed_failure = error.code().value() == ENOENT;
291+
}
292+
CHECK(typed_failure);
293+
watch.collect();
294+
CHECK(watch.default_feed_creations() == 0);
295+
}
296+
275297
void default_small_case() {
276298
Directory default_directory("pineforge-feed-default-small");
277299
Environment temporary("TMPDIR", default_directory.path);
@@ -300,6 +322,7 @@ int main() {
300322
using namespace feed_projection_scratch_directory_test;
301323
try {
302324
configured_directory_case();
325+
configured_spill_failure_case();
303326
default_small_case();
304327
} catch (const std::exception& error) {
305328
std::fprintf(stderr, "projection scratch-directory setup failed: %s\n", error.what());

0 commit comments

Comments
 (0)