Skip to content

Commit f7ecc1d

Browse files
luisleo526claude
andcommitted
Merge the equity-curve window commit into the train
The commit makes a sequential pass over a spilled equity curve read whole 2,730-point blocks again, with the tests and documentation that pin it. One documentation paragraph was edited on both sides: the train's copy carried the shifted line numbers of the writer constants, the commit's copy carried the new description of the reader. The merged paragraph keeps both: the reader description of the commit and the line numbers read from the merged sources (the constants at report_scratch.hpp:20-22, the writer state at report_scratch.cpp:160-168, the reader state at :278-334). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
2 parents f8e9589 + a1a2abe commit f7ecc1d

8 files changed

Lines changed: 394 additions & 20 deletions

‎docs/design/tail-page-windows.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,8 @@ arguments still accepts payload budgets, adding the exact shared line charge onc
6666
The first placement defers index creation; a later mutation or reader may
6767
build it. Configuration does not alter rebuilding or query ordering.
6868

69-
`ScratchCurveWindow` rounds its payload allowance down to complete points.
70-
Each point is a 24-byte shared-cache page. Zero, a sub-point allowance and
69+
`ScratchCurveWindow` rounds its payload allowance down to complete 24-byte point pages.
70+
Backing misses adjacent to the previous lookup fill aligned blocks of up to 2,730 points; other backing misses read one point. Zero, a sub-point allowance and
7171
optional-allocation failure bypass. A returned point is a copy, so callers
7272
can retain it across later reads. A failed read publishes no successful tag
7373
and can be retried after backing repair; unrelated valid lines remain valid.
@@ -86,8 +86,10 @@ allowance C, optional shared-cache storage is
8686
payload, keys, pins and replacement metadata. Fixed owner control and
8787
transfer/commit workspace are additional. The position reader defaults to
8888
16 lines of 4,096 bytes; the curve reader defaults to 2,730 lines of 24
89-
bytes. The curve therefore adds one metadata charge per point, rather than
90-
pretending that all 2,730 records share one tag.
89+
bytes, with one metadata charge per point. `cache_control_bytes()` reports
90+
`sizeof(ScratchCurveWindow) + sizeof(State)`, including the fixed 65,520-byte
91+
block workspace. Shared-cache statistics charge the optional line tags and
92+
payload separately from that control total. Bypass reads do not use the block.
9193

9294
Native pending storage additionally uses `rows * record_bytes` when the
9395
optional batch exists, plus one inline fallback record per store. At the

‎docs/feed-projections.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -60,10 +60,14 @@ selection. An idle policy change takes effect on inheriting tapes at the next
6060
preparation without resizing their previous caches. No policy bytes are added
6161
to a per-tape budget, and page tags are charged exactly once.
6262

63-
Production builders keep projections of at most one fixed transfer page in an
64-
in-memory reader and spill larger projections through an unlinked `ScratchFile`
65-
in the report-sink scratch directory (or the default when empty); publication
66-
finishes tails and destruction never flushes.
63+
Production builders use an in-memory reader for up to one 4096-byte page
64+
**per calendar column or recorded tape**: 512 int64 rows per calendar column,
65+
or 256 recorded rows of 16 bytes (`{timestamp,value}`). Each of the three
66+
calendar columns is its own projection. Larger projections spill through an
67+
unlinked `ScratchFile` in the report-sink scratch directory (or the default
68+
when empty), and an attached arena supplies those pages, reporting its typed
69+
exhaustion error at the limit; publication finishes tails and destruction
70+
never flushes.
6771
Ordinary file failures retain `scratch_io_failed` and errno. An attached
6872
arena retains its existing typed backing and exhaustion failures; arena
6973
backing is still memory. Selecting a smaller cache cannot increase the

‎docs/pages/memory-management.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,9 +37,9 @@ src/engine_report.cpp:1814-1828).
3737
The writer batches **2,730 points of 24 B**, within a **64 KiB** window
3838
(`kScratchWindowPoints` src/report_scratch.hpp:20-22;
3939
`ScratchCurveWriter::State` src/report_scratch.cpp:160-168).
40-
Metrics re-read scratch through bounded windows
40+
Metrics re-read scratch through 24-byte point-cache lines plus one fixed 65,520-byte block workspace, included in `cache_control_bytes()`
4141
(`compute_report_sink_metrics` src/engine_report.cpp:1791-1812;
42-
`ScratchCurveWindow` src/report_scratch.cpp:278-309).
42+
`ScratchCurveWindow::State` src/report_scratch.cpp:278-334); backing misses adjacent to the previous lookup fill aligned blocks of up to 2,730 points; other backing misses and bypasses read one point.
4343
This bounds resident equity-report buffers; the scratch backing and output
4444
still grow with the run. The replay needs no descriptor of its own in a jailed run (the arena's inherited descriptor serves it) and exactly one transient anonymous file otherwise; a run leaves the process's descriptor count as it found it.
4545

@@ -511,7 +511,7 @@ This setter adds no environment variables or JSON limits.
511511
| Erased minima and consumed cycles | `erased_minima_pages`, `consumed_cycle_pages` | 32 KiB each, shared by that owner's data/index files |
512512
| Position-tail read window | `position_tail_pages` | 64 KiB in 16 complete 4 KiB pages, plus one shared metadata charge per page |
513513
| Successor-root, group-receipt and chain-segment read caches | `successor_root_pages`, `receipt_tail_pages`, `chain_segment_pages` | 64, 64 and 16 fixed-width payload rows respectively; pending write batches are separate fixed transaction workspace |
514-
| Ordinary report-sink metric curve reader | `equity_curve_pages` | 2,730 complete 24-byte points; the writer's retry batch is mandatory workspace |
514+
| Ordinary report-sink metric curve reader | `equity_curve_pages` | 2,730 complete 24-byte point lines, plus a fixed 65,520-byte sequential block workspace charged by `cache_control_bytes()`; the writer's retry batch is separate mandatory workspace |
515515
| Numeric Series/rings, HeldHistory, extrema and WMA windows constructed by the run | `ta_history_pages` | Existing owning logical window; finite allowances select backed numeric pages |
516516
| Native-day trade-day/week/month columns | `feed_calendar_pages` | 256 KiB inclusive per column |
517517
| Recorded request tapes installed without an explicit C++ capacity | `recorded_series_pages` | 256 KiB inclusive per tape |

‎src/report_scratch.cpp‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -279,6 +279,10 @@ struct ScratchCurveWindow::State {
279279
metrics::ScratchCurveRegion region;
280280
ScratchReadStats* inner_reads;
281281
detail::ReadThroughWindow cache;
282+
std::array<pf_equity_point_t, kScratchWindowPoints> window{};
283+
std::int64_t first = 0;
284+
std::int64_t count = 0;
285+
std::int64_t previous_index = -1;
282286

283287
State(const metrics::ScratchCurveRegion& value, std::size_t cache_bytes,
284288
ScratchReadStats* reads)
@@ -293,10 +297,39 @@ struct ScratchCurveWindow::State {
293297
static void read_backing(void* context, std::uint64_t offset,
294298
void* destination, std::size_t bytes) {
295299
auto& self = *static_cast<State*>(context);
296-
read_scratch_points(self.region,
297-
static_cast<std::int64_t>(offset / sizeof(pf_equity_point_t)),
300+
const auto index = static_cast<std::int64_t>(offset / sizeof(pf_equity_point_t));
301+
// The shared cache has already attempted allocation. Its bypass
302+
// must read directly without consulting the fixed block workspace.
303+
if (self.cache.payload_bytes() != 0) {
304+
if ((index < self.first || index - self.first >= self.count)
305+
&& self.previous_index >= 0
306+
&& (index == self.previous_index + 1 || index + 1 == self.previous_index)) {
307+
constexpr auto capacity = static_cast<std::int64_t>(kScratchWindowPoints);
308+
const auto first = index / capacity * capacity;
309+
const auto count = std::min(self.region.point_count - first, capacity);
310+
// A short read can overwrite the old block. Publish its
311+
// replacement only after all requested bytes are available.
312+
self.count = 0;
313+
read_scratch_points(self.region, first, self.window.data(), count, self.inner_reads);
314+
self.first = first;
315+
self.count = count;
316+
}
317+
if (index >= self.first && index - self.first < self.count) {
318+
std::memcpy(destination, &self.window[static_cast<std::size_t>(index - self.first)],
319+
bytes);
320+
return;
321+
}
322+
}
323+
read_scratch_points(self.region, index,
298324
static_cast<pf_equity_point_t*>(destination),
299325
static_cast<std::int64_t>(bytes / sizeof(pf_equity_point_t)), self.inner_reads);
326+
// Retain a lone point until a sequential block replaces it, even
327+
// if a distant probe collides with its shared-cache line.
328+
if (self.cache.payload_bytes() != 0 && self.count == 0) {
329+
std::memcpy(self.window.data(), destination, bytes);
330+
self.first = index;
331+
self.count = 1;
332+
}
300333
}
301334
};
302335

@@ -333,6 +366,7 @@ pf_equity_point_t ScratchCurveWindow::operator[](std::int64_t index) const {
333366
}
334367
pf_equity_point_t point;
335368
state.cache.get(static_cast<std::uint64_t>(index), &point, sizeof(point));
369+
state.previous_index = index;
336370
return point;
337371
}
338372

‎src/report_scratch.hpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,8 @@ void read_scratch_points(const metrics::ScratchCurveRegion& region,
119119
class ScratchCurveWindow {
120120
public:
121121
// Byte payload budget; complete shared line metadata is additional.
122+
// Adjacent backing misses fill aligned blocks of up to kScratchWindowPoints.
123+
// One fixed 65,520-byte block workspace is included in cache_control_bytes().
122124
// Sub-point sizes and optional allocation failure bypass.
123125
// An optional counter must outlive this reader.
124126
explicit ScratchCurveWindow(const metrics::ScratchCurveRegion& region,

‎tests/test_curve_window_sizes.cpp‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
#include <cstdlib>
1313
#include <cstring>
1414
#include <limits>
15+
#include <random>
1516
#include <stdexcept>
1617
#include <system_error>
1718
#include <unistd.h>
@@ -34,6 +35,161 @@ bool equal(const pf_equity_point_t& a, const pf_equity_point_t& b) {
3435
return std::memcmp(&a, &b, sizeof(a)) == 0;
3536
}
3637

38+
constexpr std::size_t kBoundaryBlockAllowance = 2;
39+
40+
std::size_t block_count(std::size_t points) {
41+
return points == 0 ? 0
42+
: (points + rs::kScratchWindowPoints - 1) / rs::kScratchWindowPoints;
43+
}
44+
45+
void curve_pass_read_cost(const pineforge::metrics::ScratchCurveRegion& region,
46+
const std::vector<pf_equity_point_t>& points, bool reverse) {
47+
require(points.size() > rs::kScratchWindowPoints,
48+
"curve read-cost row must exceed the configured point capacity");
49+
rs::ScratchReadStats inner;
50+
rs::ScratchCurveWindow curve(region, rs::kScratchWindowBytes, &inner);
51+
require(equal(curve[0], points[0]), "first boundary probe is exact");
52+
require(equal(curve[static_cast<std::int64_t>(points.size() - 1)], points.back()),
53+
"last boundary probe is exact");
54+
const auto probes = inner.calls;
55+
// The two nonadjacent endpoints are distinct cold reads before the pass.
56+
require(probes == 2, "two boundary probes reach backing before the pass");
57+
if (!reverse) {
58+
for (std::size_t i = 0; i != points.size(); ++i)
59+
require(equal(curve[static_cast<std::int64_t>(i)], points[i]),
60+
"forward block walk is exact");
61+
} else {
62+
for (std::size_t i = points.size(); i-- > 0;)
63+
require(equal(curve[static_cast<std::int64_t>(i)], points[i]),
64+
"reverse block walk is exact");
65+
}
66+
const auto blocks = block_count(points.size());
67+
const auto bound = blocks + kBoundaryBlockAllowance;
68+
std::printf("curve %s points=%zu blocks=%zu allowance=%zu probes=%llu "
69+
"backing_reads=%llu backing_bytes=%llu bound=%zu\n",
70+
reverse ? "reverse" : "forward", points.size(), blocks,
71+
kBoundaryBlockAllowance, static_cast<unsigned long long>(probes),
72+
static_cast<unsigned long long>(inner.calls),
73+
static_cast<unsigned long long>(inner.bytes), bound);
74+
require(inner.calls <= bound, "sequential curve reads stay at block cost");
75+
}
76+
77+
void bypass_pass_read_cost(const pineforge::metrics::ScratchCurveRegion& region,
78+
const std::vector<pf_equity_point_t>& points) {
79+
for (const auto bytes : {std::size_t{0}, sizeof(pf_equity_point_t) - 1}) {
80+
rs::ScratchReadStats inner;
81+
rs::ScratchCurveWindow curve(region, bytes, &inner);
82+
for (std::size_t i = 0; i != points.size(); ++i)
83+
require(equal(curve[static_cast<std::int64_t>(i)], points[i]),
84+
"bypass curve walk is exact");
85+
std::printf("curve bypass requested=%zu points=%zu backing_reads=%llu "
86+
"backing_bytes=%llu\n",
87+
bytes, points.size(), static_cast<unsigned long long>(inner.calls),
88+
static_cast<unsigned long long>(inner.bytes));
89+
require(inner.calls == points.size()
90+
&& inner.bytes == points.size() * sizeof(pf_equity_point_t),
91+
"zero and sub-point capacities read one point per bypass access");
92+
}
93+
}
94+
95+
void random_access_read_cost(const pineforge::metrics::ScratchCurveRegion& region,
96+
const std::vector<pf_equity_point_t>& points) {
97+
constexpr std::size_t kRandomSamples = 257;
98+
std::mt19937_64 random(UINT64_C(0x4d7a2c91b8e60315));
99+
rs::ScratchReadStats inner;
100+
rs::ScratchCurveWindow curve(region, rs::kScratchWindowBytes, &inner);
101+
std::size_t adjacent_pairs = 0;
102+
std::size_t previous_index = 0;
103+
for (std::size_t sample = 0; sample != kRandomSamples; ++sample) {
104+
const auto index = static_cast<std::size_t>(
105+
random() % static_cast<std::uint64_t>(points.size()));
106+
if (sample != 0
107+
&& (index == previous_index + 1 || index + 1 == previous_index))
108+
++adjacent_pairs;
109+
require(equal(curve[static_cast<std::int64_t>(index)], points[index]),
110+
"random curve access is exact");
111+
previous_index = index;
112+
}
113+
// read_backing requests one point on a nonadjacent miss and can fill an
114+
// aligned block only for an index +/-1 from the previous lookup. Reserve
115+
// sizeof(pf_equity_point_t) per sample plus one complete block workspace
116+
// per adjacent pair in the generated sequence; hits and short final
117+
// blocks can only reduce the actual backing bytes.
118+
const auto byte_bound = kRandomSamples * sizeof(pf_equity_point_t)
119+
+ adjacent_pairs * rs::kScratchWindowPoints * sizeof(pf_equity_point_t);
120+
std::printf("curve random samples=%zu adjacent_pairs=%zu backing_reads=%llu backing_bytes=%llu "
121+
"read_bound=%zu byte_bound=%zu\n",
122+
kRandomSamples, adjacent_pairs, static_cast<unsigned long long>(inner.calls),
123+
static_cast<unsigned long long>(inner.bytes), kRandomSamples, byte_bound);
124+
require(inner.calls <= kRandomSamples && inner.bytes <= byte_bound,
125+
"random accesses do not exceed one backing operation per lookup");
126+
}
127+
128+
void extremes_read_cost(const pineforge::metrics::ScratchCurveRegion& region,
129+
const std::vector<pf_equity_point_t>& points) {
130+
pf_equity_point_t held = points.back();
131+
held.time_ms += 1;
132+
held.equity = points.back().equity + 17.0;
133+
double expected[] = {10000.0, 10000.0, 0.0, 0.0};
134+
const auto fold = [&](double equity) {
135+
if (equity > expected[0]) {
136+
expected[0] = equity;
137+
expected[1] = equity;
138+
}
139+
if (equity < expected[1]) expected[1] = equity;
140+
const double drawdown = expected[0] - equity;
141+
if (drawdown > expected[2]) expected[2] = drawdown;
142+
const double runup = equity - expected[1];
143+
if (runup > expected[3]) expected[3] = runup;
144+
};
145+
for (const auto& point : points) fold(point.equity);
146+
fold(held.equity);
147+
// The extremes check builds its own default-capacity window over the
148+
// prefix and exposes no read counter of its own: its backing reads are the
149+
// forward-pass reads bounded by curve_pass_read_cost, over the same window
150+
// class. This row pins that the check still folds every point exactly (it
151+
// throws on a mismatch) after the window changes.
152+
rs::check_scratch_equity_extremes(region, &held, 10000.0, expected);
153+
std::printf("curve extremes points=%zu exact\n", points.size());
154+
}
155+
156+
std::vector<pf_equity_point_t> bit_pattern_points(std::size_t count) {
157+
std::mt19937_64 random(UINT64_C(0x9a31f0c44b7e2086));
158+
std::vector<pf_equity_point_t> points(count);
159+
for (std::size_t i = 0; i != points.size(); ++i) {
160+
points[i].time_ms = static_cast<std::int64_t>(random());
161+
const auto equity = random();
162+
const auto open_profit = random();
163+
std::memcpy(&points[i].equity, &equity, sizeof(equity));
164+
std::memcpy(&points[i].open_profit, &open_profit, sizeof(open_profit));
165+
}
166+
return points;
167+
}
168+
169+
void bit_exact_block_fill(const char* directory_path) {
170+
const auto points = bit_pattern_points(2 * rs::kScratchWindowPoints + 17);
171+
rs::ScratchCurveWriter writer(directory_path);
172+
for (const auto& point : points) writer.append(point);
173+
const auto region = writer.region();
174+
rs::ScratchCurveWindow curve(region, rs::kScratchWindowBytes);
175+
for (std::size_t i = 0; i != points.size(); ++i)
176+
require(equal(curve[static_cast<std::int64_t>(i)], points[i]),
177+
"forward block fill preserves every point bit");
178+
for (std::size_t i = points.size(); i-- > 0;)
179+
require(equal(curve[static_cast<std::int64_t>(i)], points[i]),
180+
"reverse block fill preserves every point bit");
181+
std::puts("curve bit-exact block fill passed");
182+
}
183+
184+
void curve_read_cost_rows(const pineforge::metrics::ScratchCurveRegion& region,
185+
const std::vector<pf_equity_point_t>& points) {
186+
curve_pass_read_cost(region, points, false);
187+
curve_pass_read_cost(region, points, true);
188+
bypass_pass_read_cost(region, points);
189+
random_access_read_cost(region, points);
190+
extremes_read_cost(region, points);
191+
}
192+
37193
void sweep(const pineforge::metrics::ScratchCurveRegion& region,
38194
const std::vector<pf_equity_point_t>& points, std::size_t bytes) {
39195
rs::ScratchReadStats inner;
@@ -163,6 +319,16 @@ int main() {
163319
writer.append(points.back());
164320
}
165321
const auto region = writer.region();
322+
curve_read_cost_rows(region, points);
323+
// Endpoint probes collide in the default point cache at these sizes.
324+
for (const auto count : {rs::kScratchWindowPoints + 1, 2 * rs::kScratchWindowPoints + 1}) {
325+
auto prefix = region;
326+
prefix.point_count = static_cast<std::int64_t>(count);
327+
const std::vector<pf_equity_point_t> prefix_points(points.begin(), points.begin() + count);
328+
curve_pass_read_cost(prefix, prefix_points, false);
329+
curve_pass_read_cost(prefix, prefix_points, true);
330+
}
331+
bit_exact_block_fill(directory.path);
166332
for (const auto bytes : {std::size_t{0}, sizeof(pf_equity_point_t) - 1,
167333
sizeof(pf_equity_point_t), 2 * sizeof(pf_equity_point_t),
168334
rs::kScratchWindowBytes})

0 commit comments

Comments
 (0)