Skip to content

Commit f10f0cc

Browse files
luisleo526claude
andcommitted
integration: V19-E's K3 pin keeps the first re-issue of each revival candidate, not every one -- a flat cycle that re-issues its exits every bar no longer grows the placement table or the bar-open sweep
The Cloud Run sweep of f6e69e56 timed out on seven population probes that main, INT20, C7 and INT19 grade excellent (2 engine / 2 TV trades each). They share one shape: after a trade closes the script keeps calling strategy.exit every bar, at a stop and a limit taken from the close, for entries that are not open, so one flat position cycle runs to the end of the feed. Reproduced on spark with the probe transpiled and run on the 53,929-bar 15-minute ETH feed: main 1.23 s, this tree before the fix 65.71 s, the same 13 trades. Mechanism, from a perf profile and an instrumented copy of the sweep: erase_retired_rows was 62% self time plus its binary searches and string compares, the full pass ran at 11,985 of 12,000 bars, and the table retained thousands of exit-leg rows of the current cycle with no lifecycle target, every one kept by K3. The legs the cycle placed on its first bar keep their target (the re-issues that follow carry no projection link to them), so they stay candidates (K1) for the whole cycle, and K3 pinned every later same-id re-issue at another stop: two rows per exit per bar, each walked by the pre-pass, the full pass and K3's comparison against the candidates at every bar open -- O(bars^2) in the cycle's length. Fix, in the owning mechanism: the revival's superseded test (revive_brackets_after_margin) asks whether ANY current-cycle re-issue of the candidate exists at a later bar and another stop, and a row's family, ids, placement bar and stop never change once placed, so the lowest-incarnation such re-issue answers for every later one. K3 now pins exactly that one per candidate (computed once per sweep into RetiredRowScratch::first_reissues); every later re-issue is erased once nothing else holds it. K2 is unchanged. The tombstone audit at the revival counts its scan as the existence test it is: an erased supersessor is a hit only when no retained row answers first (the unrefined predicate also gives 0 hits over the battery; no battery revival reaches an erased second re-issue). Witness: tests/test_adapter_live_state_scaling.cpp gains a third workload of that shape on a daily chart -- one trade whose take-profit fills early, then two exits (one for that entry, one for an entry that never opens) re-issued every bar -- gated without recording (the backtest path; the sweep is the cost it holds) at 4,000 and 16,000 bars, its two sizes timed in turn: fail-before, this TU against f6e69e56: 0.2906 s / 5.3903 s (ratio 18.55), 7,748 / 31,024 rows retained -- three FAILs (both row bounds, the ratio); after: 0.0133 s / 0.0541 s (ratio 4.07; 3.89 on a rerun), 12 rows retained. The straddle at 64,000 and 256,000 bars: 0.205 s and 0.890 s, 12 rows. Nothing a run shows moves: the equivalence battery holds all 108 configurations equal with rows retained and erased and to their pinned transcripts (328 checks, 0 failures; 46,918 rows erased, was 46,861); the 100-slot fingerprint sweep, magnifier off and on, is byte-identical to this tree before the fix, broker-state hash included, and its trade fields to main's; the tombstone build reports 0 sites hit over the full ctest (378 processes), the battery and the 200 fingerprint runs (up to 45,131 tombstones). Two pinned tests read hashes that fold the retained rows and move: they are re-pinned in the next commit. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 88e2f22 commit f10f0cc

4 files changed

Lines changed: 165 additions & 48 deletions

File tree

‎docs/design/native-feature-parity.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1012,6 +1012,26 @@ not (`tests/test_adapter_live_state_equivalence.cpp` holds each script's
10121012
command events, pending orders, working requests, trades, equity and
10131013
admission fields equal with every row retained and with rows erased).
10141014
1015+
**Corrected at integration (INT21): K3 keeps one re-issue per candidate.** K3
1016+
first pinned every current-cycle re-issue of a revival candidate at a later bar
1017+
and another stop. When a trade closes and the script keeps calling
1018+
`strategy.exit` every bar for entries that are not open, the legs placed on the
1019+
cycle's first bar keep their lifecycle target and stay candidates (K1) to the
1020+
end of the cycle, so every later re-issue was pinned: two rows per exit per bar
1021+
for the rest of the cycle, each walked and compared by the sweep at every bar
1022+
open. That is O(bars^2) in the cycle's length: scripts of this shape ran
1023+
dozens of times slower than on the base, which keeps every row but walks none of
1024+
them at a bar open. The revival's superseded test asks whether any such
1025+
re-issue exists, and a re-issue's family, ids, placement bar and stop never
1026+
change, so K3 now keeps the first one (the lowest incarnation) for each
1027+
candidate and the rest are erased once nothing else holds them; no trade moves.
1028+
`tests/test_adapter_live_state_scaling.cpp`'s third workload, a daily
1029+
chart of that shape, holds it: 12 rows retained and a ×4 cost for four times the
1030+
bars (4,000 to 16,000), where the uncorrected sweep retained 7,748 and 31,024
1031+
rows and cost ×18.6.
1032+
The tombstone audit counts that scan as the existence test it is: an erased row
1033+
is a hit only when no retained row answers the test first.
1034+
10151035
**The witness, pinned now.** `tests/test_adapter_recording_hash_witness.cpp`
10161036
records every row, a read after every command and the final scalar of six
10171037
scenarios — submit, replace, cancel, OCA, re-issue with a late bracket, trail —

‎include/pineforge/source/pine_adapter.hpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1960,6 +1960,7 @@ class PineExecutionAdapter {
19601960
std::vector<std::int32_t> source_bars;
19611961
std::vector<std::pair<std::uint64_t, const PlacementSnapshot*>> candidates;
19621962
std::vector<std::uint64_t> candidate_names;
1963+
std::vector<std::uint64_t> first_reissues;
19631964
std::vector<std::uint64_t> doomed;
19641965
void clear() noexcept {
19651966
roots.clear();
@@ -1968,6 +1969,7 @@ class PineExecutionAdapter {
19681969
source_bars.clear();
19691970
candidates.clear();
19701971
candidate_names.clear();
1972+
first_reissues.clear();
19711973
doomed.clear();
19721974
}
19731975
};

‎src/source/pine_adapter.cpp‎

Lines changed: 53 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1781,7 +1781,9 @@ void PineExecutionAdapter::revive_brackets_after_margin(
17811781
std::optional<PlacementSnapshot> marketable;
17821782
native_order::RequestHandle marketable_handle{};
17831783
// No erased row is a revival candidate (K1 keeps every lifecycle of the
1784-
// current cycle), nor supersedes one that is (K2, K3).
1784+
// current cycle), and no candidate's superseded answer rests on an erased
1785+
// row: K2 keeps every row that names it as its predecessor, K3 the first
1786+
// re-issue that supersedes it, and one supersessor answers the test.
17851787
PINEFORGE_AUDIT_PLACEMENT_SCAN(placement_,
17861788
[&](std::uint64_t, const PlacementSnapshot& candidate) {
17871789
return candidate.placement_cycle == current_position_cycle_
@@ -1796,19 +1798,23 @@ void PineExecutionAdapter::revive_brackets_after_margin(
17961798
}
17971799
const auto inc = retained.first;
17981800
const auto target_inc = candidate.legs.target().incarnation;
1799-
PINEFORGE_AUDIT_PLACEMENT_SCAN(placement_,
1800-
[&](std::uint64_t incarnation, const PlacementSnapshot& peer) {
1801-
return peer.placement_cycle == current_position_cycle_
1802-
&& ((peer.projection_predecessor != 0
1803-
&& (peer.projection_predecessor == inc
1804-
|| peer.projection_predecessor == target_inc))
1805-
|| (incarnation > inc && peer.family == candidate.family
1806-
&& peer.source_id == candidate.source_id
1807-
&& peer.from_entry == candidate.from_entry
1808-
&& peer.placement_script_open_ms > candidate.placement_script_open_ms
1809-
&& !same_double_bits(peer.exit_levels.stop,
1810-
candidate.exit_levels.stop)));
1811-
});
1801+
const auto supersedes = [&](std::uint64_t incarnation, const PlacementSnapshot& peer) {
1802+
return peer.placement_cycle == current_position_cycle_
1803+
&& ((peer.projection_predecessor != 0
1804+
&& (peer.projection_predecessor == inc
1805+
|| peer.projection_predecessor == target_inc))
1806+
|| (incarnation > inc && peer.family == candidate.family
1807+
&& peer.source_id == candidate.source_id
1808+
&& peer.from_entry == candidate.from_entry
1809+
&& peer.placement_script_open_ms > candidate.placement_script_open_ms
1810+
&& !same_double_bits(peer.exit_levels.stop,
1811+
candidate.exit_levels.stop)));
1812+
};
1813+
// The test below stops at the first supersessor it finds: an erased
1814+
// one only reaches it when no retained row answers first.
1815+
const bool answered = std::any_of(placement_.begin(), placement_.end(),
1816+
[&](const auto& peer) { return supersedes(peer.first, peer.second); });
1817+
if (!answered) PINEFORGE_AUDIT_PLACEMENT_SCAN(placement_, supersedes);
18121818
}
18131819
#endif
18141820
for (auto row : placement_) {
@@ -2650,8 +2656,13 @@ bool PineExecutionAdapter::lifecycle_readable(const PlacementSnapshot& row) cons
26502656
// and the pair hold rewrite lifecycles, and exit() asks whether an
26512657
// opened origin's leg was consumed;
26522658
// K2 a current-cycle row naming a revival candidate as its predecessor,
2653-
// K3 a current-cycle leg re-issuing a revival candidate at a later bar at
2654-
// another stop: the revival's superseded test;
2659+
// K3 the first current-cycle leg re-issuing a revival candidate at a later
2660+
// bar at another stop: the revival's superseded test asks whether ANY
2661+
// such re-issue exists, and a row's family, ids, placement bar and stop
2662+
// never change, so the lowest-incarnation one answers for every later
2663+
// one (pinning them all kept two rows per exit per bar through a flat
2664+
// cycle that re-issues its exits every bar, each walked at every bar
2665+
// open);
26552666
// K4 an immediate strategy.close placed on this bar or later
26562667
// (immediate_close_placed_on asks about the current bar);
26572668
// K5 an entry or raw order placed on the previous bar or later (the open
@@ -2878,6 +2889,31 @@ void PineExecutionAdapter::erase_retired_rows(const NativeDecisionContext& conte
28782889
candidate_names.push_back(value.legs.target().incarnation);
28792890
}
28802891
std::sort(candidate_names.begin(), candidate_names.end());
2892+
// K3: each candidate's first re-issue -- the lowest incarnation among the
2893+
// current-cycle rows the revival's superseded test would accept for it.
2894+
auto& first_reissues = sweep.first_reissues;
2895+
if (!candidates.empty()) {
2896+
constexpr std::uint64_t none = std::numeric_limits<std::uint64_t>::max();
2897+
first_reissues.assign(candidates.size(), none);
2898+
for (const auto& row : placement_) {
2899+
const auto& value = row.second;
2900+
if (value.placement_cycle != cycle || !exit_family(value.family)) continue;
2901+
for (std::size_t index = 0; index < candidates.size(); ++index) {
2902+
const auto& c = candidates[index];
2903+
if (row.first <= c.first || row.first >= first_reissues[index]) continue;
2904+
const PlacementSnapshot& q = *c.second;
2905+
if (q.family == value.family
2906+
&& q.placement_script_open_ms < value.placement_script_open_ms
2907+
&& !same_double_bits(q.exit_levels.stop, value.exit_levels.stop)
2908+
&& q.source_id == value.source_id && q.from_entry == value.from_entry) {
2909+
first_reissues[index] = row.first;
2910+
}
2911+
}
2912+
}
2913+
first_reissues.erase(std::remove(first_reissues.begin(), first_reissues.end(), none),
2914+
first_reissues.end());
2915+
std::sort(first_reissues.begin(), first_reissues.end());
2916+
}
28812917

28822918
auto& doomed = sweep.doomed;
28832919
for (const auto& row : placement_) {
@@ -2893,16 +2929,7 @@ void PineExecutionAdapter::erase_retired_rows(const NativeDecisionContext& conte
28932929
continue;
28942930
}
28952931
// K3
2896-
if (exit_family(value.family)
2897-
&& std::any_of(candidates.begin(), candidates.end(), [&](const auto& c) {
2898-
const PlacementSnapshot& q = *c.second;
2899-
return c.first < incarnation && q.family == value.family
2900-
&& q.source_id == value.source_id && q.from_entry == value.from_entry
2901-
&& q.placement_script_open_ms < value.placement_script_open_ms
2902-
&& !same_double_bits(q.exit_levels.stop, value.exit_levels.stop);
2903-
})) {
2904-
continue;
2905-
}
2932+
if (exit_family(value.family) && contains(first_reissues, incarnation)) continue;
29062933
}
29072934
// K7
29082935
if (value.family == PineOrderFamily::Entry && value.sequential_group != 0

‎tests/test_adapter_live_state_scaling.cpp‎

Lines changed: 90 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,26 @@
3131
// 2.42 s for 1,000 bars and 37.86 s for 4,000 (ratio 15.64) and retained 1,999
3232
// and 7,999 rows; the churn 4.57 s and 73.21 s (16.03), 2,214 and 8,834 rows.
3333
//
34+
// A third workload, added at integration (INT21), holds the erasure's own
35+
// cost: a daily chart whose one trade closes early and whose two exits (one
36+
// for that entry, one for an entry that never opens) are re-issued every bar
37+
// for the rest of the run, so one flat position cycle spans the run. The legs
38+
// the cycle placed on its first bar keep their lifecycle target, so they stay
39+
// revival candidates (K1) to the end, and K3 pinned every later re-issue at
40+
// another stop: two rows per bar per exit, each walked and compared by the
41+
// sweep at every bar open -- O(bars^2), what timed seven population probes of
42+
// this shape out on Cloud Run. Its gate runs without recording: that is the
43+
// path a backtest takes, and the sweep is the cost it holds (recording also
44+
// folds each bracket family's working tail, which the cycle's first legs keep
45+
// from settling; that fold is O(tail) per read before and after, and less
46+
// than the base's whole-table fold). It runs four times the others' bars,
47+
// its two sizes timed in turn. Fail-before, this TU against INT21's f6e69e56
48+
// (macOS arm64, Release): 4,000 bars 0.2906 s, 16,000 bars 5.3903 s (ratio
49+
// 18.55), 7,748 and 31,024 rows retained.
50+
//
3451
// Single-leg mode for peak-RSS rows: `test_adapter_live_state_scaling --leg
35-
// replay|churn <bars> [--no-recording]` runs one leg and prints its CPU time,
36-
// retained rows and ru_maxrss.
52+
// replay|churn|straddle <bars> [--no-recording]` runs one leg and prints its
53+
// CPU time, retained rows and ru_maxrss.
3754
#include <pineforge/source/pine_strategy_host.hpp>
3855

3956
#include <sys/resource.h>
@@ -72,15 +89,19 @@ int failures = 0;
7289
constexpr double kNa = std::numeric_limits<double>::quiet_NaN();
7390
constexpr std::int64_t T = 1736121600000LL;
7491

75-
enum class Workload { Replay, Churn };
92+
enum class Workload { Replay, Churn, Straddle };
7693

7794
long triangle(int index, int period) {
7895
const int phase = index % (2 * period);
7996
return phase < period ? phase : 2 * period - phase;
8097
}
8198

82-
// Quarter-tick prices (exact binary fractions) from two triangle waves.
83-
std::vector<Bar> tape(int count) {
99+
// Quarter-tick prices (exact binary fractions) from two triangle waves, one
100+
// bar per minute (or per day: the straddle's daily chart).
101+
constexpr std::int64_t kMinute = 60000;
102+
constexpr std::int64_t kDay = 86400000;
103+
104+
std::vector<Bar> tape(int count, std::int64_t step = kMinute) {
84105
std::vector<Bar> bars;
85106
bars.reserve(static_cast<std::size_t>(count));
86107
const auto at = [](int index) {
@@ -91,7 +112,7 @@ std::vector<Bar> tape(int count) {
91112
const double close = at(index + 1);
92113
bars.push_back({open, (open > close ? open : close) + 0.5,
93114
(open < close ? open : close) - 0.5, close, 1.0,
94-
T + static_cast<std::int64_t>(index) * 60000});
115+
T + static_cast<std::int64_t>(index) * step});
95116
}
96117
return bars;
97118
}
@@ -120,6 +141,18 @@ class ScalingHost final : public source::PineStrategyHost {
120141
if (position > 0.0) strategy_exit("guard", "L", bar.close * 1.60, bar.close * 0.40);
121142
return;
122143
}
144+
if (workload_ == Workload::Straddle) {
145+
// One trade, then one flat position cycle for the rest of the run:
146+
// the entry opens at bar 8 and its take-profit, a quarter above the
147+
// close, fills within a few bars. Two exits are re-issued every bar
148+
// from the first to the last, at a stop and a limit that move with
149+
// the close: one for that entry, one for an entry that never opens.
150+
(void)position;
151+
if (i == 8) strategy_entry("L", true, kNa, kNa, 1.0);
152+
strategy_exit("LX", "L", bar.close + 0.25, bar.close - 40.0);
153+
strategy_exit("SX", "S", bar.close - 40.0, bar.close + 40.0);
154+
return;
155+
}
123156
// Position cycles, alternating sides: two quantity brackets per
124157
// entry around its fill price, each in its own group and re-issued
125158
// at a moving level every bar; the tight one fills within a few bars
@@ -188,10 +221,10 @@ int attempts() {
188221
return parsed > 0 ? parsed : 5;
189222
}
190223

191-
Leg best_of(Workload workload, const std::vector<Bar>& bars) {
224+
Leg best_of(Workload workload, const std::vector<Bar>& bars, bool recording) {
192225
Leg best;
193226
for (int attempt = 0; attempt < attempts(); ++attempt) {
194-
const Leg sample = replay(workload, bars);
227+
const Leg sample = replay(workload, bars, recording);
195228
if (attempt == 0 || sample.seconds < best.seconds) best = sample;
196229
}
197230
return best;
@@ -203,40 +236,73 @@ constexpr double kShapeBound = 5.0;
203236
// current position cycle still reads, and the last bar's retirements.
204237
constexpr std::size_t kRowBound = 64;
205238

206-
void cost_and_rows_are_live(Workload workload, const char* name) {
207-
const int bars = gated() ? kBars : kBars / 4;
208-
const std::vector<Bar> small_tape = tape(bars);
209-
const std::vector<Bar> large_tape = tape(bars * 4);
210-
const Leg small = best_of(workload, small_tape);
211-
const Leg large = best_of(workload, large_tape);
239+
// The straddle's two sizes, timed in turn round after round (which goes first
240+
// alternates), each keeping its best round: its legs are milliseconds long, so
241+
// a load change between two blocks of rounds would land on one side only.
242+
void interleaved_best(Workload workload, const std::vector<Bar>& small_tape,
243+
const std::vector<Bar>& large_tape, bool recording, Leg& small,
244+
Leg& large) {
245+
for (int attempt = 0; attempt < attempts(); ++attempt) {
246+
const bool small_first = attempt % 2 == 0;
247+
const Leg a = replay(workload, small_first ? small_tape : large_tape, recording);
248+
const Leg b = replay(workload, small_first ? large_tape : small_tape, recording);
249+
const Leg& s_leg = small_first ? a : b;
250+
const Leg& l_leg = small_first ? b : a;
251+
if (attempt == 0 || s_leg.seconds < small.seconds) small = s_leg;
252+
if (attempt == 0 || l_leg.seconds < large.seconds) large = l_leg;
253+
}
254+
}
255+
256+
void cost_and_rows_are_live(Workload workload, const char* name, bool recording = true) {
257+
// The straddle runs four times longer: its bars cost microseconds.
258+
const int scale = workload == Workload::Straddle ? 4 : 1;
259+
const int bars = (gated() ? kBars : kBars / 4) * scale;
260+
const std::int64_t step = workload == Workload::Straddle ? kDay : kMinute;
261+
const std::vector<Bar> small_tape = tape(bars, step);
262+
const std::vector<Bar> large_tape = tape(bars * 4, step);
263+
Leg small;
264+
Leg large;
265+
if (workload == Workload::Straddle) {
266+
interleaved_best(workload, small_tape, large_tape, recording, small, large);
267+
} else {
268+
small = best_of(workload, small_tape, recording);
269+
large = best_of(workload, large_tape, recording);
270+
}
212271
const double ratio = small.seconds > 0.0 ? large.seconds / small.seconds : 0.0;
213-
std::printf("%s (recording): %d bars %.4fs (%d trades, %zu rows retained), "
272+
std::printf("%s (%s): %d bars %.4fs (%d trades, %zu rows retained), "
214273
"%d bars %.4fs (%d trades, %zu rows retained), ratio %.2f (bound %.1f), "
215274
"rows bound %zu\n",
216-
name, bars, small.seconds, small.trades, small.rows, bars * 4,
217-
large.seconds, large.trades, large.rows, ratio, kShapeBound, kRowBound);
275+
name, recording ? "recording" : "no recording", bars, small.seconds,
276+
small.trades, small.rows, bars * 4, large.seconds, large.trades, large.rows,
277+
ratio, kShapeBound, kRowBound);
218278
CHECK(small.seconds > 0.0);
219279
// The workloads did what they describe.
220280
if (workload == Workload::Churn) {
221281
CHECK(small.trades >= bars / 16);
222282
CHECK(large.trades >= 3 * small.trades);
223283
}
284+
if (workload == Workload::Straddle) {
285+
CHECK(small.trades == 1);
286+
CHECK(large.trades == 1);
287+
}
224288
// Memory: what the adapter retains does not grow with the run.
225289
CHECK(small.rows <= kRowBound);
226290
CHECK(large.rows <= kRowBound);
227291
if (!gated()) {
228292
std::printf(" (ratio not gated: non-Release library)\n");
229293
return;
230294
}
231-
if (workload == Workload::Replay) CHECK(ratio < kShapeBound);
295+
if (workload != Workload::Churn) CHECK(ratio < kShapeBound);
232296
}
233297

234298
int single_leg(const char* workload_name, const char* bars_text, bool recording) {
235-
const Workload workload = std::strcmp(workload_name, "churn") == 0
236-
? Workload::Churn : Workload::Replay;
299+
const Workload workload = std::strcmp(workload_name, "churn") == 0 ? Workload::Churn
300+
: std::strcmp(workload_name, "straddle") == 0 ? Workload::Straddle
301+
: Workload::Replay;
237302
const int bars = std::atoi(bars_text);
238303
if (bars <= 0) return 2;
239-
const Leg leg = replay(workload, tape(bars), recording);
304+
const Leg leg = replay(workload, tape(bars, workload == Workload::Straddle ? kDay : kMinute),
305+
recording);
240306
struct rusage usage {};
241307
getrusage(RUSAGE_SELF, &usage);
242308
std::printf("leg %s bars=%d recording=%d cpu_seconds=%.6f trades=%d rows_retained=%zu "
@@ -249,12 +315,14 @@ int single_leg(const char* workload_name, const char* bars_text, bool recording)
249315
} // namespace
250316

251317
int main(int argc, char** argv) {
252-
// --leg <replay|churn> <bars> [--no-recording]
318+
// --leg <replay|churn|straddle> <bars> [--no-recording]
253319
if ((argc == 4 || argc == 5) && std::strcmp(argv[1], "--leg") == 0)
254320
return single_leg(argv[2], argv[3],
255321
!(argc == 5 && std::strcmp(argv[4], "--no-recording") == 0));
256322
cost_and_rows_are_live(Workload::Replay, "exit re-issued every bar");
257323
cost_and_rows_are_live(Workload::Churn, "position cycles with brackets");
324+
cost_and_rows_are_live(Workload::Straddle, "two exits re-issued every bar through a flat cycle",
325+
false);
258326
if (failures == 0) std::printf("test_adapter_live_state_scaling: ok\n");
259327
return failures == 0 ? 0 : 1;
260328
}

0 commit comments

Comments
 (0)