Skip to content

Commit 822641a

Browse files
luisleo526claude
andauthored
engine: preserve cohort admission ordering and capacity (#770)
Merge gate on head 14d7d9b: the required CI result, the corpus parity check and the review all pass. In the post-merge native run on the same tree, every native shard and live-tsan pass. live-sanitizers, the derived native-live-gate and the sanitizer shards are red on runner disk (No space left on device), the same jobs that fail on main 06eb496; #765 addresses it. This change does not cause them. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 06eb496 commit 822641a

12 files changed

Lines changed: 540 additions & 23 deletions

‎ci/cr150-source-bounds.yaml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,11 +83,13 @@ ceilings:
8383
# CR-559: one map Store resize group and one group of three error-string temporaries.
8484
# Both stay OPEN; an optional pair limit with default zero establishes no fixed bound.
8585
# Ten time-format string groups use per-call pattern and written-width storage.
86-
# Forty-seven more unreviewed groups on the integrated tree (51 emitted by
86+
# Forty-six more unreviewed groups on the integrated tree (50 emitted by
8787
# --emit-seed, 4 stale entries removed): the bounded drawings, the owned
8888
# storage and runtime context service, the ledger cache
8989
# and the selected-capacity driver, reader and core. None is bounded.
90-
unreviewed: 1407
90+
# One materialize_relative_exits local-list group and three
91+
# SeriesPercentileNearestRank groups remain OPEN as well.
92+
unreviewed: 1406
9193
pending-flat-memory: 184
9294
owners:
9395
- {name: "generated-cpp", globs: ["tutorial/**"]}
@@ -923,7 +925,6 @@ entries:
923925
- {id: "U0828", file: "src/source/pine_adapter.cpp", symbol: "close_logical_units_", ops: ["subscript"], container: "unresolved by the index: close_logical_units_[...]", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 9488}
924926
- {id: "U0830", file: "src/source/pine_adapter.cpp", symbol: "closes", ops: ["assign", "decl", "grow"], container: "std::vector<std::pair<native_order::RequestHandle, PlacementSnapshot>>", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 1885}
925927
- {id: "U0833", file: "src/source/pine_adapter.cpp", symbol: "cohort_sides", ops: ["decl", "subscript"], container: "std::unordered_map<const void*, CohortSides>", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 1551}
926-
- {id: "U0834", file: "src/source/pine_adapter.cpp", symbol: "cohorts_by_id_", ops: ["grow", "subscript"], container: "unresolved by the index: cohorts_by_id_.emplace(...)", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 3020}
927928
- {id: "U0835", file: "src/source/pine_adapter.cpp", symbol: "consumed_partial_exit_cycles_", ops: ["grow", "subscript"], container: "unresolved by the index: consumed_partial_exit_cycles_[...]", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 24790}
928929
- {id: "U0836", file: "src/source/pine_adapter.cpp", symbol: "cross_side_by_entry", ops: ["decl", "grow"], container: "std::unordered_map<SourceId, bool>", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 1974}
929930
- {id: "U0837", file: "src/source/pine_adapter.cpp", symbol: "current_debited_applied_ordinals_", ops: ["grow"], container: "unresolved by the index: current_debited_applied_ordinals_.insert(...)", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 10535}

‎ci/cr150-source-reading.tsv‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -429,7 +429,6 @@ U0829 src/source/pine_adapter.cpp closed_cohorts std::vector<SourceId> grow=1 lo
429429
U0830 src/source/pine_adapter.cpp closes std::vector<std::pair<native_order::RequestHandle, PlacementSnapshot>> assign=6,decl=1,grow=1 mixed:file+local NativeExecutionConsumer* native = nullptr; adapter state source-layer stale 8 / new 8 SITES READ:growth-op+assign-count/source
430430
U0831 src/source/pine_adapter.cpp cohort_ids std::vector<SourceId> decl=2,grow=4 local } adapter state source-layer stale 6 / new 6 SITES READ:growth-op DEFECT:G2 reader (second pass); first pass DEFECT; evidence pine_adapter.cpp:7511-7513,8719-8721; Per-call copies of every resident cohort id (two sites). Resident rows are ids with exposure or a pinned roster window (U1225); reserve() is no cap.
431431
U0833 src/source/pine_adapter.cpp cohort_sides std::unordered_map<const void*, CohortSides> decl=1,subscript=1 mixed:file+member || snapshot.family == PineOrderFamily::ExitTrail) adapter state source-layer stale 2 / new 2 SITES READ:growth-op+decl-member/ns/file DEFECT:G2 reader (second pass); first pass DEFECT; evidence pine_adapter.cpp:1705,1939-1952,28778-28780; One entry per resident cohort with >16 lifetime origins and a non-empty window, erased only when that window empties (28778-28780). Follows the resident rows.
432-
U0834 src/source/pine_adapter.cpp cohorts_by_id_ unresolved by the index: cohorts_by_id_.emplace(...) grow=1 local // is a named divergence in docs/design/native-feature-parity.md §3.7: adapter state source-layer stale 2 / new 1 SITES READ:growth-op REJECT:scratch index Second pass; first pass DEFECT; evidence pine_adapter.cpp:3196-3209; disk_id_table.hpp:18-23,340-343; disk_resident_id_table.hpp:454-466; The per-id index node lives in scratch (DiskIdTable RAM = 5 scalars + a ScratchFile), and a fresh row is spillable at the next spill. Resident growth is U1225's.
433432
U0835 src/source/pine_adapter.cpp consumed_partial_exit_cycles_ unresolved by the index: consumed_partial_exit_cycles_[...] grow=1 local // `id`) released onto its new book: bound to that request, bound to the id adapter state source-layer stale 1 / new 1 SITES READ:growth-op REJECT:scratch RAM fixed reader, re-read at source by reviewer (second pass); first pass DEFECT; evidence pine_adapter.cpp:25396-25400; id_pair_store.hpp:3,34-36,811-830; IdPairStore keeps no per-key RAM directory: records and index are scratch files, with RAM static_assert <= 32 KiB (:830). Disk grows with distinct keys; RAM does not.
434433
U0836 src/source/pine_adapter.cpp cross_side_by_entry std::unordered_map<SourceId, bool> decl=1,grow=1 local // A leg whose row was erased answers through its fact (`erased`). adapter state source-layer stale 2 / new 2 SITES READ:growth-op
435434
U0837 src/source/pine_adapter.cpp current_debited_applied_ordinals_ unresolved by the index: current_debited_applied_ordinals_.insert(...) grow=1 local // A partial source close breaks the exact ShortSeed transaction book. adapter state source-layer stale 1 / new 1 SITES READ:growth-op

‎ci/cr150-source-sites.json‎

Lines changed: 24 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1127,6 +1127,30 @@
11271127
"spelling": "at = . . .",
11281128
"ordinal": 1
11291129
},
1130+
{
1131+
"file": "include/pineforge/detail/disk_resident_id_table.hpp",
1132+
"symbol": "index_",
1133+
"scope": [
1134+
"namespace pineforge :: detail #1",
1135+
"template < class Value > class DiskResidentIdTable #1",
1136+
"std :: pair < iterator , bool > place ( const std :: string & key , Value value ) #1"
1137+
],
1138+
"cls": "grow",
1139+
"spelling": "index_ . insert ( . . . )",
1140+
"ordinal": 1
1141+
},
1142+
{
1143+
"file": "include/pineforge/detail/disk_resident_id_table.hpp",
1144+
"symbol": "row",
1145+
"scope": [
1146+
"namespace pineforge :: detail #1",
1147+
"template < class Value > class DiskResidentIdTable #1",
1148+
"std :: pair < iterator , bool > place ( const std :: string & key , Value value ) #1"
1149+
],
1150+
"cls": "alloc",
1151+
"spelling": "make_shared",
1152+
"ordinal": 1
1153+
},
11301154
{
11311155
"file": "include/pineforge/detail/disk_resident_id_table.hpp",
11321156
"symbol": "index_",
@@ -31113,17 +31137,6 @@
3111331137
"spelling": "result = . . .",
3111431138
"ordinal": 1
3111531139
},
31116-
{
31117-
"file": "src/source/pine_adapter.cpp",
31118-
"symbol": "cohorts_by_id_",
31119-
"scope": [
31120-
":: pineforge :: source :: detail :: PositionTailDiagnostics :: pineforge :: source :: detail :: PositionTailNoDiagnostics namespace pineforge :: source #1",
31121-
"native_order :: CohortHandle PineExecutionAdapter :: cohort_for ( const SourceId & id ) #1"
31122-
],
31123-
"cls": "grow",
31124-
"spelling": "cohorts_by_id_ . emplace ( . . . )",
31125-
"ordinal": 1
31126-
},
3112731140
{
3112831141
"file": "src/source/pine_adapter.cpp",
3112931142
"symbol": "adopted_reissues",

‎docs/design/arena-bound.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -340,8 +340,8 @@ indexPeak(k) = 24 * slots(k) # k > 32
340340
| --- | --- | --- |
341341
| Equity writer / selected hold-back / SpillCurve | 24*E | `pf_equity_point_t` size assertion `include/pineforge/metrics_stream.hpp:16`; `src/report_scratch.cpp:238` |
342342
| Position tail | (30+L)*T | `src/source/pine_position_tail.hpp:20`; footer packing at `src/source/pine_position_tail.cpp:94` |
343-
| Cohort AVL and five-word summary | (96+L)*D + 1[D>0] | `Links` at `include/pineforge/detail/disk_id_table.hpp:165-182`, `end_` at `:348`; `CohortFacts::Record` at `include/pineforge/source/pine_adapter.hpp:2059-2062` |
344-
| Entry-rank AVL | (60+L)*J + 1[J>0] | `exit_rank_keys_` at `src/source/pine_adapter.cpp:6800-6801`; `Links` at `include/pineforge/detail/disk_id_table.hpp:165-182`, `end_` at `:348` |
343+
| Cohort AVL and five-word summary | (96+L)*D + 1[D>0] | `Links` at `include/pineforge/detail/disk_id_table.hpp:165-182`, `end_` at `:349`; `CohortFacts::Record` at `include/pineforge/source/pine_adapter.hpp:2059-2062` |
344+
| Entry-rank AVL | (60+L)*J + 1[J>0] | `exit_rank_keys_` at `src/source/pine_adapter.cpp:6800-6801`; `Links` at `include/pineforge/detail/disk_id_table.hpp:165-182`, `end_` at `:349` |
345345
| Erased minima data | (40+L)*D + (40+2*L)*Q | `erased_minima_` at `include/pineforge/source/pine_adapter.hpp:960`; 40-byte header at `include/pineforge/detail/id_pair_store.hpp:69` |
346346
| Erased minima index | indexPeak(D+Q) | 16-byte slot at `include/pineforge/detail/id_pair_store.hpp:69`; replacement at `:642` |
347347
| Consumed partial-cycle data | (41+2*L)*Q | `include/pineforge/source/pine_adapter.hpp:2838`; joined key at `src/source/pine_adapter.cpp:25506` |

‎include/pineforge/detail/disk_id_table.hpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ class DiskIdTable {
6161
return file_.scratch_owner() && file_.scratch_resource() == ScratchResource::cohort;
6262
}
6363
void set_directory(const std::string& directory) { file_.set_directory(directory); }
64+
void open_for_insert() { file_.open("pineforge-ids"); }
6465
void bind_scratch(const std::shared_ptr<ScratchSlab>& slab, ScratchResource resource) {
6566
if (size_ != 0)
6667
throw coded<std::logic_error>(RunFailureCode::engine_invariant, {},

‎include/pineforge/detail/disk_resident_id_table.hpp‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -440,17 +440,30 @@ class DiskResidentIdTable {
440440
const auto row = resident_.find(at->second);
441441
return {this, row->first, row->second.owner};
442442
}
443-
std::pair<iterator, bool> emplace(const std::string& key, Value value) {
443+
void admit(const std::string& key, bool spillable) {
444444
// A refused new logical row must not first overwrite/evict an eligible
445445
// resident row. This search reads index bytes only, without restoring.
446446
index_.require_insert(key);
447447
std::size_t extra = 0;
448-
if (index_.bounded_cohorts() && !value.spillable() && !index_.find(key)) {
448+
if (index_.bounded_cohorts() && !spillable && !index_.find(key)) {
449449
extra = 1; // An existing logical row ignores the incoming value.
450450
require_live_growth(extra);
451451
}
452452
spill_before_admission(extra);
453453
spill_implicitly();
454+
index_.open_for_insert();
455+
}
456+
std::pair<iterator, bool> place(const std::string& key, Value value) {
457+
const auto placed = index_.insert(key, value.retired_record());
458+
if (!placed.second)
459+
throw coded<std::logic_error>(RunFailureCode::engine_invariant, {},
460+
"resident id placement found duplicate after admission");
461+
auto row = std::make_shared<Row>(key, std::move(value));
462+
insert_resident(placed.first, row);
463+
return {{this, placed.first, std::move(row)}, true};
464+
}
465+
std::pair<iterator, bool> emplace(const std::string& key, Value value) {
466+
admit(key, value.spillable());
454467
const auto placed = index_.insert(key, value.retired_record());
455468
if (!placed.second) return {{this, placed.first, load(placed.first)}, false};
456469
auto row = std::make_shared<Row>(key, std::move(value));

‎scripts/test_cr150_source_gate.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1216,12 +1216,12 @@ def test_real_tree_scans_files_and_sites_and_is_not_a_pass_by_construction(self)
12161216
self.assertEqual(c['findings'], 0)
12171217
self.assertEqual(c['PASS(bounded)'], 0)
12181218
# Exact manifest union: 1,344 original unreviewed groups, ten time-format
1219-
# groups and two map groups, and 47 groups of the integrated tree (51
1219+
# groups and two map groups, and 46 groups of the integrated tree (50
12201220
# emitted, 4 stale entries removed), one group of the local list of
12211221
# materialize_relative_exits and three groups of
12221222
# SeriesPercentileNearestRank (its ring, the growth of the ring and its
12231223
# window); all 184 pending-flat-memory groups remain.
1224-
self.assertEqual(c['OPEN'], 1356 + 51 + 184)
1224+
self.assertEqual(c['OPEN'], 1356 + 50 + 184)
12251225
# Every matched entry is exactly one of PASS(bounded) / OPEN.
12261226
self.assertEqual(c['PASS(bounded)'] + c['OPEN'], c['entries_matched'])
12271227
strict = run_gate(ROOT, '--strict')

‎src/source/pine_adapter.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3217,16 +3217,16 @@ native_order::CohortHandle PineExecutionAdapter::cohort_for(const SourceId& id)
32173217
auto found = cohorts_by_id_.find(id);
32183218
if (found != cohorts_by_id_.end()) return found->second.handle;
32193219
CohortFacts facts;
3220+
cohorts_by_id_.admit(id, facts.spillable());
32203221
facts.handle = require_host().cohort_open();
32213222
if (facts.handle.value == 0) {
32223223
throw coded<std::logic_error>(RunFailureCode::engine_invariant, {},
32233224
"native cohort allocation refused");
32243225
}
32253226
const auto result = facts.handle;
3226-
cohorts_by_id_.emplace(id, std::move(facts));
3227+
cohorts_by_id_.place(id, std::move(facts));
32273228
return result;
32283229
}
3229-
32303230
PineSizingSnapshot PineExecutionAdapter::sizing_snapshot() const {
32313231
PineSizingSnapshot snapshot;
32323232
const auto& host = require_host();
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
if(PINEFORGE_BUILD_SOURCE_LAYER)
2+
list(APPEND TEST_SOURCES test_cohort_for_live_limit_new_id)
3+
pineforge_add_unit_test(test_cohort_for_live_limit_new_id)
4+
list(APPEND TEST_SOURCES test_cohort_for_existing_id_cadence)
5+
pineforge_add_unit_test(test_cohort_for_existing_id_cadence)
6+
list(APPEND TEST_SOURCES test_cohort_for_admission_io_refusal)
7+
pineforge_add_unit_test(test_cohort_for_admission_io_refusal)
8+
endif()
Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
// CR-365 R2-A2 (extra to the brief): an admission refusal that the read-only
2+
// live-limit ticket cannot see must still precede the host cohort handle.
3+
//
4+
// cohort_for opens the table's backing file during admission. When that open is
5+
// refused (here: the scratch directory is a regular file) the new id must be
6+
// refused before require_host().cohort_open(), so no kernel cohort handle or
7+
// receipt is consumed. The witness is dynamic: it reads the kernel's next
8+
// handle before and after the refused call, and it first proves, through the
9+
// table's own emplace and without touching the kernel, that the injected fault
10+
// really refuses placement. Both facts are read from the real objects.
11+
#include <pineforge/source/pine_strategy_host.hpp>
12+
13+
#include <cstddef>
14+
#include <cstdint>
15+
#include <cstdio>
16+
#include <stdexcept>
17+
#include <string>
18+
#include <system_error>
19+
#include <type_traits>
20+
#include <vector>
21+
22+
namespace {
23+
using Adapter = pineforge::source::PineExecutionAdapter;
24+
25+
// The same explicit-instantiation access seam as test_id_adapter_storage.cpp:
26+
// no changed class definition or production friend.
27+
template<class Tag, auto Member> struct Access {
28+
friend auto access(Tag) { return Member; }
29+
};
30+
struct Cohorts { friend auto access(Cohorts); };
31+
struct CohortFor { friend auto access(CohortFor); };
32+
template struct Access<Cohorts, &Adapter::cohorts_by_id_>;
33+
template struct Access<CohortFor, &Adapter::cohort_for>;
34+
35+
class IoRefusalHost final : public pineforge::source::PineStrategyHost {
36+
public:
37+
IoRefusalHost() {
38+
pineforge::source::PineStrategyConfig config;
39+
config.initial_capital = 1e12;
40+
config.margin_long = config.margin_short = 0.0;
41+
config.pyramiding = 5000;
42+
configure_pine_strategy(config);
43+
set_syminfo_mintick(0.25);
44+
set_syminfo_timezone("UTC");
45+
set_syminfo_session("24x7");
46+
}
47+
48+
void on_source_bar(const pineforge::Bar&) override {
49+
if (pine_bar_index() == 0) probe();
50+
}
51+
52+
bool probed = false;
53+
std::string verdict;
54+
55+
private:
56+
void fail(const std::string& why) {
57+
if (verdict.empty()) verdict = why;
58+
}
59+
60+
void probe() {
61+
probed = true;
62+
auto& table = adapter_.*access(Cohorts{});
63+
// A regular file is not a directory: a scratch child of it cannot be created.
64+
table.set_directory("/dev/null/pineforge-cr365-r2-refused");
65+
// Setup proof, independent of cohort_for and of the kernel: the table
66+
// itself refuses to place a row, and stays empty.
67+
using Facts = std::decay_t<decltype(table.find(std::string())->second)>;
68+
bool setup_refused = false;
69+
try {
70+
table.emplace("cr365-setup-probe", Facts{});
71+
} catch (const std::system_error&) {
72+
setup_refused = true;
73+
}
74+
const auto logical = table.size();
75+
const auto first = cohort_open();
76+
const auto hash_before = broker_state_hash_projection();
77+
bool refused = false;
78+
std::string what;
79+
try {
80+
(void)(adapter_.*access(CohortFor{}))("cr365-admission-io-refused");
81+
} catch (const std::system_error& error) {
82+
refused = true;
83+
what = error.what();
84+
}
85+
const auto hash_after = broker_state_hash_projection();
86+
const auto second = cohort_open();
87+
std::printf("R2-A2 setup_refused=%d logical=%llu first_handle=%llu second_handle=%llu "
88+
"refused=%d same_hash=%d text=\"%s\"\n",
89+
setup_refused ? 1 : 0, static_cast<unsigned long long>(logical),
90+
static_cast<unsigned long long>(first.value),
91+
static_cast<unsigned long long>(second.value), refused ? 1 : 0,
92+
hash_after == hash_before ? 1 : 0, what.c_str());
93+
if (!setup_refused || logical != 0)
94+
return fail("SETUP: the injected scratch fault did not refuse a table placement");
95+
if (!refused)
96+
return fail("BEHAVIOR: cohort_for accepted a new id although the table refuses placement");
97+
if (table.size() != logical)
98+
return fail("BEHAVIOR: the refused cohort_for left a source row behind");
99+
if (second.value != first.value + 1)
100+
return fail("BEHAVIOR: the refused cohort_for consumed a host cohort handle (next="
101+
+ std::to_string(second.value) + " expected="
102+
+ std::to_string(first.value + 1) + ")");
103+
if (hash_after != hash_before)
104+
return fail("BEHAVIOR: the refused cohort_for changed the broker state");
105+
}
106+
};
107+
} // namespace
108+
109+
int main() {
110+
try {
111+
std::vector<pineforge::Bar> bars(4);
112+
for (std::size_t i = 0; i < bars.size(); ++i)
113+
bars[i] = {100.0, 101.0, 99.0, 100.0, 1000.0,
114+
1736121600000LL + static_cast<std::int64_t>(i) * 60000};
115+
IoRefusalHost host;
116+
host.run(bars.data(), static_cast<int>(bars.size()), "1", "1");
117+
if (!host.probed) throw std::runtime_error("SETUP: the probe bar was never reached");
118+
if (!host.verdict.empty()) throw std::runtime_error(host.verdict);
119+
std::puts("cohort_for admission io refusal: OK");
120+
return 0;
121+
} catch (const std::exception& error) {
122+
std::fprintf(stderr, "FAIL: %s\n", error.what());
123+
return 1;
124+
}
125+
}

0 commit comments

Comments
 (0)