Skip to content

Commit 0190528

Browse files
luisleo526claude
andcommitted
Admission journal: fold a new event in one validated, hashed pass over its encoding
The admission journal fold used to scan an event record to validate it and then read the same record a second time to hash it, so every appended event cost two reads of each of its pages (16 source reads for a 3,773-byte record on 512-byte pages). A new fused fold, fold_validated_journal_record, validates and hashes the same forward field stream in a single pass (8 reads for that record) on both the bound and the unbound path. The standalone scanner and folder remain for the other consumers. The result is identical: digest, summary, sequence, rejection gate and the published frontier are unchanged, and a failure publishes the same status and diagnostic as before. After a long-text skip the old scanner's refill alignment differs from the fused reader's, so on a source read failure the fused fold replays the validation once to recover the old error priority; it never retries the hash and never turns the original read failure into success. The workspace is a fixed constant: a compile-time ledger keeps the fold's own state, diagnostic transport, summaries and controls within 1,024 bytes, with a 256-byte reused field buffer and no new heap allocation. No state survives the call. Tests: the fold-cost row now counts work (reads, encoder words and writer-status checks per new event, derived from the schema's field geometry and the page size, with the unchanged window costing zero), a failure-publication row corrupts a stored record and checks that nothing is published, and a new differential unit compares the fused fold with the separate composition for all event kinds, text and array sizes, page sizes, malformed encodings and byte-range I/O faults. The schema checker and its mutation control name the new call. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent fc0a2df commit 0190528

11 files changed

Lines changed: 681 additions & 13 deletions

‎docs/design/native-feature-parity.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1119,7 +1119,7 @@ A `sample` call tree of each run splits the adapter fold: the placement
11191119
snapshots are 55 % of it on the order-and-cancel shape and 97 % on the
11201120
re-issue shape, the journal reflection the other 45 % and 1 %, the scheduler
11211121
prefix under 2 %. Most of a snapshot's bytes are field paths the reflection
1122-
builds as strings, one per field (`Reflect::field` market_admission.cpp:658-661).
1122+
builds as strings, one per field (`Reflect::field` market_admission.cpp:662-665).
11231123
11241124
Why it is not fixed in this lane. The fold can be made incremental without
11251125
moving a single value, but not from inside lane F9's files:

‎include/pineforge/detail/journal_codec.hpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ struct WorkObserver {
1818

1919
// Borrowed registration on the calling thread. A nonnull observer must have a
2020
// valid callback and outlive its registration; restore the returned pointer.
21+
// payload_source/read cover stored and admission-encoded sources alike.
2122
// payload_read reports requested bytes; the other operations report zero.
2223
inline thread_local const WorkObserver* work_observer = nullptr;
2324
inline const WorkObserver* set_work_observer(const WorkObserver* observer) noexcept {

‎scripts/check_market_admission_schema.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ def check(root: Path = ROOT) -> int:
3131
for token in (
3232
"void reflect(const Draft& value", "void reflect(const Event& value",
3333
"void Journal::reflect", "Journal::Prefix Journal::folded_window() const",
34-
"fold_journal_record(source,page,result.digest)",
34+
"fold_validated_journal_record(source,page,summary,result.digest,",
3535
):
3636
if token not in source:
3737
raise ValueError("admission reflection/hash fold is missing: " + token)

‎scripts/test_market_admission_schema.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ def test_missing_adapter_hash_is_rejected(self):
3838
try:
3939
for relative, token in (
4040
("src/source/market_admission.cpp",
41-
"fold_journal_record(source,page,result.digest)"),
41+
"fold_validated_journal_record(source,page,summary,result.digest,"),
4242
("src/source/pine_state_hash.cpp",
4343
"const auto folded = admission_journal.folded_window();"),
4444
("src/source/pine_state_hash.cpp",

‎src/source/admission_journal_codec.cpp‎

Lines changed: 132 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -98,14 +98,72 @@ struct Save {
9898
}
9999
};
100100

101-
template<bool Owning> struct Load {
101+
class FoldSource final : public detail::ScratchByteSource {
102+
const detail::ScratchByteSource& source_;
103+
public:
104+
mutable Status failure = Status::ok;
105+
mutable std::optional<detail::ReportScratchIoError> carried;
106+
explicit FoldSource(const detail::ScratchByteSource& source) : source_(source) {}
107+
std::uint64_t size() const noexcept override { return source_.size(); }
108+
Status read(std::uint64_t offset, detail::ScratchBuffer out) const noexcept override {
109+
const auto status = source_.read(offset,out);
110+
if (status != Status::ok && failure == Status::ok) {
111+
failure = status;
112+
if (status == Status::io_error)
113+
if (const auto* error = source_.carried_failure()) carried = *error;
114+
}
115+
return status;
116+
}
117+
};
118+
struct CanonicalFold {
119+
std::uint64_t digest;
120+
std::uint64_t after_sequence;
121+
bool active = true;
122+
bool validation_skipped = false;
123+
void bytes(const char* data, std::size_t size) noexcept {
124+
for (std::size_t i = 0; i != size; ++i)
125+
digest = (digest ^ static_cast<unsigned char>(data[i])) * 1099511628211ULL;
126+
}
127+
void word(std::uint64_t value) noexcept {
128+
for (unsigned i = 0; i != 8; ++i)
129+
digest = (digest ^ ((value >> (8 * i)) & 255)) * 1099511628211ULL;
130+
}
131+
template<class T> void scalar(const T& value) noexcept {
132+
if constexpr (std::is_same_v<T, double>) {
133+
std::uint64_t bits = 0;
134+
std::memcpy(&bits, &value, sizeof bits);
135+
if ((bits & 0x7fffffffffffffffULL) == 0) bits = 0;
136+
else if ((bits & 0x7ff0000000000000ULL) == 0x7ff0000000000000ULL
137+
&& (bits & 0x000fffffffffffffULL)) bits = 0x7ff8000000000000ULL;
138+
word(bits);
139+
} else word(static_cast<std::uint64_t>(value));
140+
}
141+
};
142+
constexpr std::size_t kJournalFoldWorkspaceLimit = 1024;
143+
constexpr std::size_t kJournalFoldFieldBytes = 256;
144+
// Fixed candidate/read-failure state, two diagnostic transport slots, two
145+
// summaries and eight words for scalar/pointer controls, plus the existing
146+
// comparison buffer reused for text. Nothing is retained between calls.
147+
// ReportScratchIoError copies share their prepared message/argument payload.
148+
// The Reader page and non-owning schema frame remain the existing workspace.
149+
constexpr std::size_t kJournalFoldWorkspaceBytes = sizeof(CanonicalFold) + sizeof(FoldSource)
150+
+ 2 * sizeof(std::optional<detail::ReportScratchIoError>) + 2 * sizeof(JournalRecordSummary)
151+
+ 8 * sizeof(std::uint64_t) + kJournalFoldFieldBytes;
152+
static_assert(kJournalFoldWorkspaceBytes <= kJournalFoldWorkspaceLimit,
153+
"fixed admission fold workspace");
154+
static_assert(std::is_nothrow_copy_constructible_v<detail::ReportScratchIoError>
155+
&& std::is_nothrow_copy_assignable_v<detail::ReportScratchIoError>,
156+
"diagnostic transport must be nonthrowing");
157+
158+
template<bool Owning, bool Folding = false> struct Load {
102159
static constexpr bool loading = true;
103160
static constexpr bool owning = Owning;
104161
jc::Reader& in;
105162
std::uint64_t rows = 0, expected_rows = 0;
106163
JournalRecordSummary summary;
107164
Outcome outcome = Outcome::NoAdmission;
108165
int bar = -1;
166+
CanonicalFold* fold = nullptr;
109167
bool good() const { return in.status == Status::ok; }
110168
template<class T> void field(const Path& path, const char* name, T& value) {
111169
if (!good()) return;
@@ -114,13 +172,39 @@ template<bool Owning> struct Load {
114172
const auto length = in.word();
115173
if (!good()) return;
116174
if (length != key.length) { in.status = Status::corrupt; return; }
117-
std::array<char, 256> bytes{};
175+
std::array<char, kJournalFoldFieldBytes> bytes{};
118176
in.bytes(bytes.data(), key.length);
119177
if (!good()) return;
120178
if (std::memcmp(bytes.data(), key.bytes.data(), key.length) || in.word() != tag<T>()) {
121179
in.status = Status::corrupt; return;
122180
}
123-
if constexpr (std::is_same_v<T, std::string>) in.text(value, Owning);
181+
if constexpr (Folding) {
182+
if (fold->active) {
183+
fold->word(length); fold->bytes(bytes.data(), key.length); fold->word(tag<T>());
184+
}
185+
}
186+
if constexpr (std::is_same_v<T, std::string>) {
187+
if constexpr (!Folding) in.text(value, Owning);
188+
else {
189+
const auto size = in.word();
190+
if (!good()) return;
191+
if (size > in.remaining() || size > value.max_size()) {
192+
in.status = Status::corrupt; return;
193+
}
194+
if (!fold->active) in.skip(size);
195+
else {
196+
fold->word(size);
197+
const auto end = in.position + size;
198+
if (end > in.cached_at + in.cached_size) fold->validation_skipped = true;
199+
while (in.position != end && good()) {
200+
const auto count = static_cast<std::size_t>(
201+
std::min<std::uint64_t>(bytes.size(), end - in.position));
202+
in.bytes(bytes.data(), count);
203+
if (good()) fold->bytes(bytes.data(), count);
204+
}
205+
}
206+
}
207+
}
124208
else if constexpr (std::is_same_v<T, double>) in.real(value);
125209
else if constexpr (std::is_same_v<T, bool>) {
126210
const auto raw = in.word();
@@ -130,11 +214,16 @@ template<bool Owning> struct Load {
130214
std::underlying_type_t<T> raw{}; in.integer(raw); value = static_cast<T>(raw);
131215
} else in.integer(value);
132216
if constexpr (std::is_integral_v<T> || std::is_enum_v<T>) {
133-
if (key.is("event.observation.command") || key.is("event.receipt.sequence"))
217+
if (key.is("event.observation.command") || key.is("event.receipt.sequence")) {
134218
summary.sequence = static_cast<std::uint64_t>(value);
219+
if constexpr (Folding)
220+
if (summary.sequence <= fold->after_sequence) fold->active = false;
221+
}
135222
if (key.is("event.outcome")) outcome = static_cast<Outcome>(value);
136223
if (key.is("event.observation.bar")) bar = static_cast<int>(value);
137224
}
225+
if constexpr (Folding && !std::is_same_v<T, std::string>)
226+
if (good() && fold->active) fold->scalar(value);
138227
}
139228
};
140229

@@ -328,17 +417,18 @@ template<class A> struct Schema {
328417
}
329418
};
330419

331-
template<bool Owning> Status decode(const detail::ScratchByteSource& source,
420+
template<bool Owning, bool Folding = false> Status decode(const detail::ScratchByteSource& source,
332421
detail::ScratchBuffer page, Event& event, JournalRecordSummary& summary,
333-
const detail::ReportScratchIoError*& failure) noexcept {
422+
const detail::ReportScratchIoError*& failure, CanonicalFold* fold = nullptr) noexcept {
334423
failure = nullptr;
335424
if (!page.data || !page.size) return Status::programming_error;
336425
try {
337426
jc::Reader in{source,page};
338-
Load<Owning> load{in};
427+
Load<Owning,Folding> load{in};
428+
load.fold = fold;
339429
load.expected_rows = in.word();
340430
if (load.expected_rows > in.remaining() / 24) return Status::corrupt;
341-
Schema<Load<Owning>>{load}.event(event);
431+
Schema<Load<Owning,Folding>>{load}.event(event);
342432
if (load.rows != load.expected_rows && in.status == Status::ok) in.status = Status::corrupt;
343433
const auto status = in.finish();
344434
if (status == Status::ok) {
@@ -356,6 +446,7 @@ template<bool Owning> Status decode(const detail::ScratchByteSource& source,
356446
} // namespace
357447

358448
JournalRecordSource::JournalRecordSource(const Event& event) noexcept : event_(&event) {
449+
jc::observe_work(jc::WorkOperation::payload_source);
359450
try {
360451
jc::Writer counter;
361452
Save count{counter,true};
@@ -375,6 +466,7 @@ detail::ScratchStatus JournalRecordSource::read(std::uint64_t offset,
375466
if (offset > length_ || buffer.size > length_ - offset || (buffer.size && !buffer.data))
376467
return Status::programming_error;
377468
if (!buffer.size) return Status::ok;
469+
jc::observe_work(jc::WorkOperation::payload_read, buffer.size);
378470
try {
379471
// Forward pages resume within the current array element. A seek before
380472
// the saved field boundary (including the second pass) starts anew.
@@ -412,6 +504,38 @@ detail::ScratchStatus scan_journal_record(const detail::ScratchByteSource& sourc
412504
const detail::ReportScratchIoError* failure = nullptr;
413505
return decode<false>(source,page,scratch,summary,failure);
414506
}
507+
detail::ScratchStatus fold_validated_journal_record(const detail::ScratchByteSource& source,
508+
detail::ScratchBuffer page, JournalRecordSummary& summary, std::uint64_t& digest,
509+
std::uint64_t after_sequence, std::optional<detail::ReportScratchIoError>* failure) noexcept {
510+
if (failure) failure->reset();
511+
Event scratch;
512+
FoldSource input(source);
513+
CanonicalFold fold{digest,after_sequence};
514+
JournalRecordSummary candidate;
515+
const detail::ReportScratchIoError* unused = nullptr;
516+
const auto status = decode<false,true>(input,page,scratch,candidate,unused,&fold);
517+
if (input.failure != Status::ok && fold.validation_skipped) {
518+
// A skipped text region changes the old scan's refill boundaries.
519+
// On a read failure only, replay that scan to retain its validation
520+
// priority (including its tag-read error handling). Never retry the
521+
// hash or turn the original read failure into a successful query.
522+
JournalRecordSummary checked;
523+
const auto validation = scan_journal_record(source,page,checked);
524+
if (validation != Status::ok) {
525+
if (failure && validation == Status::io_error)
526+
if (const auto* error = source.carried_failure()) *failure = *error;
527+
return validation;
528+
}
529+
if (failure) *failure = input.carried;
530+
return input.failure;
531+
}
532+
if (failure && status == Status::io_error) *failure = input.carried;
533+
if (status == Status::ok) {
534+
summary = candidate;
535+
if (summary.sequence > after_sequence) digest = fold.digest;
536+
}
537+
return status;
538+
}
415539
detail::ScratchStatus fold_journal_record(const detail::ScratchByteSource& source,
416540
detail::ScratchBuffer page, std::uint64_t& digest) noexcept {
417541
if (!page.data || !page.size) return Status::programming_error;

‎src/source/admission_journal_codec.hpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
#pragma once
22
#include <pineforge/source/market_admission.hpp>
3+
#include <pineforge/detail/scratch_arena.hpp>
34

45
namespace pineforge::admission {
56
inline namespace market_admission_v2 {
@@ -20,5 +21,13 @@ detail::ScratchStatus scan_journal_record(const detail::ScratchByteSource&,
2021
detail::ScratchBuffer, JournalRecordSummary&) noexcept;
2122
detail::ScratchStatus fold_journal_record(const detail::ScratchByteSource&,
2223
detail::ScratchBuffer, std::uint64_t&) noexcept;
24+
// Validate the schema and extend the canonical fold in one forward pass.
25+
// A sequence at/below the frontier is validated without extending the digest.
26+
// Neither value output is published on failure. The optional diagnostic owns
27+
// the original I/O failure across any validation-only error replay.
28+
detail::ScratchStatus fold_validated_journal_record(const detail::ScratchByteSource&,
29+
detail::ScratchBuffer, JournalRecordSummary&, std::uint64_t&,
30+
std::uint64_t after_sequence = 0,
31+
std::optional<detail::ReportScratchIoError>* failure = nullptr) noexcept;
2332
}
2433
}

‎src/source/market_admission.cpp‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -271,9 +271,13 @@ Journal::Prefix Journal::folded_window() const {
271271
auto retained_rejected_bar = retired_prefix_.last_rejected_command_bar;
272272
const auto append = [&](const detail::ScratchByteSource& source, detail::ScratchBuffer page) {
273273
JournalRecordSummary summary;
274-
CodecState::check(scan_journal_record(source,page,summary), codec_.get());
274+
std::optional<detail::ReportScratchIoError> failure;
275+
const auto status = fold_validated_journal_record(source,page,summary,result.digest,
276+
result.last_sequence,&failure);
277+
if (status == detail::ScratchStatus::io_error && failure)
278+
detail::rethrow_carried_scratch_failure(&*failure);
279+
CodecState::check(status, codec_.get());
275280
if (summary.sequence <= result.last_sequence) return;
276-
CodecState::check(fold_journal_record(source,page,result.digest), codec_.get());
277281
if (result.count == std::numeric_limits<std::size_t>::max())
278282
throw coded<std::overflow_error>(RunFailureCode::engine_invariant, {},
279283
"admission event count exhausted");
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
if(PINEFORGE_BUILD_SOURCE_LAYER)
2+
list(APPEND TEST_SOURCES test_admission_streaming_fold)
3+
pineforge_add_unit_test(test_admission_streaming_fold)
4+
endif()

0 commit comments

Comments
 (0)