Skip to content

Commit f5e929d

Browse files
authored
Sync runtime history portably on macOS (#798)
Engine v1.11.0's source does not compile on macOS: `src/runtime_context.cpp` calls `::fdatasync` in the runtime history flush, and macOS does not declare it (`error: no member named 'fdatasync' in the global namespace`). The engine's own CI has no macOS leg, so it showed up only in a downstream macOS build. Engine release archives are Linux-only and are not affected. ## Change - `src/runtime_context.cpp`: on Apple platforms the flush uses `fcntl(F_FULLFSYNC)` with an `fsync` fallback, the pattern the other `fdatasync` users in the engine already carry. The EINTR retry and the storage error on failure are kept. The non-Apple `fdatasync` loop is unchanged. - `scripts/test_native_source_guard.py`: a source guard that fails when an engine source calls `fdatasync` without an Apple branch. ## Red first - The guard alone (first commit) fails on engine main `51919507` at `src/runtime_context.cpp`; with the fix (second commit) it passes. - The macOS compile itself is checked by a downstream macOS build with the engine at this head before merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent a447232 commit f5e929d

3 files changed

Lines changed: 224 additions & 3 deletions

File tree

‎docs/design/native-feature-parity.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,7 @@ spans, including nullable Calmar.
208208
| Targets | object libs `pineforge_kernel_objs` + `pineforge_source_objs`; `pineforge_kernel` STATIC, alias `PineForge::kernel`; `pineforge` STATIC = both | `pineforge_kernel` OBJECT; `pineforge` STATIC = kernel objects + conditional source list | `pineforge_kernel` + `pineforge` as the compatibility aggregate | F's shape. `pineforge` keeps its name, alias and install rules, so codegen-built strategies and `find_package(PineForge)` consumers do not move |
209209
| What moves | `compat/pine/market_admission.cpp` → source list | same + delete dead `admission_retention` | same ("a concrete boundary bug") | **Done.** The unit is in `PINEFORGE_SOURCE_LAYER_SOURCES` CMakeLists.txt:91 with the other five `compat/pine/` units, and `admission_retention` is deleted — the name is nowhere in `src/` or `include/`. Its callers were the adapter only and tests (both retired with the legacy loop in R4 slice C, `73817c1d`), so the move was link-neutral |
210210
| Installed headers | OFF skips `include/pineforge/source`, `include/pineforge/compat` | install conditional on the option | installed closure with source + compat headers physically removed | Conditional install; S's closure test is the acceptance |
211-
| CI / gate | link the two native examples + the 20 source-free native tests against `pineforge_kernel` only; `nm` assertion in the independence checker (it already forbids `pineforge::source` / `compat::pine` symbols check_native_include_independence.py:72-78) | new CI lane: configure with the source layer OFF, build, run the checker against that prefix | clean kernel-only compile; `nm` shows no source / compat symbol; keep the textual ban (`FORBIDDEN_INCLUDE` test_native_source_guard.py:43) as a fast preflight | All three, one CI lane | <!-- verified HEAD -->
211+
| CI / gate | link the two native examples + the 20 source-free native tests against `pineforge_kernel` only; `nm` assertion in the independence checker (it already forbids `pineforge::source` / `compat::pine` symbols check_native_include_independence.py:72-78) | new CI lane: configure with the source layer OFF, build, run the checker against that prefix | clean kernel-only compile; `nm` shows no source / compat symbol; keep the textual ban (`FORBIDDEN_INCLUDE` test_native_source_guard.py:44) as a fast preflight | All three, one CI lane | <!-- verified HEAD -->
212212
| Size | M | M | L | M (R5-11) |
213213

214214
### 2.ii Coupling extraction (L11 neutral, L12 hash-visible) — R5-8
@@ -222,7 +222,7 @@ spans, including nullable Calmar.
222222
| e | Dead helpers in the public kernel header | **Done.** All four are gone from `engine.hpp`: `round_to_mintick_directional` and `apply_slippage` were the pre-R5 spelling of the directional snap and the slippage step and the kernel now fills on `native_matching::grid_round_directional` / `native_matching::apply_slippage` src/native_matching.hpp:259 (the engine.hpp comment that named them went with the dead-path cleanup's residue sweep); `apply_limit_fill` and `reserve_percent_commission` are deleted outright, and only the adapter still spells the first two names, in its own comments. `round_to_mintick` engine.hpp:1048 and `bar_fill_price` engine.hpp:1066 stay, live through the call `bar_fill_price` native_execution_consumer.cpp:7823 | NEUTRAL | Reuse as the reference arithmetic of the price-grid and sizing work, delete the rest | O (single-source) |
223223
| f | Dead legacy stream path | **Done.** `stream_feed_input_bar` and `dispatch_source_stream_script_bar` are both gone — neither name is in `src/` or `include/` — and the virtual left with the epoch that carried it. `stream_state_hash` engine.hpp:2144 stays, and the declaration/definition pairing is still enforced by `scripts/test_native_source_guard.py` | NEUTRAL (removing the virtual changes the vtable → shared bump) | Delete both | O; F asked (its Q6) |
224224
| g | Legacy path resolver | **Done, and then deleted.** `src/engine_path_resolve.cpp` is 118 lines with five functions: `bar_path_uses_high_first`, both `first_touch_position` overloads, `fill_bar_path_points_ordered`, and `BacktestEngine::declare_opened_lot_entry_bar_mask`. Ten test files still mention `resolve_exit_path_fill` in historical comments; the resolver body and test-only oracle are gone. | NEUTRAL (pure move; verify with the link gate) | Keep the generic path helpers in the kernel TU; the old source resolver is not a live counterpart | F (single-source) | <!-- verified HEAD --> |
225-
| h | `pine_*`-named generic API | `timeframe_time` session_time.hpp:35-211 (live, e.g. `session_in_market`, which generated code reaches through `pine_session_ismarket` session_time.hpp:263), `str_utils.hpp`, `deterministic_random` math.hpp:9, `tz_util` timezone.hpp:8-30; the guard bans `pine_*` only inside `IDENTIFIER_ROOTS` (test_native_source_guard.py:34, test_native_source_guard.py:34-42) | NEUTRAL, M | Neutral aliases; `pine_*` kept as deprecated inlines for codegen. Large blast radius in generated code. **Status (L11 + N14): done** — session_time / str_utils / math / timezone in L11; N14 applied the same treatment to the two public matrix types: `PineMatrix` → `NumericMatrix`, `PineGenericMatrix<T>` → `GenericMatrix<T>`, old spellings kept as exact aliases (no version script pins either name) | O (single-source) |
225+
| h | `pine_*`-named generic API | `timeframe_time` session_time.hpp:35-211 (live, e.g. `session_in_market`, which generated code reaches through `pine_session_ismarket` session_time.hpp:263), `str_utils.hpp`, `deterministic_random` math.hpp:9, `tz_util` timezone.hpp:8-30; the guard bans `pine_*` only inside `IDENTIFIER_ROOTS` (test_native_source_guard.py:35, test_native_source_guard.py:35-43) | NEUTRAL, M | Neutral aliases; `pine_*` kept as deprecated inlines for codegen. Large blast radius in generated code. **Status (L11 + N14): done** — session_time / str_utils / math / timezone in L11; N14 applied the same treatment to the two public matrix types: `PineMatrix` → `NumericMatrix`, `PineGenericMatrix<T>` → `GenericMatrix<T>`, old spellings kept as exact aliases (no version script pins either name) | O (single-source) |
226226
| i | `pf_pending_order_v1_t` Pine-named fields | `tv_carry_qty` pending_order_mirror.hpp:73, `pine_birth_reach` pending_order_mirror.hpp:135, `pine_exit_activation_owner_cycle_at_birth` pending_order_mirror.hpp:147-156, `pine_frozen_market_instruction_kind` pending_order_mirror.hpp:164-169; enumerated by `PF_PO_FIELD` pending_order_mirror.cpp:144-178; written only by `pine_birth_reach` pine_adapter.cpp:27895 | no change | Freeze (ABI). The neutral view arrives beside it (L7 / L13) | O (single-source) |
227227
| j | Pine-shaped lot flags | **Mostly done.** `market_pyramid_add` moved to the adapter side table the row asked for — `market_pyramid_adds_` src/source/pine_state_hash.cpp:773, whose own comment records that it was `PyramidEntry::market_pyramid_add` — and `ordinary_market_open`, `pooc_terminal_market_entry`, `ordinary_stop_open`, `bracket_slot_shadowed` and `entry_path_position` are all deleted. What is left on the lot is the intrabar-fill excursion mask, `skip_entry_bar_high` / `skip_entry_bar_low` engine.hpp:169-170, hashed (`skip_entry_bar_high` engine_state_hash.cpp:80) and, since the detached-comment and mask cleanup, written by no host at all: a host that owns lot excursions declares where its fill sat (`declare_opened_lot_entry_bar_mask` engine_path_resolve.cpp:95; the Pine host did until the adapter state consolidation left its excursions to the kernel's sampler) and the kernel derives the pair | HASH-VISIBLE | Move the live flag to an adapter side table keyed by `entry_incarnation` (the adapter already keeps `PlacementTable` pine_adapter.hpp:591-968) and fold it through `hash_source_extension`; delete the dead ones; update the hash-coverage waivers. Whether a kernel TU beyond the hash reads them is unverified (O) | F O S |
228228
| k | `margin_call_enabled_` | **Done.** The generic kernel model is enabled by `NativeRunSpec::margin` native_run_spec.hpp:672. TradingView-specific toggling remains `source::PineStrategyHost::set_margin_call_enabled` pine_strategy_host.hpp:582; the frozen C symbol remains source-owned. | HASH-VISIBLE (R5-8) | Keep the source setter in the adapter; the kernel toggle is the presence of `spec.margin` | F O |

‎scripts/test_native_source_guard.py‎

Lines changed: 213 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,13 @@
11
#!/usr/bin/env python3
2-
"""Fail closed if a native/generic include root reaches a source/Pine header.
2+
"""Guard native source boundaries, member definitions and sync portability.
33
44
This is the inexpensive textual companion to check_native_include_independence:
55
the latter proves the installed closure and object symbols after a build; this
66
guard runs in preflight and rejects a forbidden include before compilation.
77
"""
88
from __future__ import annotations
99

10+
from bisect import bisect_right
1011
from contextlib import redirect_stderr
1112
import io
1213
from pathlib import Path
@@ -123,6 +124,13 @@ def blank(start: int, end: int) -> None:
123124
blank(index, end)
124125
index = end
125126
continue
127+
# C++17 digit separators are not opening character-literal quotes.
128+
if (text[index].isdigit()
129+
and (index == 0 or not re.match(r"\w", text[index - 1]))):
130+
index += 1
131+
while index < length and (text[index].isalnum() or text[index] in "_.'"):
132+
index += 1
133+
continue
126134
if text[index] in {'"', "'"}:
127135
quote = text[index]
128136
end = index + 1
@@ -298,7 +306,209 @@ def run_scan(pattern: re.Pattern[str], paths: list[Path], label: str,
298306
return 0
299307

300308

309+
def _apple_condition(expression: str) -> set[bool]:
310+
"""Possible truth values on Apple; unsupported syntax stays unknown."""
311+
tokens = re.findall(r"[A-Za-z_]\w*|\d+|&&|\|\||[!()]|\S", expression)
312+
position = 0
313+
314+
def take() -> str:
315+
nonlocal position
316+
if position == len(tokens):
317+
raise ValueError("incomplete condition")
318+
token = tokens[position]
319+
position += 1
320+
return token
321+
322+
def parse(level: int = 0) -> set[bool]:
323+
nonlocal position
324+
if level < 2:
325+
left = parse(level + 1)
326+
operator = ("||", "&&")[level]
327+
while position < len(tokens) and tokens[position] == operator:
328+
position += 1
329+
right = parse(level + 1)
330+
left = {(a or b) if level == 0 else (a and b)
331+
for a in left for b in right}
332+
return left
333+
token = take()
334+
if token == "!":
335+
return {not value for value in parse(2)}
336+
if token == "(":
337+
value = parse()
338+
if take() != ")":
339+
raise ValueError("unclosed condition")
340+
return value
341+
if token == "defined":
342+
token = take()
343+
if token == "(":
344+
token = take()
345+
if take() != ")":
346+
raise ValueError("unclosed defined")
347+
if not re.fullmatch(r"[A-Za-z_]\w*", token):
348+
raise ValueError("invalid defined")
349+
if token in {"__APPLE__", "1"}:
350+
return {True}
351+
if token in {"__linux__", "0"}:
352+
return {False}
353+
return {False, True}
354+
355+
try:
356+
result = parse()
357+
return result if position == len(tokens) else {False, True}
358+
except ValueError:
359+
return {False, True}
360+
361+
362+
def _apple_fdatasync(text: str) -> tuple[int, list[int]]:
363+
"""Count lexical calls and locate those not provably excluded on Apple."""
364+
# Splice before masking, as C++ does, including continued // comments.
365+
# Keep enough offset information to report physical source line numbers.
366+
splices = [match.start() - 2 * index for index, match in
367+
enumerate(re.finditer(r"\\\n", text))]
368+
masked = _mask_cpp(text.replace("\\\n", ""))
369+
calls: dict[int, list[int]] = {}
370+
count = 0
371+
for match in re.finditer(r"::\s*fdatasync\s*\(", masked):
372+
logical_line = masked.count("\n", 0, match.start()) + 1
373+
physical_offset = match.start() + 2 * bisect_right(splices, match.start())
374+
calls.setdefault(logical_line, []).append(
375+
text.count("\n", 0, physical_offset) + 1)
376+
count += 1
377+
if not count:
378+
return 0, []
379+
380+
# Each frame holds the parent's reachability, whether a later arm may
381+
# still be selected, and whether #else has already appeared.
382+
stack: list[tuple[bool, bool, bool]] = []
383+
reachable = True
384+
exposed: list[int] = []
385+
for line_number, line in enumerate(masked.splitlines(), 1):
386+
directive = re.match(r"^\s*#\s*(if|ifdef|ifndef|elif|else|endif)\b(.*)", line)
387+
if directive:
388+
kind, expression = directive.groups()
389+
if kind in {"if", "ifdef", "ifndef"}:
390+
if kind != "if":
391+
expression = ("!" if kind == "ifndef" else "") + (
392+
"defined(" + expression.strip() + ")")
393+
condition = _apple_condition(expression)
394+
stack.append((reachable, False in condition, False))
395+
reachable = reachable and True in condition
396+
else:
397+
if not stack or (kind != "endif" and stack[-1][2]):
398+
raise ValueError(f"line {line_number}: unexpected #{kind}")
399+
parent, remaining, _ = stack[-1]
400+
if kind == "endif":
401+
reachable = parent
402+
stack.pop()
403+
else:
404+
condition = _apple_condition(expression) if kind == "elif" else {True}
405+
reachable = parent and remaining and True in condition
406+
stack[-1] = (parent, remaining and False in condition, kind == "else")
407+
if reachable:
408+
exposed.extend(calls.get(line_number, []))
409+
if stack:
410+
raise ValueError("unterminated preprocessor conditional")
411+
return count, exposed
412+
413+
414+
def run_portability_scan(root: Path) -> int:
415+
"""Read every C/C++ source/header in both roots; fail on incomplete input."""
416+
suffixes = {".c", ".cc", ".cpp", ".cxx", ".h", ".hh", ".hpp", ".hxx", ".inc", ".inl"}
417+
findings: list[str] = []
418+
calls = 0
419+
path = root
420+
try:
421+
for relative in ("src", "include"):
422+
path = root / relative
423+
if not path.is_dir():
424+
raise OSError("missing source directory")
425+
pending = [path]
426+
files = 0
427+
while pending:
428+
path = pending.pop()
429+
if path.is_symlink():
430+
raise OSError("refusing symbolic link in source tree")
431+
if path.is_dir():
432+
pending.extend(sorted(path.iterdir(), reverse=True))
433+
continue
434+
if path.suffix.lower() not in suffixes:
435+
continue
436+
text = path.read_text(encoding="utf-8")
437+
files += 1
438+
count, exposed = _apple_fdatasync(text)
439+
calls += count
440+
findings.extend(
441+
f"{path}:{line}: ::fdatasync call may be compiled on Apple; "
442+
"exclude Apple and use its portable sync branch\n"
443+
for line in exposed)
444+
if not files:
445+
raise OSError("no C/C++ source/header inputs")
446+
if not calls:
447+
raise ValueError("no ::fdatasync calls scanned")
448+
except (OSError, UnicodeError, ValueError) as error:
449+
print(f"native source guard: portability scan failed: {path}: {error}",
450+
file=sys.stderr)
451+
return 1
452+
if findings:
453+
print("native source guard: Apple-exposed ::fdatasync call(s) found",
454+
file=sys.stderr)
455+
sys.stderr.writelines(findings)
456+
return 1
457+
return 0
458+
459+
460+
def portability_self_test() -> int:
461+
# Independent discriminators exercise the same scanner as the real tree;
462+
# no production path, call count or line number is an acceptance whitelist.
463+
cases = (
464+
("unguarded", "::fdatasync(fd);\n", 1, [1]),
465+
("Apple else", "#if defined(__APPLE__)\n::fsync(fd);\n"
466+
"#else\n::fdatasync(fd);\n#endif\n", 1, []),
467+
("Linux else", "#if defined(__linux__)\n::fdatasync(fd);\n"
468+
"#else\n::fsync(fd);\n#endif\n", 1, []),
469+
("unrelated Apple arm", "#ifndef __APPLE__\n::fsync(fd);\n#endif\n"
470+
"::fdatasync(fd);\n", 1, [4]),
471+
("Apple positive", "#if defined(__APPLE__)\n::fdatasync(fd);\n#endif\n", 1, [2]),
472+
("wrong nested arm", "#ifndef __APPLE__\n::fsync(fd);\n#else\n"
473+
"#if FEATURE\n::fdatasync(fd);\n#endif\n#endif\n", 1, [5]),
474+
("outer exclusion", "#ifndef __APPLE__\n#if FEATURE\n"
475+
"::fdatasync(fd);\n#endif\n#endif\n", 1, []),
476+
("elif exclusion", "#if FEATURE\n::fsync(fd);\n"
477+
"#elif defined(__linux__)\n::fdatasync(fd);\n"
478+
"#elif defined(__APPLE__)\n::fsync(fd);\n#endif\n", 1, []),
479+
("elif Apple", "#if defined(__linux__)\n::fsync(fd);\n"
480+
"#elif defined(__APPLE__)\n::fdatasync(fd);\n#endif\n", 1, [4]),
481+
("unknown else", "#if FEATURE\n::fsync(fd);\n#else\n"
482+
"::fdatasync(fd);\n#endif\n", 1, [4]),
483+
("boolean exclusion", "#if !defined(__APPLE__) && FEATURE\n"
484+
"::fdatasync(fd);\n#endif\n", 1, []),
485+
("boolean exposure", "#if !defined(__APPLE__) || FEATURE\n"
486+
"::fdatasync(fd);\n#endif\n", 1, [2]),
487+
("unsupported condition", "#if __APPLE__ == 1\n::fdatasync(fd);\n#endif\n", 1, [2]),
488+
("comments and strings", "// #ifndef __APPLE__\n/* __APPLE__ */\n"
489+
'const char* marker = "__APPLE__";\n::fdatasync(fd);\n', 1, [4]),
490+
("masked calls", "// ::fdatasync(fd);\n/* ::fdatasync(fd); */\n"
491+
'const char* text = "::fdatasync(fd);";\n'
492+
'const char* raw = R"tag(::fdatasync(fd);)tag";\n', 0, []),
493+
("continued condition", "#if !defined(\\\n__APPLE__)\n"
494+
"::fdatasync(fd);\n#endif\n", 1, []),
495+
("continued comment", "// ignored \\\n#ifndef __APPLE__\n"
496+
"::fdatasync(fd);\n", 1, [3]),
497+
("split call", "::\nfdatasync\n(fd);\n", 1, [1]),
498+
("digit separator", "auto size = 1'024;\n::fdatasync(fd);\n", 1, [2]),
499+
)
500+
for label, text, expected_count, expected in cases:
501+
count, exposed = _apple_fdatasync(text)
502+
if (count, exposed) != (expected_count, expected):
503+
print(f"native source guard: portability self-test failed: {label}: "
504+
f"{(count, exposed)!r} != {(expected_count, expected)!r}", file=sys.stderr)
505+
return 1
506+
return 0
507+
508+
301509
def self_test() -> int:
510+
if portability_self_test():
511+
return 1
302512
with tempfile.TemporaryDirectory(prefix="pf-native-source-guard-") as temporary:
303513
fixture = Path(temporary) / "include/pineforge/engine.hpp"
304514
fixture.parent.mkdir(parents=True)
@@ -363,6 +573,8 @@ def self_test() -> int:
363573
def main() -> int:
364574
if self_test():
365575
return 1
576+
if run_portability_scan(ROOT):
577+
return 1
366578
roots = native_roots()
367579
if not all(path.is_file() for path in roots):
368580
missing = [str(path) for path in roots if not path.is_file()]

‎src/runtime_context.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,9 +266,18 @@ struct RuntimeContextService::Impl {
266266
}
267267
}
268268
static void flush_pages(int file, std::uint64_t offset, std::uint64_t bytes) {
269+
#if defined(__APPLE__)
270+
int result;
271+
do {
272+
result = ::fcntl(file, F_FULLFSYNC) != -1
273+
? 0 : ::fsync(file);
274+
} while (result < 0 && errno == EINTR);
275+
if (result != 0) storage_error("runtime history sync");
276+
#else
269277
while (::fdatasync(file) != 0) {
270278
if (errno != EINTR) storage_error("runtime history sync");
271279
}
280+
#endif
272281
// No whole-file mapping or resident index. Drop clean backing pages;
273282
// the final partial page and kernel IO working space are fixed-size.
274283
discard_pages(file, offset, bytes);

0 commit comments

Comments
 (0)