Skip to content

Commit 1f674c8

Browse files
committed
harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <[email protected]>
1 parent f2e97ff commit 1f674c8

15 files changed

Lines changed: 1098 additions & 234 deletions

File tree

‎Makefile.cbm‎

Lines changed: 81 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,12 @@ GCC_ONLY_FLAGS := -Wno-format-truncation -Wno-unused-result \
4343
-Wno-stringop-truncation -Wno-alloc-size-larger-than
4444
endif
4545

46+
# -Wdate-time: with -Werror, any use of __DATE__/__TIME__/__TIMESTAMP__ fails the
47+
# build. Build-time entropy is how one identical source tree produced a new hash
48+
# on every compile (see the local patch in vendored/mimalloc/src/options.c) —
49+
# once removed, it must not creep back through a new call site.
4650
CFLAGS_COMMON = -std=c11 -D_DEFAULT_SOURCE -D_GNU_SOURCE -Wall -Wextra -Werror \
47-
-Wno-unused-parameter -Wno-sign-compare \
51+
-Wno-unused-parameter -Wno-sign-compare -Wdate-time \
4852
$(GCC_ONLY_FLAGS) \
4953
-Isrc -Ivendored -Ivendored/sqlite3 \
5054
-Ivendored/mimalloc/include \
@@ -54,19 +58,32 @@ CXXFLAGS_COMMON = -std=c++14 -Wall -Wextra -Werror \
5458
-Wno-unused-parameter \
5559
-I$(CBM_DIR) -I$(TS_INCLUDE)
5660

61+
# Test seams are OPT-IN, never opt-out. Some suites drive behaviour that only a
62+
# test should be able to ask for (fork an orphan the watchdog must reap, publish
63+
# a lease-ownership marker). That code has no production caller and reads exactly
64+
# like malware to a generic classifier, so it must not be in a shipped binary.
65+
# Opt-IN means the failure mode of forgetting the flag is a CLEAN binary rather
66+
# than a leaky one — the opposite choice would make every future release depend
67+
# on someone remembering. scripts/test.sh passes TEST_SEAMS=1 for the suites that
68+
# need it; the test-runner always has them.
69+
TEST_SEAM_DEFINE :=
70+
ifeq ($(TEST_SEAMS),1)
71+
TEST_SEAM_DEFINE := -DCBM_ENABLE_TEST_SEAMS=1
72+
endif
73+
5774
# Production flags (CFLAGS_EXTRA allows CI to inject -DCBM_VERSION)
5875
# CBM_BIND_TS_ALLOCATOR=1: bind the tree-sitter runtime to mimalloc (#424). Only
5976
# the prod build uses mimalloc (MI_OVERRIDE=1); the test build is CRT+ASan, where
6077
# binding would create an alloc/free mismatch, so the guard is prod-only.
61-
CFLAGS_PROD = $(CFLAGS_COMMON) -O2 -DCBM_BIND_TS_ALLOCATOR=1 $(CFLAGS_EXTRA)
78+
CFLAGS_PROD = $(CFLAGS_COMMON) -O2 -DCBM_BIND_TS_ALLOCATOR=1 $(TEST_SEAM_DEFINE) $(CFLAGS_EXTRA)
6279
CXXFLAGS_PROD = $(CXXFLAGS_COMMON) -O2
6380

6481
# Test flags: debug + sanitizers (override SANITIZE= to disable on Windows)
6582
SANITIZE = -fsanitize=address,undefined -fno-omit-frame-pointer
6683
EDITOR_TEST_DEFINES = -DCBM_JSON_LIKE_ENABLE_TEST_API=1 \
6784
-DCBM_TOML_EDIT_ENABLE_TEST_API=1 -DCBM_YAML_ENABLE_TEST_API=1 \
6885
-DCBM_TEXT_EDIT_ENABLE_TEST_API=1 -DCBM_CLI_ENABLE_TEST_API=1 \
69-
-DCBM_DIAGNOSTICS_ENABLE_TEST_API=1
86+
-DCBM_DIAGNOSTICS_ENABLE_TEST_API=1 -DCBM_ENABLE_TEST_SEAMS=1
7087
# The build system is the single source of truth for "is this binary
7188
# instrumented": compiler-specific probes (__SANITIZE_ADDRESS__) miss
7289
# clang's feature-check spelling and every non-ASan sanitizer, so the
@@ -120,22 +137,37 @@ ifeq ($(IS_LINUX),yes)
120137
MIMALLOC_WRAP_FLAGS_POSIX := $(foreach sym,$(MIMALLOC_WRAP_SYMS_POSIX),-Wl,--wrap=$(sym))
121138
endif
122139
ifeq ($(IS_MINGW),yes)
123-
WIN32_LIBS := -lws2_32 -lpsapi -lshell32 -ladvapi32 -Wl,--allow-multiple-definition -Wl,--stack,8388608 -static
140+
# --no-insert-timestamp: the PE header otherwise carries the link wall clock, so
141+
# two Windows builds of identical source are never byte-identical and every
142+
# release is a brand-new file to a reputation system. Same motivation as the
143+
# mimalloc __DATE__ patch; ELF and Mach-O have no equivalent field to clear.
144+
WIN32_LIBS := -lws2_32 -lpsapi -lshell32 -ladvapi32 -Wl,--allow-multiple-definition -Wl,--stack,8388608 -Wl,--no-insert-timestamp -static
124145
endif
125146

126147
# STATIC=1 produces a fully static binary (for Alpine/musl portable builds)
127148
ifeq ($(STATIC),1)
128149
STATIC_FLAGS := -static
129150
endif
130151

152+
# W^X: demand a non-executable stack on ELF. This is belt to the braces of the
153+
# .note.GNU-stack annotation in vendored/nomic/code_vectors_blob.S — that note
154+
# fixes the CAUSE (an unannotated object makes ld assume the worst for the whole
155+
# link), this flag fixes the OUTCOME, and the composition gate proves it on the
156+
# shipped artifact. ELF-only: Apple's ld rejects -z noexecstack outright and it
157+
# is meaningless for PE, so it is gated rather than made "common".
158+
ELF_HARDENING_FLAGS :=
159+
ifeq ($(IS_LINUX),yes)
160+
ELF_HARDENING_FLAGS := -Wl,-z,noexecstack
161+
endif
162+
131163
# The POSIX wrap shim exists only so the profiler can observe allocations. It
132164
# must never reach a sanitized build: the Linux/macOS test builds are CRT+ASan,
133165
# and redirecting malloc into mimalloc underneath ASan's own interception mixes
134166
# two allocators on the same pointers. Windows keeps its wrap flags everywhere,
135167
# because there the shim is what makes mimalloc own the allocations at all.
136-
LDFLAGS = -lm -lstdc++ -lpthread -lz $(WIN32_LIBS) $(STATIC_FLAGS) $(MIMALLOC_WRAP_FLAGS)
137-
LDFLAGS_TEST = -lm -lstdc++ -lpthread -lz $(SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS)
138-
LDFLAGS_TSAN = -lm -lstdc++ -lpthread -lz $(TSAN_SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS)
168+
LDFLAGS = -lm -lstdc++ -lpthread -lz $(WIN32_LIBS) $(STATIC_FLAGS) $(MIMALLOC_WRAP_FLAGS) $(ELF_HARDENING_FLAGS)
169+
LDFLAGS_TEST = -lm -lstdc++ -lpthread -lz $(SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS) $(ELF_HARDENING_FLAGS)
170+
LDFLAGS_TSAN = -lm -lstdc++ -lpthread -lz $(TSAN_SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS) $(ELF_HARDENING_FLAGS)
139171

140172
# ── Source files ─────────────────────────────────────────────────
141173

@@ -358,9 +390,20 @@ MIMALLOC_CFLAGS_TEST = -std=c11 -g -O1 -w \
358390
# sqlite3 (vendored amalgamation — compiled ourselves for ASan instrumentation)
359391
# SQLITE_ENABLE_FTS5: enables the FTS5 full-text search extension used by the
360392
# BM25 search path in search_graph (see nodes_fts virtual table in store.c).
393+
#
394+
# SQLITE_OMIT_LOAD_EXTENSION: no cbm code calls sqlite3_load_extension or
395+
# sqlite3_enable_load_extension, and we never want a graph DB to be able to pull
396+
# a shared library into the process. Loading is off by default at runtime, but
397+
# "disabled by default" is a setting while "not compiled in" is a property —
398+
# this compiles the machinery out, removing the API surface and part of the
399+
# dlopen/dlsym import surface with it. The composition gate asserts the symbols
400+
# are absent from every release artifact.
401+
SQLITE3_OMIT_FLAGS = -DSQLITE_OMIT_LOAD_EXTENSION
361402
SQLITE3_SRC = vendored/sqlite3/sqlite3.c
362-
SQLITE3_CFLAGS = -std=c11 -O2 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5
363-
SQLITE3_CFLAGS_TEST = -std=c11 -g -O1 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5
403+
SQLITE3_CFLAGS = -std=c11 -O2 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5 \
404+
$(SQLITE3_OMIT_FLAGS)
405+
SQLITE3_CFLAGS_TEST = -std=c11 -g -O1 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5 \
406+
$(SQLITE3_OMIT_FLAGS)
364407

365408
# TRE regex (vendored, Windows only — POSIX uses system <regex.h>)
366409
TRE_SRC = vendored/tre/tre_all.c
@@ -647,7 +690,14 @@ $(BUILD_DIR)/mimalloc.o: $(MIMALLOC_SRC) | $(BUILD_DIR)
647690
$(BUILD_DIR)/tsan_mimalloc.o: $(MIMALLOC_SRC) | $(BUILD_DIR)
648691
$(CC) $(MIMALLOC_CFLAGS_TEST) $(TSAN_SANITIZE) -c -o $@ $<
649692

650-
$(BUILD_DIR)/prod_mimalloc.o: $(MIMALLOC_SRC) | $(BUILD_DIR)
693+
# static.c is an AMALGAMATION: it #includes options.c and friends, none of which
694+
# make can see through the single named prerequisite, and the compile flags live
695+
# in this Makefile. Both gaps bit for real — the __DATE__ removal in options.c and
696+
# SQLITE_OMIT_LOAD_EXTENSION below each silently did nothing on an incremental
697+
# build because the object was considered up to date. Depend on the included
698+
# sources AND on this Makefile so a flag change rebuilds too.
699+
$(BUILD_DIR)/prod_mimalloc.o: $(MIMALLOC_SRC) $(wildcard vendored/mimalloc/src/*.c) \
700+
$(wildcard vendored/mimalloc/include/*.h) Makefile.cbm | $(BUILD_DIR)
651701
$(CC) $(MIMALLOC_CFLAGS) -c -o $@ $<
652702

653703
# sqlite3 object files (vendored amalgamation)
@@ -661,7 +711,7 @@ $(BUILD_DIR)/sqlite3.o: $(SQLITE3_SRC) | $(BUILD_DIR)
661711
$(BUILD_DIR)/tsan_sqlite3.o: $(SQLITE3_SRC) | $(BUILD_DIR)
662712
$(CC) $(SQLITE3_CFLAGS_TEST) $(TSAN_SANITIZE) -c -o $@ $<
663713

664-
$(BUILD_DIR)/prod_sqlite3.o: $(SQLITE3_SRC) | $(BUILD_DIR)
714+
$(BUILD_DIR)/prod_sqlite3.o: $(SQLITE3_SRC) Makefile.cbm | $(BUILD_DIR)
665715
$(CC) $(SQLITE3_CFLAGS) -c -o $@ $<
666716

667717
# TRE regex (only compiled on Windows — POSIX uses system <regex.h>)
@@ -835,7 +885,26 @@ OBJS_VENDORED_PROD = $(MIMALLOC_OBJ_PROD) $(SQLITE3_OBJ_PROD) $(TRE_OBJ_PROD) $(
835885

836886
MAIN_SRC = src/main.c
837887

838-
$(BUILD_DIR)/codebase-memory-mcp: $(MAIN_SRC) $(PROD_SRCS) $(EXTRACTION_SRCS) $(AC_LZ4_SRCS) $(ZSTD_SRCS) $(SQLITE_WRITER_SRC) $(OBJS_VENDORED_PROD) | $(BUILD_DIR)
888+
# Rebuild when the build CONFIGURATION changes, not only when sources do. The
889+
# product binary is compiled in ONE shot from sources, so after a TEST_SEAMS or
890+
# version flip make finds the binary newer than every source and skips the recipe
891+
# entirely, handing back a binary built with the PREVIOUS configuration. The
892+
# dangerous direction is a release build silently keeping test seams, which is a
893+
# property no reviewer can see by reading the diff. The stamp's CONTENT is the
894+
# configuration and it is rewritten only when that content changes, so ordinary
895+
# incremental builds are unaffected. (Single-quoted: the signature may contain
896+
# the escaped quotes of -DCBM_VERSION, which are literal inside '', but must not
897+
# contain a literal single quote.)
898+
BUILD_CONFIG_SIG := TEST_SEAMS=$(TEST_SEAMS)|CFLAGS_EXTRA=$(CFLAGS_EXTRA)
899+
.PHONY: build-config-check
900+
build-config-check:
901+
$(BUILD_DIR)/.build-config: build-config-check | $(BUILD_DIR)
902+
@printf '%s\n' '$(BUILD_CONFIG_SIG)' > [email protected]
903+
@if cmp -s [email protected] $@; then rm -f [email protected]; \
904+
else mv -f [email protected] $@; rm -f $(BUILD_DIR)/codebase-memory-mcp; \
905+
echo "=== build config changed -> full rebuild: $(BUILD_CONFIG_SIG) ==="; fi
906+
907+
$(BUILD_DIR)/codebase-memory-mcp: $(MAIN_SRC) $(PROD_SRCS) $(EXTRACTION_SRCS) $(AC_LZ4_SRCS) $(ZSTD_SRCS) $(SQLITE_WRITER_SRC) $(OBJS_VENDORED_PROD) $(BUILD_DIR)/.build-config | $(BUILD_DIR)
839908
$(CC) $(CFLAGS_PROD) -o $@ \
840909
$(MAIN_SRC) $(PROD_SRCS) \
841910
$(EXTRACTION_SRCS) $(AC_LZ4_SRCS) $(ZSTD_SRCS) $(SQLITE_WRITER_SRC) \

0 commit comments

Comments
 (0)