buzz-audit: every hash chain fails verification — created_at is hashed at nanosecond precision but stored at microsecond precision #2637

Summary

buzz-audit writes each entry's hash over a nanosecond-precision created_at, then stores that timestamp in a TIMESTAMPTZ column, which Postgres keeps at microsecond precision. verify_chain re-reads the truncated value and recomputes the digest, so it never reproduces the one that was written.

The result: every audit chain backed by a real database fails verification at its first entry, on untampered data. The hash chain — the crate's whole purpose, and what ARCHITECTURE.md:497 describes as the tamper-detection mechanism — provides no tamper evidence today. Worse, a genuinely forged row is indistinguishable from the baseline failure, so HashMismatch carries no signal at all.

Verified on main @ 6a56c8bd.

Root cause

Three lines that individually look fine:

1. Write — crates/buzz-audit/src/service.rs:103

   let created_at: DateTime<Utc> = Utc::now();   // nanosecond resolution

2. Hash — crates/buzz-audit/src/hash.rs:24

   hasher.update(entry.created_at.to_rfc3339().as_bytes());

   to_rfc3339() uses SecondsFormat::AutoSi, which emits 0/3/6/9 fractional digits depending on the value. A nanosecond timestamp is hashed as …T12:00:00.123456789+00:00.

3. Store — migrations/0001_initial_schema.sql:615

   created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),

   TIMESTAMPTZ resolution is 1 µs. The sub-microsecond digits are gone.

verify_chain (service.rs:188) then recomputes over …T12:00:00.123456+00:00 — a different byte string, therefore a different digest — and returns HashMismatch at service.rs:190.

This fires whenever Utc::now()'s nanosecond component is not an exact multiple of 1000: ~100% of entries on Linux (clock_gettime, ns granularity), ~90% on Windows (100 ns ticks).

The pattern predates the multi-tenant rewrite — git log -S shows timestamp.to_rfc3339() in the hash preimage back at d99ad131, so this has never worked.

Reproduction

Postgres discards the sub-microsecond digits:

$ docker run --rm -d --name pg -e POSTGRES_PASSWORD=x -p 55432:5432 postgres:16-alpine
$ docker exec pg psql -U postgres -t -c \
    "CREATE TABLE t (ts timestamptz);
     INSERT INTO t VALUES ('2026-07-24T12:00:00.123456789Z');
     SELECT to_char(ts, 'YYYY-MM-DD\"T\"HH24:MI:SS.US') FROM t;"

 2026-07-24T12:00:00.123457

The repo's own audit tests then fail against a real database. They are #[ignore = "requires Postgres"], so this is normally never executed:

$ docker run --rm -d --name pg -e POSTGRES_USER=buzz -e POSTGRES_PASSWORD=buzz_dev \
    -e POSTGRES_DB=buzz -p 5432:5432 postgres:17-alpine
$ for f in migrations/*.sql; do docker exec -i pg psql -U buzz -d buzz -q < "$f"; done
$ cargo test -p buzz-audit --lib -- --ignored --test-threads=1

running 6 tests
test service::tests::chain_links_within_one_community ... FAILED
test service::tests::chains_are_independent_per_community ... FAILED
test service::tests::community_chain_starts_at_seq_1_with_null_prev ... ok
test service::tests::cross_community_row_does_not_verify ... ok
test service::tests::verify_detects_tampering_within_a_community ... FAILED
test service::tests::verify_empty_range_is_false ... ok

---- service::tests::chain_links_within_one_community stdout ----
panicked at crates\buzz-audit\src\service.rs:347:14:
called `Result::unwrap()` on an `Err` value: HashMismatch { seq: 2 }

---- service::tests::chains_are_independent_per_community stdout ----
panicked at crates\buzz-audit\src\service.rs:396:14:
called `Result::unwrap()` on an `Err` value: HashMismatch { seq: 1 }

---- service::tests::verify_detects_tampering_within_a_community stdout ----
panicked at crates\buzz-audit\src\service.rs:446:9:
assertion failed: matches!(r, Err(AuditError::HashMismatch { seq }) if seq == e2.seq)

test result: FAILED. 3 passed; 3 failed

Note the third failure specifically: the tamper-detection test is itself invalidated. It expects HashMismatch at the tampered entry's seq, but verification already fails at an earlier, untampered row — so the assertion that proves tamper detection works has been passing-by-absence, never actually exercised.

Of the three that "pass": verify_empty_range_is_false and community_chain_starts_at_seq_1_with_null_prev don't recompute hashes, and cross_community_row_does_not_verify asserts a failure — which it now gets for the wrong reason.

Why CI is green

• All six buzz-audit chain tests are #[ignore = "requires Postgres"] (service.rs:296, 316, 354, 415, 453, 490).
• The in-process tests in hash.rs never round-trip through Postgres, and their fixture uses "2026-01-01T00:00:00Z" — zero sub-seconds, the one value where the bug cannot appear.
• crates/buzz-relay/src/handlers/event.rs:1946 (audit_chain_is_isolated_per_tenant_through_relay_ingest) is un-ignored and asserts "A's chain must verify independently", but it early-returns with "skipping audit isolation test: Postgres/Redis unavailable" when infra is absent.

Impact

• No deployment can detect audit-log tampering. verify_chain fails identically on clean and forged data.
• AuditError::HashMismatch is unusable as an alarm — it is the steady state.
• This blocks buzz-audit: hash chain is tamper-evident in schema but not operationally verifiable; verify_chain accepts truncated chains #2620. That issue proposes a shared verify_entries walk with per-entry hash recompute plus a buzz-admin audit verify command. Its six new Postgres-free unit tests build entries in memory and would pass, but its four #[ignore] Postgres tests and the operator command itself would fail on every real chain until this is fixed. Worth landing this first so that work has a verifiable baseline.

Suggested fix

Reduce created_at to the stored precision before hashing, so the in-memory entry and the row are byte-identical:

use chrono::SubsecRound;
let created_at: DateTime<Utc> = Utc::now().trunc_subsecs(6);

log_inner is the only place that assigns created_at (every caller goes through NewAuditEntry, which carries no timestamp), so it is a single choke point.

Existing rows written before the fix stay unverifiable — they always were — so this needs no migration, but operators with a chain they care about would need to re-anchor.

I have this implemented with three Postgres-free regression tests (the invariant, the trap that caused it, and the round-trip) plus a non-ignored test on the write path, and all six Postgres chain tests now pass. PR incoming.