Skip to content

Re-baseline the upstream surface watch at Claude Code 2.1.300; hold cached-artifact checks to the baseline - #1155

Open
seathatflowsinourveins wants to merge 2 commits into
mainfrom
claude/cc-native-practice-surface-rebaseline-20261015
Open

seathatflowsinourveins wants to merge 2 commits into
mainfrom
claude/cc-native-practice-surface-rebaseline-20261015

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes, in one or two sentences: The upstream surface watch is re-baselined at Claude Code 2.1.300, up from 2.1.296. The cached-artifact tests now check the parser against the committed baseline instead of literal counts pinned to one release. At base, four tests fail on any host whose cache is at 2.1.300. The command center approved this follow-up at 2026-10-15T01:48Z.
  • Base commit: f165498aa
  • Lane: lane:foundation
  • Owned paths touched:
    • catalogs/foundation/upstream-surface-baseline.json;
    • tests/test_upstream_surface_watch.py;
    • docs/upstream-surface-watch.md, lines 91-92 reworded with the same line count;
    • manifests/evidence.json.

Re-baseline: scripts/upstream_surface_watch.py --network --write-baseline --force ran at 2026-10-15T06:38:40Z under a scratch XDG_STATE_HOME. Every source came from the network, and the host's watch state was unchanged (31 entries, mtimes and sizes compared).

Versions:

  • Claude Code (npm latest): 2.1.296 → 2.1.300.
  • claude-agent-sdk: 0.3.296 → 0.3.300, with claudeCodeVersion 2.1.300.
  • Codex: unchanged, rust-v0.162.1 and codex-cli 0.162.1.

Count changes:

  • claude:env: 417 → 422.
  • claude:setting from settings-reference.md: 171 → 172. The union stays 184, and sdk.d.ts stays 174.
  • Hooks (33), mods (6), codex:config (1057) and codex:feature (167) are unchanged.
  • Nothing was removed.

Test change:

  • Same file: when the cached artifact is the one the baseline recorded (same sha256), the parse must equal the baseline's names.
  • Newer file: otherwise the count only has to stay above the watch's own 80% floor.
  • Not circular: a one-key-drop mutant fails, so the check doesn't just compare the escaped copy with the original.

New names with no disposition row are left undecided for the dispositions review:

  • CLAUDE_CODE_ARTIFACT_PUBLISH_EDITS
  • CLAUDE_CODE_DISABLE_FILE_MENTIONS
  • CLAUDE_CODE_IDLE_COMPACT_MIN_TOKENS
  • CLAUDE_CODE_YOU_SHOULD_KNOW_DEBUG

SOTA sources

These are the upstream artifacts the new baseline records:

Evidence-class table

Claim Evidence class Command / receipt
The watched versions and counts above native_proven the re-baseline run at 06:38:40Z; the committed baseline
The tests catch parser regressions synthetic mutation runs (below)

Local commands run

$ python3 -m unittest tests.test_upstream_surface_watch   # base f165498aa, host cache at 2.1.300, no UPSTREAM_SURFACE_TEST_CACHE
test_equivalent_string_escapes_preserve_the_real_sdk_hook_and_settings_sets: AssertionError: Tuples differ: (33, 174) != (33, 173)
test_escaped_quoted_setting_preserves_the_real_sdk_settings_set: AssertionError: 174 != 173
test_commonmark_closing_markers_on_every_heading_preserve_the_real_settings: AssertionError: 172 != 171
test_incomplete_schema_is_rejected_daily_and_when_writing_baseline: AssertionError: 1057 != 1034
$ python3 -m unittest tests.test_upstream_surface_watch tests.test_currency_due tests.test_currency_due_notice
Ran 233 tests, OK
# mutation runs on a scratch copy of the watch script, against the host cache:
#   escape handling broken                    -> 2 failures
#   closing hashes not stripped               -> 5 errors
#   one sdk.d.ts key dropped from every parse -> 4 failures
#   unresolved $ref ignored                   -> 4 failures
#   one schema path dropped                   -> 1 failure
#   control (unmutated, old and new baseline) -> pass
$ python3 scripts/validate.py
rc 0; {"components": 71, "hashed_files": 12047, "profiles": 4, "receipts": 240, "status": "passed"}

Fold (4313309)

This is the fold for the command center's FULL read at bf9153a (PASS):

Finding Fix Fail-first test
P2: the four names the re-baseline added had no disposition row, so the baseline grandfathered them out of unreviewed and currency_due (upstream_surface_watch.py:1597-1600) A baseline-unreviewed row for each name. This is the catalog's value for this case (upstream_surface_watch.py:210-220; docs/upstream-surface-watch.md:192-193). #1131's rebase turns these rows into rulings. test_every_name_a_re_baseline_added_has_a_disposition_row compares the baseline with the 2.1.296 baseline at f165498 and requires a row for every added name. It failed at bf9153a on the four names.

Scope of the invariant: it covers names a re-baseline adds. The 1,566 names that predate the 2.1.296 baseline were grandfathered under the earlier rule, so it is not a check on every baseline name.

Dropped sources rows: the new baseline no longer records the claude-doc-docs-index and claude-doc-platform-release-notes fetches. No enabled doc row reads those pages, so this is the script's correct output (docs/upstream-surface-watch.md:96-98).

Notes for the reads

  • CC FULL P3 (tests/test_upstream_surface_watch.py:61-76): the count-plus-subset check passes when one key is swapped for another, one for one.

  • Floor gap: when the cache is newer than the baseline, a parser that silently loses one key passes until the next re-baseline. The exact-match path catches it when the cache and the baseline agree.

  • Unreviewed names: the four names above are now in the baseline, so they leave the daily diff until the dispositions review adds their rows. claude:doc:memory and claude:doc:skills still differ from their reviewed digests.

  • Follow-up: the dispositions catalog's baseline_versions still says 2.1.296.

  • Test scope: only the three named suites and validate ran, not the full sharded suite.

Checklist

  • No GitHub Actions changed.
  • No workflows changed.
  • No secrets are printed, logged or committed.
  • No new paid hosting, subscription or billing surface.
  • Peer-owned untracked files and worktrees were preserved. The host's surface-watch state was not touched.

🤖 Generated with Claude Code

…d cached-artifact checks to the baseline

The command center's re-baseline follow-up, approved 2026-10-15T01:48Z. Four
CachedArtifactTests pinned counts read from an older cache (SDK 0.3.289, Codex
rust-v0.160.0) and failed against the host's watch cache at 2.1.300.

Re-baseline: `python3 -E -s -B scripts/upstream_surface_watch.py --network
--write-baseline --force`, run 2026-10-15T06:38:40Z under a scratch
XDG_STATE_HOME. Every source came from the network (coverage.from_cache empty),
and the host's watch state directory was left unchanged (mtimes compared).

Watched versions, baseline before -> after:
- Claude Code, npm latest: 2.1.296 -> 2.1.300 (dist-tags: stable 2.1.292,
  next 2.1.300)
- @anthropic-ai/claude-agent-sdk: 0.3.296 -> 0.3.300 (claudeCodeVersion
  2.1.300, matched)
- Codex stable release rust-v0.162.1 and binary codex-cli 0.162.1: unchanged

Counts that changed:
- claude:env: 417 -> 422 (+5 names)
- claude:setting from settings-reference.md: 171 -> 172 (a key sdk.d.ts
  already typed; the union stays 184 and sdk.d.ts stays 174)
Unchanged: claude:hook 33, claude:mod 6, codex:config 1057, codex:feature 167.
No name was removed and no feature stage changed. The baseline no longer
records the claude-doc-docs-index and claude-doc-platform-release-notes fetches,
because no enabled doc row reads those pages any more.

Not dispositioned here (the dispositions catalog is a separate review). Four
of the five new env names have no row, and the new baseline now holds them, so
daily diffs will not list them again:
- claude:env:CLAUDE_CODE_ARTIFACT_PUBLISH_EDITS
- claude:env:CLAUDE_CODE_DISABLE_FILE_MENTIONS
- claude:env:CLAUDE_CODE_IDLE_COMPACT_MIN_TOKENS
- claude:env:CLAUDE_CODE_YOU_SHOULD_KNOW_DEBUG
The fifth, claude:env:CLAUDE_CODE_OVERLOADED_RETRY_MAX_DELAY_MS, already has a
row. The claude:doc:memory and claude:doc:skills bodies differ from their
reviewed digests and stay unreviewed. The dispositions catalog's
baseline_versions still names 2.1.296.

Tests: CachedArtifactTests now hold the parse of each unmodified cached
artifact to the committed baseline instead of a release-specific literal (173,
(33, 173), 171, 1034, and the docs-only 10 of two more tests in the class).
When the cached body is the artifact the baseline recorded (same sha256), the
names must equal the baseline's. For a per-source parse, the count must match
and the names must lie within the kind. When the cache holds another release,
the count must keep the watch's own 80% floor (usw.floor_failure), and at least
one docs-only key must exist. The comparisons with reformatted, escaped and
corrupted copies are unchanged. One test loses "ten" from its name, now
test_closing_markers_on_the_docs_only_headings_preserve_the_real_settings.

Fail-first, at f165498 against the host cache with no
UPSTREAM_SURFACE_TEST_CACHE: four failures (172 != 171; (33, 174) != (33,
173); 174 != 173; 1057 != 1034). After: all six CachedArtifactTests pass
against the host cache and against the scratch cache the baseline was built
from.

Mutations, in a scratch copy of the watch script, against the host cache:
- u-escape decoding removed from ts_string_value: 2 failures (both escape tests)
- ATX closing hashes no longer stripped: 5 errors (both closing-marker tests)
- one sdk.d.ts key lost on every parse: 4 failures (173 != 174; docs-only
  11 != 10), which the escaped-copy comparisons alone would miss
- unresolved $ref ignored: 4 failures; one schema path lost: 1 failure
With the base commit's 2.1.296 baseline (floor branch for sdk.d.ts and the
settings page), the unmutated tests pass, and the lost-key mutant also passes:
the floor tolerates a 20% loss.

docs/upstream-surface-watch.md no longer ties these checks to SDK 0.3.289 and
Codex rust-v0.160.0; its line count is unchanged.

Checks: python3 scripts/validate.py exits 0 after registering the three
changed files in manifests/evidence.json. python3 -m unittest
tests.test_upstream_surface_watch tests.test_currency_due
tests.test_currency_due_notice ran 233 tests, OK.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…nvariant

The command center's FULL read of #1155 at bf9153a (PASS, one fold): the four env names the 2.1.300
re-baseline added (CLAUDE_CODE_ARTIFACT_PUBLISH_EDITS, CLAUDE_CODE_DISABLE_FILE_MENTIONS,
CLAUDE_CODE_IDLE_COMPACT_MIN_TOKENS, CLAUDE_CODE_YOU_SHOULD_KNOW_DEBUG) had no disposition row, so the
baseline grandfathered them out of unreviewed and currency_due (upstream_surface_watch.py:1597-1600). Each gets a
baseline-unreviewed row (the catalog's value for this case, upstream_surface_watch.py:210-220,
docs/upstream-surface-watch.md:192-193); #1131's rebase turns them into rulings.

test_every_name_a_re_baseline_added_has_a_disposition_row compares the baseline with the 2.1.296 baseline at
f165498 (from git history) and requires a row for every added name. Names that predate that baseline (1,566) are
grandfathered under the earlier rule, so the invariant covers re-baseline additions, not every baseline name.
Fail-before at bf9153a: the four names. After: tests.test_upstream_surface_watch, test_currency_due and
test_currency_due_notice OK; --check-dispositions valid (367 rows); validate rc 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant