Skip to content

Commit 925badb

Browse files
author
Qiming, Liu
committed
feat(improve-skills): add attribute-triggered red-line framework
Adds a proactive conformance gate keyed on what a target Skill *is*, so design invariants whose breach produces no error (e.g. focus theft) are caught and remediated mandatorily rather than only fixing what a run observed breaking. Four-step flow: (1) attribute collection, (2) red-line selection by attribute, (3) trigger judgment -> PASS/TRIGGERED, (4) mandatory remediation. Ships RL-FOCUS-01 (automation -> no automation may steal system focus). - scripts/redline-check.py: deterministic probe; verdict driven by mitigation presence (not raw grep count) to avoid flagging compliant reference impls; +/-2-line negation guard; hard vs soft signals; default scope = SKILL.md + scripts/ (references/ via --scan-all); self-excludes framework files. - references/red-lines.md: methodology, registry catalogue, admission rigor (5 criteria a rule must meet to be a red line, not a general constraint), and reconciliation with the evidence-before-defect constraint. - SKILL.md: step-4 gate, step-8 re-run gate, Hard Constraint github#11, Resources. Validated RED-GREEN (bad->gap/exit1, good->compliant/exit0, review->exit1) plus reverse cases (improve-skills/browser-utils/profiles-browsers -> exit0); shape gate exit0 (+426 tokens, under budget); contract tests zero-regression vs HEAD. Canonical skills/ + .specify/skills/ mirror kept in sync.
1 parent 0259ccf commit 925badb

6 files changed

Lines changed: 917 additions & 4 deletions

File tree

‎.specify/skills/improve-skills/SKILL.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Batch procedure, ownership resolution, and routing rationale: [`./references/loo
4949
- Give the user's stated optimization direction a dedicated analysis pass: confirm what is already satisfied, what is missing, and which edits address the request.
5050
- Group observations by failure mode: trigger/discovery, scope inference, missing context, wrong tool choice, unsafe step, unclear output, validation gap, resource/reference issue, **constraint non-compliance** (diagnose as a constraint-*placement* problem per [`./references/constraint-placement.md`](./references/constraint-placement.md) before rewording the rule), or **cross-skill ownership boundary**.
5151
- For each item, record: symptom, likely cause in the Skill instructions, desired next behavior, and the section to change. An item deferred for lack of evidence names the concrete evidence that would unlock it.
52+
- **Red-line conformance gate (proactive, mandatory)** — run `python3 ${SKILL_HOME}/scripts/redline-check.py <target SKILL.md>`: it collects the target's attributes (automation / sensitive-info / …), fires the red lines bound to each present attribute, and returns PASS or TRIGGERED. A TRIGGERED red line is a **MUST-fix** item — not evidence-gated, not deferrable: remediate it, or record why each flagged signal is a sanctioned, announced exception. Framework, registry, admission rigor, and the HC5 reconciliation: [`./references/red-lines.md`](./references/red-lines.md).
5253
- Gates before acting — misuse-vs-pitfall gate, fact-check gates (delegation capability / data tables / tier coverage), legacy path idiom detection (bare relative paths / `${SKILL_ROOT}/X` / agent-specific install paths → rewrite as `${SKILL_HOME}/...`), Feedback-section conformance: [`./references/loop-playbook.md`](./references/loop-playbook.md) `## Step 4`.
5354

5455
5. **按规范优化 — correct the root causes with minimal changes**
@@ -70,6 +71,7 @@ Batch procedure, ownership resolution, and routing rationale: [`./references/loo
7071
8. **最终检查 — validate the improvement loop**
7172
- Re-read the changed Skill and verify each edit maps to an observed execution issue or a user requirement.
7273
- **Run the shape gate on every `SKILL.md` you touched**: `python3 ${SKILL_HOME}/scripts/skill-shape.py <SKILL.md>` — exit `0` required, or the report carries a recorded exception per Hard Constraint 3, including the before/after controllable-token counts (step-2 baseline run vs this run) and their delta. Never finish a loop having grown a body past the gate without saying so.
74+
- **Re-run the red-line gate**: `python3 ${SKILL_HOME}/scripts/redline-check.py <SKILL.md>` must exit `0`, or the report records why each remaining signal is a sanctioned exception. A loop MUST NOT finish with a triggered red line left unremediated (see [`./references/red-lines.md`](./references/red-lines.md)).
7375
- **Reference code is executed, not eyeballed**: run or line-trace changed snippets/scripts against a real target; files-exist/links-resolve is not validation — if you cannot run it this loop, mark it as needing runtime validation rather than reporting it verified. When metadata changed, check frontmatter, resource paths, `skill_id`/directory/`name` agreement, and accept a directory-level `.github/skills -> ../.specify/skills` symlink as a valid compatibility entrypoint.
7476
- **Structural edits run the affected contract tests** (`tests/contract/` of the governing feature) — heading greps are not proof. On a red suite, prove zero regression with a clean-baseline failure-set diff (prefer `git worktree` over `git stash`): [`./references/loop-playbook.md`](./references/loop-playbook.md) `## Step 8`. **Behavior-changing edits get a pressure re-test (RED-GREEN)** per `create-skills/references/pressure-testing.md`; wording-only or resource-path edits are exempt — state which case applied.
7577
- Skill-owned executable resources belong in `./scripts/`, never documented as `.specify/scripts/`. **Intervention ledger (evidence-step Step E)**: when the run consumed findings evidence, write `intervention.json` into the baseline evidence-run directory (targetFinding / change / baselineRunId / expectedSignal); never claim "fixed" without the next-run before/after comparison.
@@ -115,8 +117,8 @@ On a genuine agent-specific obstacle (tool call unavailable, unexpected output f
115117

116118
| Directory | Contents |
117119
|-----------|----------|
118-
| `${SKILL_HOME}/references/` | `skill-slimming-principles.md`, `loop-playbook.md`, `skill-quality-checklist.md`, `hardening-examples.md`, `constraint-placement.md`, `claude-code-guide.md`, `copilot-guide.md` |
119-
| `${SKILL_HOME}/scripts/` | `skill-shape.py` — deterministic L1 shape gate for a `SKILL.md` (token budget, fence ratio, long fences, example sections); exit `0` pass / `10` slim-recommended; `--help` for thresholds and calibration |
120+
| `${SKILL_HOME}/references/` | `skill-slimming-principles.md`, `loop-playbook.md`, `skill-quality-checklist.md`, `hardening-examples.md`, `constraint-placement.md`, `red-lines.md` (attribute-triggered red-line framework, registry catalogue, admission rigor), `claude-code-guide.md`, `copilot-guide.md` |
121+
| `${SKILL_HOME}/scripts/` | `skill-shape.py` — deterministic L1 shape gate for a `SKILL.md` (token budget, fence ratio, long fences, example sections); exit `0` pass / `10` slim-recommended; `--help` for thresholds and calibration. `redline-check.py` — deterministic attribute-triggered red-line probe (collect attributes → fire bound red lines → verdict `not-applicable`/`gap`/`review`/`compliant`); exit `0` pass / `1` triggered; `--json`, `--scan-all`, `--help` |
120122

121123
## Hard Constraints
122124

@@ -132,6 +134,7 @@ Objective conditions for finishing a loop. Each is checkable, not a matter of ju
132134
8. **Removal preserves content.** Every slimming move is delete-and-absorb in the same edit; never delete a section and defer relocating its substance.
133135
9. **No claim of "fixed" without before/after.** Improvement outcomes are decided by the intervention ledger's next-run comparison, not by asserting the edit works.
134136
10. **Wrap-up commits verify the staging area.** Before any loop-end commit, `git status --short` and confirm only this loop's files are staged; unstage unrelated pre-staged entries or commit by explicit pathspec — never `git add -A`.
137+
11. **Red lines are mandatory, not recommended.** If `redline-check.py` returns TRIGGERED for the target, the loop MUST NOT finish until every triggered red line is remediated, or each flagged signal is explicitly recorded as a sanctioned, announced exception. Adding a new red line requires passing **all** admission criteria in [`./references/red-lines.md`](./references/red-lines.md) — a general best practice MUST NOT be promoted to a red line (that devalues the real ones).
135138

136139
## Self-Improvement Routing
137140

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# Red-Line Framework (attribute-triggered mandatory constraints)
2+
3+
This file owns the **red-line framework**: a proactive conformance gate that runs while
4+
improving a Skill. Unlike evidence-gated optimization (fix what a run observed breaking), a
5+
red line is a **design invariant** — its violation is a defect by definition, whether or not
6+
any execution crashed. The executable half is `${SKILL_HOME}/scripts/redline-check.py`; this
7+
file owns the methodology, the registry catalogue, and the admission rigor.
8+
9+
## Why this exists
10+
11+
The improvement loop was purely reactive: it optimized from observed execution evidence and
12+
user direction. That misses invariants whose breach produces no error — a browser-automation
13+
Skill that steals system focus "works" (the run completes) while silently disrupting the
14+
user's real work. Such invariants must be checked **proactively, keyed on what the Skill is**,
15+
and remediated **mandatorily**, not recommended.
16+
17+
## The generic 4-step flow
18+
19+
```
20+
1. attribute collection classify the target Skill from its own text/scripts
21+
(automation? sensitive-info? … — extensible)
22+
2. red-line selection map each present attribute to the red lines it fires
23+
(attribute present ≠ red line fires; only bound red lines run)
24+
3. trigger judgment evaluate compliance -> PASS (not-applicable / compliant)
25+
or TRIGGERED (gap / review)
26+
4. remediation TRIGGERED => MUST fix (or prove each signal a sanctioned,
27+
announced exception). Red-line remediation is never optional.
28+
```
29+
30+
Steps 1–3 are deterministic and run in `redline-check.py`. Step 4 is judgment + MUST-DO and
31+
stays with the improver. Run it in **workflow step 4** (before acting) and re-run it in
32+
**step 8** (must exit `0` before finishing).
33+
34+
```bash
35+
python3 ${SKILL_HOME}/scripts/redline-check.py <target/SKILL.md> # gate: exit 0 pass / 1 triggered
36+
python3 ${SKILL_HOME}/scripts/redline-check.py <target/SKILL.md> --json # machine verdict + evidence
37+
python3 ${SKILL_HOME}/scripts/redline-check.py <target/SKILL.md> --scan-all # deeper audit incl. references/
38+
```
39+
40+
## What the probe decides vs what stays judgment
41+
42+
| Deterministic (the probe) | Judgment (the improver) |
43+
|---------------------------|-------------------------|
44+
| which attributes are present | whether a flagged signal is a *sanctioned, announced exception* (e.g. a human-in-the-loop SSO login window) or a real violation |
45+
| which red lines therefore apply | the concrete remediation edit that satisfies the invariant |
46+
| where violation / compliance signals appear (file:line) | whether a `review` verdict's hard signals are all legitimately documented |
47+
48+
The probe never auto-convicts: a `review` verdict means "remediate **unless** you can point at
49+
why each signal is sanctioned". This division follows the skill's own principle — deterministic
50+
logic to code, judgment to the LLM.
51+
52+
### Verdict model (exit code driven by *mitigation presence*, not raw signal count)
53+
54+
| Verdict | Condition | Meaning |
55+
|---------|-----------|---------|
56+
| `not-applicable` | trigger attribute absent | red line does not fire → PASS |
57+
| `gap` | attribute present, **zero** focus-safe vocabulary | drives automation with no mitigation → MANDATORY remediation |
58+
| `review` | attribute present, mitigation present, un-negated **hard** signal remains | verify each is sanctioned, else fix → MANDATORY |
59+
| `compliant` | attribute present, mitigation present, no un-negated hard signal | PASS |
60+
61+
Why mitigation-presence and not signal-count: a reference implementation documents the
62+
*banned* mechanisms (in "rejected" tables, API listings, `never call X` prohibitions). Counting
63+
raw greps would flag the most-compliant skills — the calibration trap `skill-shape.py` warns
64+
about. So the probe (a) excludes `references/` from the default gate scope, (b) applies a ±2-line
65+
**negation guard** (a wrapped `never call\n bringToFront()` is a prohibition, not a violation),
66+
(c) splits **hard** signals (always worth judgment) from **soft** context signals (only matter
67+
when no mitigation exists), and (d) skips its own framework files. `--scan-all` opts into the
68+
deeper, knowingly-noisier reference audit.
69+
70+
## Red-line catalogue
71+
72+
The executable registry (detection patterns, compliance markers, remediation text) lives in
73+
`redline-check.py` `RED_LINES` — that is the SSOT for *detection*. This table is the
74+
human-facing index; keep ids in sync.
75+
76+
| id | trigger attribute | invariant (absolute) | mandatory remediation |
77+
|----|-------------------|----------------------|-----------------------|
78+
| `RL-FOCUS-01` | `automation` (drives a GUI/browser/desktop surface that maps real windows) | No automation method may steal the user's system focus or pop a window/tab in front of their work. | Route every launch/attach through a focus-safe path: background `open -g` (macOS) or headless / virtual display; attach to a running instance instead of relaunching; reuse background tabs; never `bringToFront()`; any genuinely-headed window is announced, small and fixed. Foreground/direct-binary launch only as an announced human-in-the-loop exception, gated + documented. See browser-utils `focus-safe-launch.md`, profiles-browsers SKILL.md § 焦点红线. |
79+
80+
Collected attributes with **no** red line bound yet (framework is extensible; binding one
81+
requires passing admission below): `sensitive-info` (handles credentials/tokens/cookies/PII).
82+
83+
## Admission criteria — what qualifies as a red line (rigor gate)
84+
85+
The remediation for a red line is **MUST**, so the bar for calling something a red line is high.
86+
A candidate qualifies **only if all** hold — otherwise it is a normal constraint (recommended,
87+
evidence-gated), and MUST NOT be added to `RED_LINES`:
88+
89+
1. **Protects an irreversible or user-harming invariant** — data loss, security/credential
90+
breach, destructive/irreversible action, or disruption of the user's own work (focus theft).
91+
Not efficiency, structure, style, or a preference.
92+
2. **Objectively detectable** — a static signal decides it (presence of a pattern / absence of
93+
required mitigation), not taste. If two competent reviewers would disagree on whether it was
94+
violated, it is not a red line.
95+
3. **Has one required alternative** — remediation is a single unambiguous MUST-DO, not a menu of
96+
acceptable options. Pair the prohibition with the required alternative (avoids negation-priming).
97+
4. **Universal within its trigger attribute** — it binds *every* Skill with the attribute, with
98+
at most narrow, nameable, announced exceptions (e.g. human-in-the-loop login). A rule with
99+
broad case-by-case discretion is a guideline, not a red line.
100+
5. **Worth blocking a loop over** — breaching it must be severe enough that finishing the
101+
improvement without fixing it is unacceptable.
102+
103+
Anti-pattern: promoting a general best practice ("prefer headless", "announce long runs",
104+
"keep scripts idempotent") to a red line. Those are constraints/recommendations; making them
105+
`MUST` devalues the real red lines and trains people to ignore the gate.
106+
107+
## Reconciliation with "Evidence before defect" (Hard Constraint 5)
108+
109+
HC5 forbids labeling something a defect without an observed symptom. A red-line violation does
110+
not contradict it: **the violating pattern's presence in the Skill's own instructions/scripts is
111+
the observed symptom** (a static artifact), exactly like a runtime stack trace is evidence. The
112+
probe surfaces that artifact; the improver still must not invent violations the probe did not
113+
find. `not-applicable` and `compliant` verdicts are never defects.
114+
115+
## Extending the framework
116+
117+
To add a red line: (1) confirm the candidate passes **all** admission criteria above; (2) add its
118+
trigger attribute signals to `ATTRIBUTES` (or reuse an existing attribute); (3) add a `RED_LINES`
119+
entry — `id`, `attribute`, `invariant`, `violation` patterns tagged `hard`/`soft`, `compliance`
120+
markers, `remediation` (the required alternative), `refs`; (4) add its row to the catalogue table
121+
here; (5) validate per the reverse-case discipline — run the probe against a compliant Skill
122+
(must NOT trigger) and a genuinely-violating sample (must trigger) before wiring it as a gate.

0 commit comments

Comments
 (0)