Skip to content

Commit 8a2c015

Browse files
authored
docs: Update the threat model and associated reporting skill (agent-substrate#3355)
The security-status-report skill passes every column of `docs/threats.json` to its evaluators, so implementation-status claims and stale notes in the threat model were steering the scores instead of the code. This rewrites the suggested mitigations and notes as short design guidance with no status, and keeps threats and invariants unchanged except: - New Trust Boundaries section: the sandbox is the primary boundary, the worker Pod is trusted with the actors it hosts (its own confinement is defense in depth for the worker-to-node boundary), and a threat that needs another control to fail first is High priority at most - T-12 removed (documentation alone could never close it); other IDs keep their numbers - T-01 to T-03 lowered to High, since they require the outer network layer to fail first - T-11 invariant: the official path for secrets keeps them out of the sandbox - T-21 lowered to High, since it requires a sandbox escape first. Each worker permission beyond the container defaults needs a documented reason; the ones the sandbox runtime needs while workers share the host user namespace (root, runtime capabilities, AppArmor Unconfined) are accepted. A worker must not be able to modify or execute anything atelet or other workers trust - T-25 reworded to a malicious, not corrupt, snapshot, and lowered to Medium, since it requires control of snapshot storage first (same attackers as T-39) - T-26: enforcement points may cache policy as long as changes take effect within a short time, and must deny rather than serve stale policy indefinitely. Open connections and already-issued credentials are out of scope - T-30 lowered to Medium with the TBD invariant replaced, since nodes are not a boundary between actors Skill changes: evaluators score against the invariants rather than treating suggested mitigations as a checklist, don't penalize items a threat puts outside Substrate's scope or that its notes accept with a documented reason, and use a defined 0.0 to 1.0 scale. Each evaluator also writes a short mitigation summary and remaining gap; the new `render_table.py` renders them as a markdown table sorted by priority, and the chart now groups threats by priority. AI use: AI-assisted. I reviewed each section, and validated the rewrite with before/after evaluator runs and a full skill run. - [x] Tests pass - [x] Docs updated Signed-off-by: Greg Castle <[email protected]>
1 parent 659832f commit 8a2c015

8 files changed

Lines changed: 301 additions & 140 deletions

File tree

‎.agents/skills/security-status-report/SKILL.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,6 @@ Generate a security status report by evaluating threats listed in `docs/threats.
1616
4. Read `.agents/scratch/security-status-report/subagents.json` and copy its exact JSON array into your tool for invoking subagents. DO NOT manually craft or bypass the invocations. Run ALL generated sub-agents concurrently in a single tool call by default unless bound by model rate limits or harness concurrency limits (use batching intelligently if needed). You already updated the script in step 2 to output exactly what you need, so no modifications to the output should be necessary unless you made a mistake or hit limits.
1717
5. Wait for all sub-agents to complete.
1818
6. Run `python3 .agents/skills/security-status-report/scripts/compile_report.py docs/threats.json .agents/scratch/security-status-report .agents/scratch/security-status-report/final.json` to produce the final report.
19-
7. Run `python3 .agents/skills/security-status-report/scripts/render_chart.py .agents/scratch/security-status-report/final.json .agents/scratch/security-status-report/chart.png` to render a bar chart of the scores.
19+
7. Run `python3 .agents/skills/security-status-report/scripts/render_chart.py .agents/scratch/security-status-report/final.json .agents/scratch/security-status-report/chart.png` to render a bar chart of the scores, grouped by priority.
20+
8. Run `python3 .agents/skills/security-status-report/scripts/render_table.py .agents/scratch/security-status-report/final.json .agents/scratch/security-status-report/report.md` to render a markdown table of scores, mitigation summaries, and remaining gaps, highest priority first.
2021

‎.agents/skills/security-status-report/scripts/compile_report.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@ def compile_report(threats_json_path, results_dir, output_path):
6060
t["strengths"] = str(res["strengths"])
6161
if "weaknesses" in res:
6262
t["weaknesses"] = str(res["weaknesses"])
63+
for key in ("mitigation_summary", "remaining_gap"):
64+
if key in res:
65+
t[key] = str(res[key])
6366

6467
# Validate and normalize citations schema
6568
if "citations" in res:

‎.agents/skills/security-status-report/scripts/render_chart.py‎

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,17 +24,24 @@
2424
print("Error: 'matplotlib' is required to render the chart. Please install it using 'pip install matplotlib'.", file=sys.stderr)
2525
sys.exit(1)
2626

27+
PRIORITY_ORDER = ["Critical", "High", "Medium", "Low"]
28+
29+
30+
def sort_key(item):
31+
"""Orders threats by priority (Critical first), then by threat ID."""
32+
p = item.get("priority", "")
33+
rank = PRIORITY_ORDER.index(p) if p in PRIORITY_ORDER else len(PRIORITY_ORDER)
34+
tid = item.get("threat_id", "")
35+
if tid.startswith("T-") and tid[2:].isdigit():
36+
return (rank, 0, int(tid[2:]), "")
37+
return (rank, 1, 0, tid)
38+
39+
2740
def render_chart(final_report_path, output_png_path):
2841
with open(final_report_path, 'r') as f:
2942
data = json.load(f)
3043

31-
# Sort data by threat_id (e.g. T-01, T-02, ...)
32-
def sort_key(item):
33-
tid = item.get("threat_id", "")
34-
if tid.startswith("T-") and tid[2:].isdigit():
35-
return (0, int(tid[2:]), "")
36-
return (1, 0, tid)
37-
44+
# Sort by priority (Critical first), then by threat_id (T-01, T-02, ...).
3845
sorted_data = sorted(data, key=sort_key)
3946

4047
threat_ids = [item.get("threat_id", f"UNKNOWN-{i+1:02d}") for i, item in enumerate(sorted_data)]
@@ -90,11 +97,22 @@ def sort_key(item):
9097
if err:
9198
ax.text(i, 0.02, "ERROR", rotation=90, ha='center', va='bottom', fontsize=8, fontweight='bold', color='white')
9299

100+
# Separate and label the priority groups.
101+
priorities = [item.get("priority", "") for item in sorted_data]
102+
start = 0
103+
for i in range(1, len(priorities) + 1):
104+
if i == len(priorities) or priorities[i] != priorities[start]:
105+
if start > 0:
106+
ax.axvline(start - 0.5, color='black', linewidth=2)
107+
ax.text((start + i - 1) / 2, 1.01, priorities[start] or "Unknown", ha='center', va='bottom',
108+
fontsize=10, fontweight='bold', transform=ax.get_xaxis_transform())
109+
start = i
110+
93111
ax.set_ylim(0.0, 1.0)
94112
ax.set_xlim(-0.5, len(threat_ids) - 0.5)
95113
ax.set_ylabel("Quality Score (0.0 - 1.0)", fontsize=12, fontweight='bold')
96114
ax.set_xlabel("Threat ID", fontsize=12, fontweight='bold')
97-
ax.set_title("Substrate Security Threat Posture Scores", fontsize=16, fontweight='bold', pad=15)
115+
ax.set_title("Substrate Security Threat Posture Scores", fontsize=16, fontweight='bold', pad=25)
98116

99117
# Standard matplotlib way to rotate tick labels cleanly
100118
plt.xticks(rotation=90, fontsize=9, fontweight='bold')
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Copyright 2026 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Renders final.json as a markdown table, highest priority first."""
16+
17+
import json
18+
import os
19+
import statistics
20+
import sys
21+
22+
PRIORITY_ORDER = ["Critical", "High", "Medium", "Low"]
23+
24+
25+
def sort_key(item):
26+
p = item.get("priority", "")
27+
rank = PRIORITY_ORDER.index(p) if p in PRIORITY_ORDER else len(PRIORITY_ORDER)
28+
tid = item.get("threat_id", "")
29+
if tid.startswith("T-") and tid[2:].isdigit():
30+
return (rank, 0, int(tid[2:]), "")
31+
return (rank, 1, 0, tid)
32+
33+
34+
def cell(value):
35+
return str(value).replace("|", "/").replace("\n", " ").strip()
36+
37+
38+
def render_table(data):
39+
rows = sorted(data, key=sort_key)
40+
scored = [t["quality"] for t in rows if "error" not in t and "quality" in t]
41+
lines = []
42+
if scored:
43+
lines += [f"Mean score: {statistics.mean(scored):.2f}", ""]
44+
lines += [
45+
"| Threat | Priority | Score | Mitigation summary | Remaining gap |",
46+
"| :---- | :---- | :---- | :---- | :---- |",
47+
]
48+
for t in rows:
49+
if "error" in t:
50+
score, mitigation, gap = "ERROR", "", cell(t["error"])
51+
else:
52+
score = f"{float(t.get('quality', 0.0)):.2f}"
53+
mitigation = cell(t.get("mitigation_summary", ""))
54+
gap = cell(t.get("remaining_gap", ""))
55+
lines.append(f"| {cell(t.get('threat_id', ''))} | {cell(t.get('priority', ''))} | {score} | {mitigation} | {gap} |")
56+
return "\n".join(lines) + "\n"
57+
58+
59+
if __name__ == '__main__':
60+
report_file = sys.argv[1] if len(sys.argv) > 1 else ".agents/scratch/security-status-report/final.json"
61+
output_md = sys.argv[2] if len(sys.argv) > 2 else ".agents/scratch/security-status-report/report.md"
62+
with open(report_file, 'r') as f:
63+
table = render_table(json.load(f))
64+
output_dir = os.path.dirname(output_md)
65+
if output_dir:
66+
os.makedirs(output_dir, exist_ok=True)
67+
with open(output_md, 'w') as f:
68+
f.write(table)
69+
print(f"Table rendered successfully to {output_md}")
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Copyright 2026 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Tests for compile_report.py and render_table.py.
16+
17+
Run with: python3 -m unittest discover -s .agents/skills/security-status-report/scripts
18+
"""
19+
20+
import json
21+
import os
22+
import tempfile
23+
import unittest
24+
25+
from compile_report import compile_report
26+
from render_table import render_table
27+
28+
29+
class CompileReportTest(unittest.TestCase):
30+
def test_copies_summaries(self):
31+
with tempfile.TemporaryDirectory() as d:
32+
threats = os.path.join(d, "threats.json")
33+
with open(threats, "w") as f:
34+
json.dump([{"threat_id": "T-01", "priority": "High", "threat": "x"}], f)
35+
with open(os.path.join(d, "T-01.json"), "w") as f:
36+
json.dump({"threat_id": "T-01", "quality": 0.5, "mitigation_summary": "m", "remaining_gap": "g"}, f)
37+
out = os.path.join(d, "final.json")
38+
compile_report(threats, d, out)
39+
with open(out) as f:
40+
got = json.load(f)[0]
41+
self.assertEqual(got["mitigation_summary"], "m")
42+
self.assertEqual(got["remaining_gap"], "g")
43+
44+
45+
class RenderTableTest(unittest.TestCase):
46+
def test_sorts_by_priority_then_id(self):
47+
data = [
48+
{"threat_id": "T-10", "priority": "Low", "quality": 0.1},
49+
{"threat_id": "T-03", "priority": "Critical", "quality": 0.2},
50+
{"threat_id": "T-02", "priority": "High", "quality": 0.3},
51+
{"threat_id": "T-01", "priority": "Critical", "quality": 0.4},
52+
]
53+
rows = [l.split(" | ")[0].lstrip("| ") for l in render_table(data).splitlines() if l.startswith("| T-")]
54+
self.assertEqual(rows, ["T-01", "T-03", "T-02", "T-10"])
55+
56+
def test_error_row_and_pipe_escaping(self):
57+
data = [
58+
{"threat_id": "T-01", "priority": "High", "error": "Missing result file"},
59+
{"threat_id": "T-02", "priority": "High", "quality": 1.0, "mitigation_summary": "a|b", "remaining_gap": "None"},
60+
]
61+
out = render_table(data)
62+
self.assertIn("| T-01 | High | ERROR | | Missing result file |", out)
63+
self.assertIn("| T-02 | High | 1.00 | a/b | None |", out)
64+
self.assertIn("Mean score: 1.00", out)
65+
66+
67+
if __name__ == "__main__":
68+
unittest.main()

‎.agents/skills/security-status-report/scripts/template_dispatch.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,10 @@ def generate_dispatch_payload(output_path):
3434
3535
- Focus on this threat only.
3636
- Review the entire repo.
37-
- Produce a gut-feel "quality score" based on the current security posture of the repo with respect to that threat.
37+
- Score against the mitigating invariants. The suggested mitigations are options, not a checklist.
38+
- Don't penalize items the threat puts outside Substrate's scope. Do penalize gaps in Substrate's own code, manifests, docs, and tooling.
39+
- Read the Trust Boundaries section of docs/security/threat-model.md. Don't penalize anything the threat's notes accept with a documented reason, such as a permission the sandbox runtime needs. Do penalize anything the notes don't accept.
40+
- Produce a quality score for the current security posture of the repo with respect to that threat, from 0.0 (not mitigated) to 1.0 (fully mitigated).
3841
- Output your results using the following schema, by writing them to .agents/scratch/security-status-report/{t["threat_id"]}.json,
3942
where {t["threat_id"]} matches the id in the threat json you were initially provided.
4043
@@ -45,6 +48,8 @@ def generate_dispatch_payload(output_path):
4548
"quality": <Decimal between 0 (no effective mitigation) and 1 (perfectly mitigated).>,
4649
"strengths": "<Specific positive code/design mechanisms responsible for the score.>",
4750
"weaknesses": "<Specific negative code/design mechanisms responsible for the score.>",
51+
"mitigation_summary": "<Up to 15 words naming the main mechanisms behind the strengths.>",
52+
"remaining_gap": "<Up to 15 words naming the most important remaining gap, or None.>",
4853
"citations": ["<repo-relative/path/to/file1.go>"]
4954
}}
5055
```'''

0 commit comments

Comments
 (0)