Skip to content

Outline command#2685

Merged
HerringtonDarkholme merged 18 commits into
mainfrom
outline-command
Jun 7, 2026
Merged

Outline command#2685
HerringtonDarkholme merged 18 commits into
mainfrom
outline-command

Conversation

@HerringtonDarkholme

@HerringtonDarkholme HerringtonDarkholme commented Jun 6, 2026

Copy link
Copy Markdown
Member

Summary by CodeRabbit

  • New Features

    • Added sg outline command for extracting and filtering code structure from repositories, supporting role-based filtering (definition, import, export), symbol type filtering, and multiple output formats (text, JSON).
  • Documentation

    • Added specification and usage guide for the new outline command.

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds the sg outline command implementation with extractor-catalog-based symbol extraction, view-based filtering and multiple output formats; plus a complete benchmark harness for evaluating outline as a codebase exploration tool using paired Claude API arms (with-outline vs without-outline).

Changes

Outline CLI Feature

Layer / File(s) Summary
CLI contracts and command wiring
crates/cli/src/lib.rs, crates/cli/src/utils/*
Module declaration, imports, Outline(OutlineArg) subcommand variant, dispatch to run_outline, Clone derivation for InputArgs, crate-visible read_file, and CLI tests for valid/invalid outline invocations.
Extractor rules and catalog compilation
crates/cli/src/outline/default_rule.rs, crates/cli/src/outline.rs
DEFAULT_OUTLINE_RULES YAML constant covering nine languages; catalog loading that optionally includes bundled defaults, parses extractors, and compiles per-language matchers with ExportPolicy and NameSource logic.
Symbol extraction and tree normalization
crates/cli/src/outline.rs
Converts ast-grep matches into OutlineItems with resolved names and roles (including forwarded-export detection); deduplicates by range+symbol type, nests by byte-range containment, prunes invalid container/member relationships (with JS/function special-casing), and reparents Go receiver methods under type nodes.
View filtering, digest computation, and output rendering
crates/cli/src/outline.rs
Applies role/type/regex filters with anchor-selection semantics; resolves view (Auto→Digest for direct-file, Names for directories); computes grouped child_digest strings; renders text with grouped labels and indented trees, or JSON (pretty/compact/streaming JSONL) with flattened per-symbol records and optional container metadata.
Outline command entry point and control flow
crates/cli/src/outline.rs
Public run_outline entry point that validates role-filter mixing, loads catalog, chooses stdin vs path-traversal, extracts and sorts outline files.
Outline command tests
crates/cli/src/outline.rs
Multi-language extraction, JS/TS membership nesting, role filtering with auto semantics, anchor/expanded filtering, custom YAML loading, direct-file retention, language support reporting, Go receiver methods, digest expectations, and test helpers.
Outline command design documentation
docs/design/outline-command.md
CLI surface, role/type/match/view filtering, text/JSON output contracts, symbol-type mapping, extractor-catalog strategy, runtime expectations, and open questions on language accessibility.

Outline Benchmark Harness

Layer / File(s) Summary
Scenario dataset and benchmark configuration
benchmarks/outline-agent-scenarios.json, benchmarks/outline_claude_benchmark.py
Seven-codebase scenario definitions with required-claim specifications; benchmark constants for paths, forbidden-option regex, source extensions, and mode-specific system prompts; immutable dataclasses for scenarios, rubric results, agent runs, summaries, and judge verdicts.
Repository management and scenario loading
benchmarks/outline_claude_benchmark.py
Scenario loading from JSON; repository path resolution; optional git sync (clone-if-missing, optional fetch/reset-on-refresh) enabling persistent repo-dir reuse.
Command building and agent execution orchestration
benchmarks/outline_claude_benchmark.py
Arm scheduling per iteration; builds claude CLI commands with tool permissions and arm-specific outline policies; orchestrates agent runs: executes Claude subprocess, writes raw JSONL trace, parses answers/costs/tokens/tool-calls, scores via rubric validation, handles exit codes.
Claude stream parsing and event handling
benchmarks/outline_claude_benchmark.py
Stream-json parsing for init events (with tool/skill validation), final answers, structured outputs, usage aggregation, and tool-call counting; validates init events; extracts JSON from fenced/mixed text.
Answer scoring, statistical aggregation, and comparisons
benchmarks/outline_claude_benchmark.py
Scores answers against required-claim terms; aggregates per-arm medians; computes WITH vs WITHOUT coverage/precision/F1; generates Markdown summary tables with per-codebase metrics; JSON serialization for runs/rubrics/judges/alignments.
Alignment judge execution and verdict parsing
benchmarks/outline_claude_benchmark.py
Judge prompt template; orchestrates separate Claude JSON-mode calls comparing WITH/WITHOUT answers per iteration; parses verdict fields (missing/extra/contradictions) and persists judge outputs.
Validation and artifact writing
benchmarks/outline_claude_benchmark.py
Writes metadata.json, runs.json, alignment.json, optional judge-alignment.json; validates run directories (required files, JSON shapes, record completeness, init events, outline-usage constraints by scanning trace tool events); implements trace parsing for tool commands and permission denials.
CLI orchestration and main entrypoint
benchmarks/outline_claude_benchmark.py
Parses benchmark/judge/validation arguments; filters scenarios; optionally syncs repos; builds task plan with deterministic arm ordering; runs sequentially or concurrently (ThreadPoolExecutor); writes outputs; supports standalone --validate-run mode.
Offline validation tests and fixtures
benchmarks/test_outline_claude_benchmark.py
Dynamically loads benchmark module; tests validate_run_dir() for valid runs, expanded views, forbidden options, skill mismatches, wrong-arm outline usage, empty/duplicate runs, missing alignment, permission denials; provides write_run_dir(), run_record(), write_trace() fixtures for JSON/JSONL artifacts.
Benchmark operations documentation
benchmarks/outline-benchmark.md
Benchmark purpose and measurement goals; scenario definitions and rubric model; paired-arm structure; runner constraints; metrics parsing; repository management; instructions for running scenarios and offline validation; artifact locations; sample completed-run reporting with cost/token/coverage analysis.

🎯 4 (Complex) | ⏱️ ~60 minutes

🐰 A CLI so fine,
With outlines and benchmarks aligned,
From rules to parade,
Claude judges, well made,
Codebase structure revealed, by design

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 8.37% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Outline command' directly and accurately summarizes the main change—introducing a new 'sg outline' subcommand with full CLI implementation, rules, benchmarks, and documentation.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch outline-command

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/cli/src/outline.rs`:
- Around line 1622-1632: The function extract_quoted uses the ? operator inside
the for loop which causes the whole function to return early instead of trying
other quote characters; update extract_quoted to handle the Option results
explicitly (use if let Some(start) = text.find(quote) { ... } else { continue }
and similarly for end on rest) so a missing start or end for one quote char just
continues the loop and only returns Some when a matching pair is found,
otherwise return None after the loop.

In `@docs/design/outline-command.md`:
- Around line 790-798: The example shows `sg outline crates` but the text
describes streaming JSON records; update the example to `sg outline crates
--json=stream` (or equivalent `--format=json --output=stream`) so the command
matches the claimed behavior, ensuring references to `sg outline crates` and the
`--json=stream` flag are used to locate and change the example; alternatively,
if you prefer text output, revise the bullets to describe plain text output
behavior instead of JSON streaming so the contract is unambiguous.

In `@scripts/outline_claude_benchmark.py`:
- Around line 1131-1158: The validator currently derives expected scenarios only
from runs.json and silently overwrites duplicate (scenario, arm, iteration)
entries; update validate_run_dir so it (1) checks that runs is a list and is
non-empty and adds an issue if empty, (2) before assigning into
by_key[(scenario, arm, iteration)] detect duplicates and append an issue (do not
silently overwrite), and (3) build the set of expected scenarios as the union of
scenarios found in runs and any scenario keys present in alignment (so missing
run records are caught); use the existing symbols runs, by_key, scenarios,
alignment and keep calls to validate_run_record/validate_raw_trace unchanged.
- Around line 1248-1252: parse_trace_tool_events currently only appends
tool_result content when it's a plain str, dropping structured contents like
lists of blocks; update the handling of item.get("content") in the branch for
item.get("type") == "tool_result" so that if result is a list/dict you extract
and append the textual parts (e.g., for list of blocks extract each
block["text"] when block.get("type") == "text", or recursively flatten strings),
otherwise convert non-string scalars to string and append; adjust code around
parse_trace_tool_events/tool_results to normalize all tool_result content into
strings so validate_raw_trace sees permission/argument text.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 48e5b8a1-a24d-4a83-936c-8801067b6ce3

📥 Commits

Reviewing files that changed from the base of the PR and between 796ccba and 747f5b5.

📒 Files selected for processing (9)
  • benchmarks/outline-agent-scenarios.json
  • crates/cli/src/lib.rs
  • crates/cli/src/outline.rs
  • crates/cli/src/utils/args.rs
  • crates/cli/src/utils/mod.rs
  • docs/design/outline-benchmark.md
  • docs/design/outline-command.md
  • scripts/outline_claude_benchmark.py
  • scripts/test_outline_claude_benchmark.py

Comment thread crates/cli/src/outline.rs
Comment thread docs/design/outline-command.md Outdated
Comment thread benchmarks/outline_claude_benchmark.py
Comment thread benchmarks/outline_claude_benchmark.py
@codecov

codecov Bot commented Jun 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.02340% with 283 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.21%. Comparing base (7797f68) to head (c90b3cd).
⚠️ Report is 21 commits behind head on main.

Files with missing lines Patch % Lines
crates/cli/src/outline.rs 82.86% 282 Missing ⚠️
crates/cli/src/lib.rs 95.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2685      +/-   ##
==========================================
+ Coverage   85.48%   86.21%   +0.73%     
==========================================
  Files         116      117       +1     
  Lines       20174    21840    +1666     
==========================================
+ Hits        17246    18830    +1584     
- Misses       2928     3010      +82     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
crates/cli/src/outline.rs (2)

448-450: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Propagate extract_outline failures instead of silently skipping files.

A broken --outline-rules file or matcher compilation error currently just disappears here, so path mode can return partial/empty output with exit code 0. Please send the first Err back to run_outline and fail the command instead of Continue-ing.

Based on outline-command design, extractor loading/compilation is part of command setup and should not be silently ignored.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/cli/src/outline.rs` around lines 448 - 450, The code currently
swallows errors from extract_outline and continues; instead propagate the first
Err back to run_outline so the command fails. Replace the let Ok(...) else {
return WalkState::Continue; } with code that does not ignore errors—either make
the surrounding visitor/closure return Result and call extract_outline(...) ? so
errors bubble to run_outline, or on Err(e) record the error and abort the walk
(e.g., return WalkState::Quit) and then return Err(e) from run_outline;
reference extract_outline, run_outline and avoid returning WalkState::Continue
on Err.

763-770: ⚠️ Potential issue | 🟠 Major

Fix brittle Rust impl_item name extraction in outline output (resolve_name)

In crates/cli/src/outline.rs (around lines 763-770), the impl_item case derives the outline name by splitting node.text() after impl. This is incorrect for common Rust forms like:

  • impl<T> Foo<T> { ... } (the current split can return T> Foo instead of Foo)
  • impl Trait for Foo { ... } (the current split returns Trait instead of Foo)

That makes outline symbol names/digests (and any name-dependent matching) unstable. Resolve the implementing type from the impl_item AST structure/fields (e.g., the type/self_ty portion, and for trait impls the type after for) instead of tokenizing raw impl source text.

  if node.kind().as_ref() == "impl_item" {
    let text = node.text();
    let name = text
      .trim_start()
      .strip_prefix("impl")
      .map(str::trim)
      .and_then(|s| s.split([' ', '{', '<']).find(|s| !s.is_empty()))
      .map(str::to_string);
    return name;
  }
♻️ Duplicate comments (1)
crates/cli/src/outline.rs (1)

829-839: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Keep scanning quote styles instead of returning on the first miss.

Using ? inside this loop exits extract_quoted on the first quote character that is absent, so single-quoted or backtick-quoted imports/exports never reach their matching branch.

🐛 Minimal fix
 fn extract_quoted(text: &str) -> Option<String> {
   for quote in ['"', '\'', '`'] {
-    let start = text.find(quote)?;
+    let Some(start) = text.find(quote) else {
+      continue;
+    };
     let rest = &text[start + quote.len_utf8()..];
-    let end = rest.find(quote)?;
+    let Some(end) = rest.find(quote) else {
+      continue;
+    };
     if end > 0 {
       return Some(rest[..end].to_string());
     }
   }
   None
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/cli/src/outline.rs` around lines 829 - 839, The function
extract_quoted currently uses the `?` operator inside the loop which returns
from the whole function when a particular quote char isn't found; update
extract_quoted to try each quote style and only skip that iteration if start or
end is missing instead of returning early. Replace uses of `text.find(quote)?`
and `rest.find(quote)?` with explicit option handling (e.g., if let Some(start)
= text.find(quote) { ... } else { continue }) and similarly continue the loop
when end is None or zero, keeping the existing check that end > 0 before
returning the matched substring.
🧹 Nitpick comments (1)
crates/cli/src/outline.rs (1)

498-499: 🏗️ Heavy lift

Avoid recompiling extractor matchers for every file.

extract_outline rebuilds Vec<RuleSpec> and reruns SerializableRuleCore::get_matcher on every file. On repo-sized inputs that turns matcher compilation into repeated per-file work. Cache compiled rules per SgLang when the catalog is loaded and share them across workers.

Also applies to: 531-533, 545-571

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/cli/src/outline.rs` around lines 498 - 499, extract_outline currently
calls outline_rules(lang, catalog) and runs SerializableRuleCore::get_matcher
for every file, causing repeated matcher compilation; instead compile matchers
once when the catalog is loaded and reuse them per language. Modify the
catalog-loading path to produce and cache a Vec<RuleSpec> (or augment RuleSpec
to include the compiled matcher) keyed by SgLang (e.g., HashMap<SgLang,
Vec<RuleSpec>>), store the cache in a shared, thread-safe container (Arc/RwLock
or similar), and update extract_outline (and the other spots that call
outline_rules/SerializableRuleCore::get_matcher at the referenced sites) to
fetch the precompiled RuleSpec list from the cache rather than
rebuilding/compiling per file. Ensure the cache uses the same unique types:
SgLang, RuleSpec, SerializableRuleCore::get_matcher, and outline_rules.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/cli/src/outline/default_rule.rs`:
- Around line 510-516: The current Go extractor rule named "go-type" in
default_rule.rs always assigns kind: interface (deserializing to
SymbolType::Interface), causing type declarations like "type RouterGroup struct
{}" to be misclassified; split the rule into two distinct rules (e.g.,
"go-type-struct" and "go-type-interface") that match the AST forms for struct vs
interface and emit kind: struct for struct declarations and kind: interface for
interface declarations (so SymbolType::Struct is produced for structs), and
update the Go outline test in crates/cli/src/outline.rs to assert that
RouterGroup has symbol_type == Struct instead of Interface.

---

Outside diff comments:
In `@crates/cli/src/outline.rs`:
- Around line 448-450: The code currently swallows errors from extract_outline
and continues; instead propagate the first Err back to run_outline so the
command fails. Replace the let Ok(...) else { return WalkState::Continue; } with
code that does not ignore errors—either make the surrounding visitor/closure
return Result and call extract_outline(...) ? so errors bubble to run_outline,
or on Err(e) record the error and abort the walk (e.g., return WalkState::Quit)
and then return Err(e) from run_outline; reference extract_outline, run_outline
and avoid returning WalkState::Continue on Err.

---

Duplicate comments:
In `@crates/cli/src/outline.rs`:
- Around line 829-839: The function extract_quoted currently uses the `?`
operator inside the loop which returns from the whole function when a particular
quote char isn't found; update extract_quoted to try each quote style and only
skip that iteration if start or end is missing instead of returning early.
Replace uses of `text.find(quote)?` and `rest.find(quote)?` with explicit option
handling (e.g., if let Some(start) = text.find(quote) { ... } else { continue })
and similarly continue the loop when end is None or zero, keeping the existing
check that end > 0 before returning the matched substring.

---

Nitpick comments:
In `@crates/cli/src/outline.rs`:
- Around line 498-499: extract_outline currently calls outline_rules(lang,
catalog) and runs SerializableRuleCore::get_matcher for every file, causing
repeated matcher compilation; instead compile matchers once when the catalog is
loaded and reuse them per language. Modify the catalog-loading path to produce
and cache a Vec<RuleSpec> (or augment RuleSpec to include the compiled matcher)
keyed by SgLang (e.g., HashMap<SgLang, Vec<RuleSpec>>), store the cache in a
shared, thread-safe container (Arc/RwLock or similar), and update
extract_outline (and the other spots that call
outline_rules/SerializableRuleCore::get_matcher at the referenced sites) to
fetch the precompiled RuleSpec list from the cache rather than
rebuilding/compiling per file. Ensure the cache uses the same unique types:
SgLang, RuleSpec, SerializableRuleCore::get_matcher, and outline_rules.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a995bf07-720f-47e5-bc4a-f9340cb69601

📥 Commits

Reviewing files that changed from the base of the PR and between f38bd59 and 3dad6b8.

📒 Files selected for processing (4)
  • crates/cli/src/outline.rs
  • crates/cli/src/outline/default_rule.rs
  • docs/design/outline-benchmark.md
  • scripts/outline_claude_benchmark.py
✅ Files skipped from review due to trivial changes (1)
  • docs/design/outline-benchmark.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/outline_claude_benchmark.py

Comment thread crates/cli/src/outline/default_rule.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)
benchmarks/outline-benchmark.md (1)

296-312: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Make the “Completed Run” block explicitly time-scoped or example-only.

This section presents a specific run ID/model as current truth, which will go stale quickly and confuse operators. Please add a date stamp and/or label it as a historical example snapshot.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmarks/outline-benchmark.md` around lines 296 - 312, The "The last
completed full run is:" block currently presents a specific run ID and model as
if current; update that block to explicitly mark it as a historical example by
adding a timestamp and/or the phrase "Example run (date: YYYY-MM-DD)" next to
the run ID (target/outline-agent-benchmark/4b9c98bc) and include the resolved
model string (claude-sonnet-4-6) in the same labeled sentence so readers know
it's a snapshot, e.g., annotate the run header and the offline validation
command (python3 benchmarks/outline_claude_benchmark.py --validate-run
target/outline-agent-benchmark/4b9c98bc) with the date or the word "example" to
avoid implying it is the current truth.
benchmarks/outline_claude_benchmark.py (2)

1489-1509: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Fail-fast is ineffective in the concurrent path.

All Claude jobs are submitted up front. If one run fails, returning from inside the executor block still leaves the already-submitted futures running, so the harness keeps spending time and budget after the benchmark is already doomed.

A safer pattern here is incremental scheduling: keep at most args.jobs runs in flight, submit the next task only after a completed one passes report_run, and stop scheduling new work as soon as any run returns a fatal exit code.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmarks/outline_claude_benchmark.py` around lines 1489 - 1509, The current
concurrent scheduling submits all tasks at once so a fatal run detected by
report_run cannot stop already-submitted futures; change to incremental
scheduling: submit only up to args.jobs initial futures from tasks, then in a
loop use as_completed to get each finished future, call future.result() and
report_run(run); if report_run returns a fatal exit_code, stop submitting new
tasks, cancel any not-yet-started futures and return that exit_code; otherwise
append to indexed_runs and submit the next task from tasks (using run_agent with
the same args) to keep at most args.jobs in flight. Update the logic around
future_to_index, indexed_runs, run_agent, and report_run to support popping the
next task and cancelling outstanding futures when a fatal error occurs.

1177-1203: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Guard validate_run_dir against malformed artifact fields.

Line 1191 and Line 1203 still coerce unvalidated JSON with .get()/int(...), so a bad metadata.json or runs.json can raise and make --validate-run crash instead of reporting issues.

Suggested fix
     try:
         metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
         runs = json.loads(runs_path.read_text(encoding="utf-8"))
         alignment = json.loads(alignment_path.read_text(encoding="utf-8"))
     except json.JSONDecodeError as err:
         return [f"invalid JSON: {err}"]

+    if not isinstance(metadata, dict):
+        issues.append("metadata.json is not an object")
+        return issues
     if not isinstance(runs, list):
         issues.append("runs.json is not a list")
         return issues
     if not isinstance(alignment, dict):
         issues.append("alignment.json is not an object")
         return issues

-    repeats = int(metadata.get("options", {}).get("repeats", 0) or 0)
+    options = metadata.get("options")
+    if not isinstance(options, dict):
+        issues.append("metadata.options is not an object")
+        return issues
+    try:
+        repeats = int(options.get("repeats", 0) or 0)
+    except (TypeError, ValueError):
+        issues.append("metadata.options.repeats is not an integer")
+        repeats = 0
     if repeats > 0 and not runs:
         issues.append("runs.json has no run records")
@@
-        iteration = int(raw_run.get("iteration", 0) or 0)
+        try:
+            iteration = int(raw_run.get("iteration", 0) or 0)
+        except (TypeError, ValueError):
+            issues.append(f"{scenario} {arm} run {raw_run.get('iteration')}: invalid iteration")
+            continue
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmarks/outline_claude_benchmark.py` around lines 1177 - 1203, The
validate_run_dir logic must not coerce untrusted JSON directly; instead, check
types and handle malformed fields by appending to issues rather than raising.
For metadata->options->repeats, ensure metadata and its "options" is a dict and
the "repeats" value is an int (or parse safely with try/except) before assigning
repeats; on parse failure append an issue. For each raw_run in runs, verify
raw_run is a dict (already done), then validate that raw_run.get("scenario") and
raw_run.get("arm") are strings and raw_run.get("iteration") is an int-like value
before casting; if any field is missing or wrong type, append a descriptive
issue (e.g., referencing the run or index) and skip that run instead of using
str(...) or int(...) coercion. Ensure any JSON parsing/conversion errors are
caught and added to issues so --validate-run reports them instead of crashing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@benchmarks/outline_claude_benchmark.py`:
- Around line 1489-1509: The current concurrent scheduling submits all tasks at
once so a fatal run detected by report_run cannot stop already-submitted
futures; change to incremental scheduling: submit only up to args.jobs initial
futures from tasks, then in a loop use as_completed to get each finished future,
call future.result() and report_run(run); if report_run returns a fatal
exit_code, stop submitting new tasks, cancel any not-yet-started futures and
return that exit_code; otherwise append to indexed_runs and submit the next task
from tasks (using run_agent with the same args) to keep at most args.jobs in
flight. Update the logic around future_to_index, indexed_runs, run_agent, and
report_run to support popping the next task and cancelling outstanding futures
when a fatal error occurs.
- Around line 1177-1203: The validate_run_dir logic must not coerce untrusted
JSON directly; instead, check types and handle malformed fields by appending to
issues rather than raising. For metadata->options->repeats, ensure metadata and
its "options" is a dict and the "repeats" value is an int (or parse safely with
try/except) before assigning repeats; on parse failure append an issue. For each
raw_run in runs, verify raw_run is a dict (already done), then validate that
raw_run.get("scenario") and raw_run.get("arm") are strings and
raw_run.get("iteration") is an int-like value before casting; if any field is
missing or wrong type, append a descriptive issue (e.g., referencing the run or
index) and skip that run instead of using str(...) or int(...) coercion. Ensure
any JSON parsing/conversion errors are caught and added to issues so
--validate-run reports them instead of crashing.

In `@benchmarks/outline-benchmark.md`:
- Around line 296-312: The "The last completed full run is:" block currently
presents a specific run ID and model as if current; update that block to
explicitly mark it as a historical example by adding a timestamp and/or the
phrase "Example run (date: YYYY-MM-DD)" next to the run ID
(target/outline-agent-benchmark/4b9c98bc) and include the resolved model string
(claude-sonnet-4-6) in the same labeled sentence so readers know it's a
snapshot, e.g., annotate the run header and the offline validation command
(python3 benchmarks/outline_claude_benchmark.py --validate-run
target/outline-agent-benchmark/4b9c98bc) with the date or the word "example" to
avoid implying it is the current truth.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3aeafe22-e647-4180-8020-6aadc69051e6

📥 Commits

Reviewing files that changed from the base of the PR and between 3dad6b8 and c90b3cd.

📒 Files selected for processing (4)
  • benchmarks/outline-benchmark.md
  • benchmarks/outline_claude_benchmark.py
  • benchmarks/test_outline_claude_benchmark.py
  • crates/cli/src/outline.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/cli/src/outline.rs

@HerringtonDarkholme
HerringtonDarkholme merged commit dc968c1 into main Jun 7, 2026
5 of 6 checks passed
@HerringtonDarkholme
HerringtonDarkholme deleted the outline-command branch June 7, 2026 18:45
@phpmac

phpmac commented Jun 17, 2026

Copy link
Copy Markdown

This is a genuinely useful primitive for coding agents. The defaults are well thought out — --items exports --view names for directories gives a cheap overview before an agent decides which file to read, and the precise range metadata lets it open the smallest useful slice instead of dumping whole files into context. That directly targets the main context-window cost in agentic code navigation.

Two questions before I build agent tooling on top of it:

  1. Is there a target release for the first version shipping outline? Looks like it would land in 0.44.0 given the PRs merged since 0.43.0 — just want to plan around it rather than pin to main.

  2. Are the --json=stream entry shape and the default digest/names views considered stable enough to depend on, or still in flux before release? Specifically the symbol.symbolType / range / container fields in the stream entry — I want to avoid hardcoding fields that might shift post-release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants