Outline command#2685
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds the ChangesOutline CLI Feature
Outline Benchmark Harness
🎯 4 (Complex) | ⏱️ ~60 minutes
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/cli/src/outline.rs`:
- Around line 1622-1632: The function extract_quoted uses the ? operator inside
the for loop which causes the whole function to return early instead of trying
other quote characters; update extract_quoted to handle the Option results
explicitly (use if let Some(start) = text.find(quote) { ... } else { continue }
and similarly for end on rest) so a missing start or end for one quote char just
continues the loop and only returns Some when a matching pair is found,
otherwise return None after the loop.
In `@docs/design/outline-command.md`:
- Around line 790-798: The example shows `sg outline crates` but the text
describes streaming JSON records; update the example to `sg outline crates
--json=stream` (or equivalent `--format=json --output=stream`) so the command
matches the claimed behavior, ensuring references to `sg outline crates` and the
`--json=stream` flag are used to locate and change the example; alternatively,
if you prefer text output, revise the bullets to describe plain text output
behavior instead of JSON streaming so the contract is unambiguous.
In `@scripts/outline_claude_benchmark.py`:
- Around line 1131-1158: The validator currently derives expected scenarios only
from runs.json and silently overwrites duplicate (scenario, arm, iteration)
entries; update validate_run_dir so it (1) checks that runs is a list and is
non-empty and adds an issue if empty, (2) before assigning into
by_key[(scenario, arm, iteration)] detect duplicates and append an issue (do not
silently overwrite), and (3) build the set of expected scenarios as the union of
scenarios found in runs and any scenario keys present in alignment (so missing
run records are caught); use the existing symbols runs, by_key, scenarios,
alignment and keep calls to validate_run_record/validate_raw_trace unchanged.
- Around line 1248-1252: parse_trace_tool_events currently only appends
tool_result content when it's a plain str, dropping structured contents like
lists of blocks; update the handling of item.get("content") in the branch for
item.get("type") == "tool_result" so that if result is a list/dict you extract
and append the textual parts (e.g., for list of blocks extract each
block["text"] when block.get("type") == "text", or recursively flatten strings),
otherwise convert non-string scalars to string and append; adjust code around
parse_trace_tool_events/tool_results to normalize all tool_result content into
strings so validate_raw_trace sees permission/argument text.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 48e5b8a1-a24d-4a83-936c-8801067b6ce3
📒 Files selected for processing (9)
benchmarks/outline-agent-scenarios.jsoncrates/cli/src/lib.rscrates/cli/src/outline.rscrates/cli/src/utils/args.rscrates/cli/src/utils/mod.rsdocs/design/outline-benchmark.mddocs/design/outline-command.mdscripts/outline_claude_benchmark.pyscripts/test_outline_claude_benchmark.py
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2685 +/- ##
==========================================
+ Coverage 85.48% 86.21% +0.73%
==========================================
Files 116 117 +1
Lines 20174 21840 +1666
==========================================
+ Hits 17246 18830 +1584
- Misses 2928 3010 +82 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
crates/cli/src/outline.rs (2)
448-450:⚠️ Potential issue | 🟠 Major | ⚡ Quick winPropagate
extract_outlinefailures instead of silently skipping files.A broken
--outline-rulesfile or matcher compilation error currently just disappears here, so path mode can return partial/empty output with exit code 0. Please send the firstErrback torun_outlineand fail the command instead ofContinue-ing.Based on outline-command design, extractor loading/compilation is part of command setup and should not be silently ignored.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/cli/src/outline.rs` around lines 448 - 450, The code currently swallows errors from extract_outline and continues; instead propagate the first Err back to run_outline so the command fails. Replace the let Ok(...) else { return WalkState::Continue; } with code that does not ignore errors—either make the surrounding visitor/closure return Result and call extract_outline(...) ? so errors bubble to run_outline, or on Err(e) record the error and abort the walk (e.g., return WalkState::Quit) and then return Err(e) from run_outline; reference extract_outline, run_outline and avoid returning WalkState::Continue on Err.
763-770:⚠️ Potential issue | 🟠 MajorFix brittle Rust
impl_itemname extraction in outline output (resolve_name)In
crates/cli/src/outline.rs(around lines 763-770), theimpl_itemcase derives the outline name by splittingnode.text()afterimpl. This is incorrect for common Rust forms like:
impl<T> Foo<T> { ... }(the current split can returnT> Fooinstead ofFoo)impl Trait for Foo { ... }(the current split returnsTraitinstead ofFoo)That makes outline symbol names/digests (and any name-dependent matching) unstable. Resolve the implementing type from the
impl_itemAST structure/fields (e.g., thetype/self_typortion, and for trait impls the type afterfor) instead of tokenizing rawimplsource text.if node.kind().as_ref() == "impl_item" { let text = node.text(); let name = text .trim_start() .strip_prefix("impl") .map(str::trim) .and_then(|s| s.split([' ', '{', '<']).find(|s| !s.is_empty())) .map(str::to_string); return name; }
♻️ Duplicate comments (1)
crates/cli/src/outline.rs (1)
829-839:⚠️ Potential issue | 🟠 Major | ⚡ Quick winKeep scanning quote styles instead of returning on the first miss.
Using
?inside this loop exitsextract_quotedon the first quote character that is absent, so single-quoted or backtick-quoted imports/exports never reach their matching branch.🐛 Minimal fix
fn extract_quoted(text: &str) -> Option<String> { for quote in ['"', '\'', '`'] { - let start = text.find(quote)?; + let Some(start) = text.find(quote) else { + continue; + }; let rest = &text[start + quote.len_utf8()..]; - let end = rest.find(quote)?; + let Some(end) = rest.find(quote) else { + continue; + }; if end > 0 { return Some(rest[..end].to_string()); } } None }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/cli/src/outline.rs` around lines 829 - 839, The function extract_quoted currently uses the `?` operator inside the loop which returns from the whole function when a particular quote char isn't found; update extract_quoted to try each quote style and only skip that iteration if start or end is missing instead of returning early. Replace uses of `text.find(quote)?` and `rest.find(quote)?` with explicit option handling (e.g., if let Some(start) = text.find(quote) { ... } else { continue }) and similarly continue the loop when end is None or zero, keeping the existing check that end > 0 before returning the matched substring.
🧹 Nitpick comments (1)
crates/cli/src/outline.rs (1)
498-499: 🏗️ Heavy liftAvoid recompiling extractor matchers for every file.
extract_outlinerebuildsVec<RuleSpec>and rerunsSerializableRuleCore::get_matcheron every file. On repo-sized inputs that turns matcher compilation into repeated per-file work. Cache compiled rules perSgLangwhen the catalog is loaded and share them across workers.Also applies to: 531-533, 545-571
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/cli/src/outline.rs` around lines 498 - 499, extract_outline currently calls outline_rules(lang, catalog) and runs SerializableRuleCore::get_matcher for every file, causing repeated matcher compilation; instead compile matchers once when the catalog is loaded and reuse them per language. Modify the catalog-loading path to produce and cache a Vec<RuleSpec> (or augment RuleSpec to include the compiled matcher) keyed by SgLang (e.g., HashMap<SgLang, Vec<RuleSpec>>), store the cache in a shared, thread-safe container (Arc/RwLock or similar), and update extract_outline (and the other spots that call outline_rules/SerializableRuleCore::get_matcher at the referenced sites) to fetch the precompiled RuleSpec list from the cache rather than rebuilding/compiling per file. Ensure the cache uses the same unique types: SgLang, RuleSpec, SerializableRuleCore::get_matcher, and outline_rules.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/cli/src/outline/default_rule.rs`:
- Around line 510-516: The current Go extractor rule named "go-type" in
default_rule.rs always assigns kind: interface (deserializing to
SymbolType::Interface), causing type declarations like "type RouterGroup struct
{}" to be misclassified; split the rule into two distinct rules (e.g.,
"go-type-struct" and "go-type-interface") that match the AST forms for struct vs
interface and emit kind: struct for struct declarations and kind: interface for
interface declarations (so SymbolType::Struct is produced for structs), and
update the Go outline test in crates/cli/src/outline.rs to assert that
RouterGroup has symbol_type == Struct instead of Interface.
---
Outside diff comments:
In `@crates/cli/src/outline.rs`:
- Around line 448-450: The code currently swallows errors from extract_outline
and continues; instead propagate the first Err back to run_outline so the
command fails. Replace the let Ok(...) else { return WalkState::Continue; } with
code that does not ignore errors—either make the surrounding visitor/closure
return Result and call extract_outline(...) ? so errors bubble to run_outline,
or on Err(e) record the error and abort the walk (e.g., return WalkState::Quit)
and then return Err(e) from run_outline; reference extract_outline, run_outline
and avoid returning WalkState::Continue on Err.
---
Duplicate comments:
In `@crates/cli/src/outline.rs`:
- Around line 829-839: The function extract_quoted currently uses the `?`
operator inside the loop which returns from the whole function when a particular
quote char isn't found; update extract_quoted to try each quote style and only
skip that iteration if start or end is missing instead of returning early.
Replace uses of `text.find(quote)?` and `rest.find(quote)?` with explicit option
handling (e.g., if let Some(start) = text.find(quote) { ... } else { continue })
and similarly continue the loop when end is None or zero, keeping the existing
check that end > 0 before returning the matched substring.
---
Nitpick comments:
In `@crates/cli/src/outline.rs`:
- Around line 498-499: extract_outline currently calls outline_rules(lang,
catalog) and runs SerializableRuleCore::get_matcher for every file, causing
repeated matcher compilation; instead compile matchers once when the catalog is
loaded and reuse them per language. Modify the catalog-loading path to produce
and cache a Vec<RuleSpec> (or augment RuleSpec to include the compiled matcher)
keyed by SgLang (e.g., HashMap<SgLang, Vec<RuleSpec>>), store the cache in a
shared, thread-safe container (Arc/RwLock or similar), and update
extract_outline (and the other spots that call
outline_rules/SerializableRuleCore::get_matcher at the referenced sites) to
fetch the precompiled RuleSpec list from the cache rather than
rebuilding/compiling per file. Ensure the cache uses the same unique types:
SgLang, RuleSpec, SerializableRuleCore::get_matcher, and outline_rules.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a995bf07-720f-47e5-bc4a-f9340cb69601
📒 Files selected for processing (4)
crates/cli/src/outline.rscrates/cli/src/outline/default_rule.rsdocs/design/outline-benchmark.mdscripts/outline_claude_benchmark.py
✅ Files skipped from review due to trivial changes (1)
- docs/design/outline-benchmark.md
🚧 Files skipped from review as they are similar to previous changes (1)
- scripts/outline_claude_benchmark.py
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
benchmarks/outline-benchmark.md (1)
296-312:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winMake the “Completed Run” block explicitly time-scoped or example-only.
This section presents a specific run ID/model as current truth, which will go stale quickly and confuse operators. Please add a date stamp and/or label it as a historical example snapshot.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/outline-benchmark.md` around lines 296 - 312, The "The last completed full run is:" block currently presents a specific run ID and model as if current; update that block to explicitly mark it as a historical example by adding a timestamp and/or the phrase "Example run (date: YYYY-MM-DD)" next to the run ID (target/outline-agent-benchmark/4b9c98bc) and include the resolved model string (claude-sonnet-4-6) in the same labeled sentence so readers know it's a snapshot, e.g., annotate the run header and the offline validation command (python3 benchmarks/outline_claude_benchmark.py --validate-run target/outline-agent-benchmark/4b9c98bc) with the date or the word "example" to avoid implying it is the current truth.benchmarks/outline_claude_benchmark.py (2)
1489-1509:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy liftFail-fast is ineffective in the concurrent path.
All Claude jobs are submitted up front. If one run fails, returning from inside the executor block still leaves the already-submitted futures running, so the harness keeps spending time and budget after the benchmark is already doomed.
A safer pattern here is incremental scheduling: keep at most
args.jobsruns in flight, submit the next task only after a completed one passesreport_run, and stop scheduling new work as soon as any run returns a fatal exit code.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/outline_claude_benchmark.py` around lines 1489 - 1509, The current concurrent scheduling submits all tasks at once so a fatal run detected by report_run cannot stop already-submitted futures; change to incremental scheduling: submit only up to args.jobs initial futures from tasks, then in a loop use as_completed to get each finished future, call future.result() and report_run(run); if report_run returns a fatal exit_code, stop submitting new tasks, cancel any not-yet-started futures and return that exit_code; otherwise append to indexed_runs and submit the next task from tasks (using run_agent with the same args) to keep at most args.jobs in flight. Update the logic around future_to_index, indexed_runs, run_agent, and report_run to support popping the next task and cancelling outstanding futures when a fatal error occurs.
1177-1203:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winGuard
validate_run_diragainst malformed artifact fields.Line 1191 and Line 1203 still coerce unvalidated JSON with
.get()/int(...), so a badmetadata.jsonorruns.jsoncan raise and make--validate-runcrash instead of reporting issues.Suggested fix
try: metadata = json.loads(metadata_path.read_text(encoding="utf-8")) runs = json.loads(runs_path.read_text(encoding="utf-8")) alignment = json.loads(alignment_path.read_text(encoding="utf-8")) except json.JSONDecodeError as err: return [f"invalid JSON: {err}"] + if not isinstance(metadata, dict): + issues.append("metadata.json is not an object") + return issues if not isinstance(runs, list): issues.append("runs.json is not a list") return issues if not isinstance(alignment, dict): issues.append("alignment.json is not an object") return issues - repeats = int(metadata.get("options", {}).get("repeats", 0) or 0) + options = metadata.get("options") + if not isinstance(options, dict): + issues.append("metadata.options is not an object") + return issues + try: + repeats = int(options.get("repeats", 0) or 0) + except (TypeError, ValueError): + issues.append("metadata.options.repeats is not an integer") + repeats = 0 if repeats > 0 and not runs: issues.append("runs.json has no run records") @@ - iteration = int(raw_run.get("iteration", 0) or 0) + try: + iteration = int(raw_run.get("iteration", 0) or 0) + except (TypeError, ValueError): + issues.append(f"{scenario} {arm} run {raw_run.get('iteration')}: invalid iteration") + continue🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/outline_claude_benchmark.py` around lines 1177 - 1203, The validate_run_dir logic must not coerce untrusted JSON directly; instead, check types and handle malformed fields by appending to issues rather than raising. For metadata->options->repeats, ensure metadata and its "options" is a dict and the "repeats" value is an int (or parse safely with try/except) before assigning repeats; on parse failure append an issue. For each raw_run in runs, verify raw_run is a dict (already done), then validate that raw_run.get("scenario") and raw_run.get("arm") are strings and raw_run.get("iteration") is an int-like value before casting; if any field is missing or wrong type, append a descriptive issue (e.g., referencing the run or index) and skip that run instead of using str(...) or int(...) coercion. Ensure any JSON parsing/conversion errors are caught and added to issues so --validate-run reports them instead of crashing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@benchmarks/outline_claude_benchmark.py`:
- Around line 1489-1509: The current concurrent scheduling submits all tasks at
once so a fatal run detected by report_run cannot stop already-submitted
futures; change to incremental scheduling: submit only up to args.jobs initial
futures from tasks, then in a loop use as_completed to get each finished future,
call future.result() and report_run(run); if report_run returns a fatal
exit_code, stop submitting new tasks, cancel any not-yet-started futures and
return that exit_code; otherwise append to indexed_runs and submit the next task
from tasks (using run_agent with the same args) to keep at most args.jobs in
flight. Update the logic around future_to_index, indexed_runs, run_agent, and
report_run to support popping the next task and cancelling outstanding futures
when a fatal error occurs.
- Around line 1177-1203: The validate_run_dir logic must not coerce untrusted
JSON directly; instead, check types and handle malformed fields by appending to
issues rather than raising. For metadata->options->repeats, ensure metadata and
its "options" is a dict and the "repeats" value is an int (or parse safely with
try/except) before assigning repeats; on parse failure append an issue. For each
raw_run in runs, verify raw_run is a dict (already done), then validate that
raw_run.get("scenario") and raw_run.get("arm") are strings and
raw_run.get("iteration") is an int-like value before casting; if any field is
missing or wrong type, append a descriptive issue (e.g., referencing the run or
index) and skip that run instead of using str(...) or int(...) coercion. Ensure
any JSON parsing/conversion errors are caught and added to issues so
--validate-run reports them instead of crashing.
In `@benchmarks/outline-benchmark.md`:
- Around line 296-312: The "The last completed full run is:" block currently
presents a specific run ID and model as if current; update that block to
explicitly mark it as a historical example by adding a timestamp and/or the
phrase "Example run (date: YYYY-MM-DD)" next to the run ID
(target/outline-agent-benchmark/4b9c98bc) and include the resolved model string
(claude-sonnet-4-6) in the same labeled sentence so readers know it's a
snapshot, e.g., annotate the run header and the offline validation command
(python3 benchmarks/outline_claude_benchmark.py --validate-run
target/outline-agent-benchmark/4b9c98bc) with the date or the word "example" to
avoid implying it is the current truth.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3aeafe22-e647-4180-8020-6aadc69051e6
📒 Files selected for processing (4)
benchmarks/outline-benchmark.mdbenchmarks/outline_claude_benchmark.pybenchmarks/test_outline_claude_benchmark.pycrates/cli/src/outline.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/cli/src/outline.rs
|
This is a genuinely useful primitive for coding agents. The defaults are well thought out — Two questions before I build agent tooling on top of it:
|
Summary by CodeRabbit
New Features
sg outlinecommand for extracting and filtering code structure from repositories, supporting role-based filtering (definition, import, export), symbol type filtering, and multiple output formats (text, JSON).Documentation