Greptile Summary

This PR fixes two independent bugs in the RDS synthetic test pipeline: (1) _seed_plan_actions now filters out any LLM-suggested actions that are not in available_action_names, and a controller-level fallback forces the first valid available action when the planner returns an empty list after filtering; (2) score_result in run_suite.py now gates the failover event-sequence check on whether the scenario's own required_keywords contain all five failover timeline tokens, rather than keying off aws_rds_events presence — preventing scenario 008 (storage-full) from being misgraded. Both fixes are covered by new unit tests that would have caught the original regressions.

Confidence Score: 5/5

Safe to merge; both fixes are correct, well-tested, and limited in scope.

All remaining findings are P2 style issues (non-idiomatic set constructor, local variable naming convention). No logic errors, data integrity issues, or correctness problems were found. Both changed code paths are covered by targeted regression tests.

No files require special attention.

Important Files Changed

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[plan_actions called] --> B[select_actions\nfilters executed actions]
    B --> C[plan_actions_with_llm\nLLM returns planned_actions]
    C --> D[_seed_plan_actions\nfilter: keep only available_action_names]
    D --> E{plan.actions empty?}
    E -- No --> F[Enforce tool_budget cap]
    E -- Yes --> G["Fallback: plan.actions = [available_action_names[0]]"]
    G --> F
    F --> H[Return plan]

    subgraph score_result ["score_result (run_suite.py)"]
        I[Compute normalized_required_keywords\nfrom fixture answer_key] --> J{failover_required_tokens\n⊆ normalized_required_keywords?}
        J -- No --> K[Skip failover event\nsequence check]
        J -- Yes --> L[Check RDS event reasoning\nin root_cause + validated_claims]
        L --> M{mentions_event_reasoning?}
        M -- No --> N[FAIL: RDS events not used\nas primary reasoning signal]
        M -- Yes --> O{sequence_present?}
        O -- No --> P[FAIL: sequence not\nexplicitly listed]
        O -- Yes --> Q[PASS]
    end

_{Reviews (1): Last reviewed commit: "fix(synthetic-rds): stop misgrading and ..." | Re-trigger Greptile}

Addressing the two Greptile nits in a follow-up commit:

• switched the typed set construction to the annotated form
• renamed the local failover sequence tuple to lowercase to match local-variable naming conventions

Re-ran focused regression tests and ruff checks after the change.

hey @yas789
Tested locally with --mock-grafana. Scorer fix works, scenario 008 passes. ✅

But the planner-loop fix doesn't fully cover this case. The investigation hit max_loops: 4 because the planner kept selecting run_diagnostic_code four times in a row, even though it failed every time with the same TypeError:

[WARNING] Action failed: run_diagnostic_code:FAILED(TypeError: run_diagnostic_code() missing 1 require)

Thanks for flagging this. I reproduced the issue locally with a focused regression test: after a deterministic run_diagnostic_code failure with a TypeError, the next planning pass still offered run_diagnostic_code again because failed actions were not recorded in the planner’s exclusion history.

I’m addressing this by separating successful and failed action history:

• actions remains successful actions only, so synthetic trajectory scoring and report summaries are not polluted by failed attempts.
• failed_actions records failure metadata for audit/debugging.
• exhausted_actions records failed actions that should be excluded from future planning.

The planner now blocks successful actions plus exhausted actions. Non-retryable failures like TypeError, missing required arguments, unknown actions, and invalid responses are exhausted immediately. Retryable failures like timeouts, throttling, connection errors, and 5xx errors remain retryable until they hit a deterministic retry threshold.

I also added tests for:

• reproducing the repeated failed-action selection
• immediate exhaustion for non-retryable failures
• retryable failures remaining selectable before the threshold
• retryable failures becoming exhausted at the threshold
• reroute logic not repeatedly triggering on exhausted actions

Local verification is green: focused planner tests, lint, format check, typecheck, make test-cov, and the original synthetic 008-storage-full-missing-metric scenario. I’ll push the implementation after one final review of the diff.

Re-tested with be04428. The exhausted-action tracking solves the loop:

• loops_used: 3 (down from 4/4)
• Zero run_diagnostic_code calls in the trace
• 25/25 unit tests pass, including the 4 new failure-classification tests
• make lint / format-check / typecheck all clean

Thanks @yas789 for the deeper architectural fix. 🙌

🎊 Achievement unlocked: PR Merged. @yas789 passed code review, survived CI, and shipped. Respect. 🤝

👋 Join us on Discord - OpenSRE : hang out, contribute, or hunt for features and issues. Everyone's welcome.