Skip to content

feat(blueprints): add autonomous incident triage with multi-provider LLM fallback and Slack escalation - #342

Merged
MartinRst merged 2 commits into
kestra-io:mainfrom
somuai:feat-autonomous-incident-triage-blueprint
Oct 4, 2026
Merged

MartinRst merged 2 commits into
kestra-io:mainfrom
somuai:feat-autonomous-incident-triage-blueprint

Conversation

@somuai

@somuai somuai commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

What

Adds an enterprise-grade production workflow blueprint: Autonomous Incident Triage with Multi-Provider LLM Fallback and Slack Escalation (flows/autonomous-incident-triage-with-fallback.yaml).

Blueprint

  • File: flows/autonomous-incident-triage-with-fallback.yaml
  • ID: autonomous-incident-triage-with-fallback
  • Namespace: company.team
  • Tags: AI, Infrastructure, Business

What it does

  1. Inbound Incident Event Handling: Listens via Webhook and scheduled cron triggers for alert payloads containing error stack traces, service names, and error counts.
  2. Three-Tier LLM Failover Architecture:
    • Tier 1 (Primary): OpenAI gpt-4o-mini for high-accuracy root-cause analysis and severity scoring.
    • Tier 2 (Fallback): Google Gemini gemini-1.5-flash with zero-latency failover when OpenAI rate-limits or errors.
    • Tier 3 (Emergency Heuristic Engine): Rule-based Python regex AST analyzer guaranteeing zero dropped incidents even under complete cloud provider outages.
  3. Automated Blast Radius Evaluation: Computes impacted downstream services and SLO degradation.
  4. Human-in-the-Loop Approval Gate: Uses Kestra Pause task (PT1H timeout) for critical/P0 severity incidents requiring human sign-off before automated mitigation runs.
  5. Slack Ops Escalation & Audit Trail: Posts structured triage cards to #production-incidents and logs end-to-end execution metadata.

Prerequisites

  • OpenAI API Key (OPENAI_API_KEY) and/or Google Gemini API Key (GEMINI_API_KEY).
  • Slack Incoming Webhook URL (SLACK_WEBHOOK_URL).

Validation

  • PyYAML syntax validated.
  • Blueprint tag workflow (.github/workflows/validate-blueprint-tags.yml) verified: valid tags, metaDescription length <= 160 characters.
  • 100% emoji-free output.

@MartinRst MartinRst left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@MartinRst
MartinRst merged commit 073aa70 into kestra-io:main Oct 4, 2026
@MilosPaunovic MilosPaunovic added the kind/external Pull requests raised by community contributors label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/external Pull requests raised by community contributors

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants