Skip to content

feat(blueprints): add continuous GenAI evaluation and hallucination regression gate with DeepEval - #1299

Merged
MartinRst merged 2 commits into
kestra-io:mainfrom
somuai:feat/deepeval-llm-continuous-evaluation-gate
Oct 14, 2026
Merged

MartinRst merged 2 commits into
kestra-io:mainfrom
somuai:feat/deepeval-llm-continuous-evaluation-gate

Conversation

@somuai

@somuai somuai commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds flows/deepeval-llm-continuous-evaluation-and-regression-gate.yaml for Kestra Hacktober Track 02 (Blueprints).

What it does

An enterprise-grade continuous evaluation, hallucination detection, and deployment regression gate for production LLM and RAG pipelines using DeepEval:

  1. Trigger on CI/CD or Schedule: Executes on demand via authenticated Webhook (io.kestra.plugin.core.trigger.Webhook) when a new prompt or model version is tagged in Git, or on a recurring nightly Schedule (io.kestra.plugin.core.trigger.Schedule).
  2. Benchmark Test Execution: Runs an automated DeepEval evaluation suite inside an isolated Docker container (python:3.11-slim) against golden reference datasets.
  3. Multi-Metric Scoring: Evaluates critical production metrics:
    • Faithfulness: Verifies that model responses are strictly grounded in retrieved context without hallucinating ungrounded facts.
    • Answer Relevancy: Verifies that responses directly address user queries without evasive filler.
    • Hallucination Rate: Identifies ungrounded claims across test batches.
  4. Automated Quality Gate (If): Evaluates aggregate scores against user-defined Service Level Objectives (SLOs).
    • If targets are met, logs qualification and dispatches a scorecard to Slack.
    • If regressions are detected, quarantines the release and optionally fails the execution (io.kestra.plugin.core.execution.Fail) to halt downstream CI/CD deployment pipelines.
  5. Team Telemetry: Delivers structured Block Kit scorecards directly to Slack (io.kestra.plugin.notifications.slack.SlackIncomingWebhook).

Hacktoberfest Track 02 Requirements Checklist

  • Minimum 3 runnable tasks: 3 runnable tasks (setup_eval_context, execute_deepeval_suite, and nested alert/fail tasks).
  • At least 1 flow task: io.kestra.plugin.core.flow.If (quality gate routing).
  • At least 1 trigger: Webhook trigger (io.kestra.plugin.core.trigger.Webhook) + Schedule trigger (io.kestra.plugin.core.trigger.Schedule).
  • Non-core plugins used: Python Docker Runner (io.kestra.plugin.scripts.python.Commands, io.kestra.plugin.scripts.runner.docker.Docker), Slack Notifications (io.kestra.plugin.notifications.slack.SlackIncomingWebhook).
  • Zero hardcoded credentials: All sensitive tokens utilize {{ secret('...') }} (DEEPEVAL_WEBHOOK_KEY, SLACK_WEBHOOK_URL).
  • Tags & Metadata: Adheres to allowed tags (AI, Infrastructure), complete extend block with SEO metadata (metaDescription <= 160 chars), rich markdown documentation, and inputs table.
  • DCO Signed-off.

#KestraHacktober

…egression gate with DeepEval

Signed-off-by: Soumyajit Ghosh <[email protected]>
@github-project-automation github-project-automation Bot moved this to To review in Pull Requests Oct 13, 2026
@MartinRst
MartinRst merged commit 2056cfb into kestra-io:main Oct 14, 2026
@github-project-automation github-project-automation Bot moved this from To review to Done in Pull Requests Oct 14, 2026
@MilosPaunovic MilosPaunovic added the kind/external Pull requests raised by community contributors label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/external Pull requests raised by community contributors

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants