Skip to content

feat(blueprints): add continuous GenAI evaluation and prompt guardrail pipeline with Comet Opik - #437

Merged
MartinRst merged 2 commits into
kestra-io:mainfrom
somuai:feat-comet-opik-evaluation-pipeline
Oct 5, 2026
Merged

feat(blueprints): add continuous GenAI evaluation and prompt guardrail pipeline with Comet Opik#437
MartinRst merged 2 commits into
kestra-io:mainfrom
somuai:feat-comet-opik-evaluation-pipeline

Conversation

@somuai

@somuai somuai commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

What

Adds an enterprise-grade production workflow blueprint: Continuous GenAI Evaluation and Prompt Regression Guardrails with Comet Opik (flows/comet-opik-llm-evaluation-guardrail.yaml).

This blueprint provides Kestra users with an automated quality gate for Generative AI applications and LLM prompt pipelines, continuously evaluating candidate prompts against benchmark datasets, measuring multi-dimensional metrics (answer relevance, factual hallucination prevention, and RFC 8259 JSON schema compliance), logging full experiment traces to Comet Opik, and enforcing hard quality thresholds with Slack alerting and human-in-the-loop review.

Blueprint Details

  • File: flows/comet-opik-llm-evaluation-guardrail.yaml
  • ID: comet-opik-llm-evaluation-guardrail
  • Namespace: company.team
  • Tags: AI, Data (strictly validated against validate-blueprint-tags.yml)
  • SubTags: LLM Evaluation & Guardrails

Problem Solved

Prompt engineering, model checkpoint updates, and retrieval-augmented generation (RAG) parameter tuning frequently introduce silent regressions. These regressions manifest as degraded answer relevance, increased hallucination rates, broken structured output formatting (such as invalid JSON), and subtle drift in brand alignment. Traditional unit tests cannot detect these non-deterministic failure modes.

Without continuous evaluation guardrails embedded directly into CI/CD pipelines and deployment lifecycles, flawed prompt changes can slip into production, degrading user trust and violating service level agreements.

Architecture and Workflow

+---------------------------------------------------------------------------------+
|                                INGESTION TRIGGERS                               |
|  +---------------------------------------+  +--------------------------------+  |
|  | CI/CD Webhook Trigger                 |  | Nightly Schedule Sweep         |  |
|  | (io.kestra.plugin.core.trigger.       |  | (io.kestra.plugin.core.        |  |
|  |  Webhook)                             |  |  trigger.Schedule)             |  |
|  +---------------------------------------+  +--------------------------------+  |
+-----------------------------------------+---------------------------------------+
                                          |
                                          v
+---------------------------------------------------------------------------------+
|                     TASK 1: SETUP EVALUATION CONTEXT                            |
|  - Logs target model, prompt version, dataset, and threshold boundaries         |
+-----------------------------------------+---------------------------------------+
                                          |
                                          v
+---------------------------------------------------------------------------------+
|                     TASK 2: RUN COMET OPIK EVALUATION                           |
|  (io.kestra.plugin.scripts.python.Commands in python:3.11-slim)                 |
|  - Installs opik and pydantic in containerized environment                      |
|  - Connects to Comet Opik with API key and workspace credentials                |
|  - Pulls or provisions evaluation benchmark dataset items                       |
|  - Computes multi-dimensional evaluation metrics:                               |
|      * Answer Relevance (semantic alignment between prompt query and answer)    |
|      * Hallucination Freedom (factual grounding against context)                |
|      * JSON Validity (RFC 8259 syntax compliance)                               |
|  - Logs experiment traces, span metadata, and feedback scores to Comet Opik     |
|  - Emits Kestra outputs via ion/output format                                   |
+-----------------------------------------+---------------------------------------+
                                          |
                                          v
+---------------------------------------------------------------------------------+
|                     TASK 3: CHECK EVALUATION GUARDRAIL                          |
|  (io.kestra.plugin.core.flow.If: outputs.run_opik_evaluation.vars.              |
|                                 passed_evaluation == true)                      |
|                                                                                 |
|            [PASS BRANCH]                               [FAIL BRANCH]            |
|                  |                                           |                  |
|                  v                                           v                  |
|     +-------------------------+                 +-------------------------+     |
|     | log_evaluation_passed   |                 | log_evaluation_failed   |     |
|     +------------+------------+                 +------------+------------+     |
|                  |                                           |                  |
|                  v                                           v                  |
|     +-------------------------+                 +-------------------------+     |
|     | notify_slack_passed     |                 | notify_slack_alert      |     |
|     +-------------------------+                 +------------+------------+     |
|                                                              |                  |
|                                                              v                  |
|                                                 +-------------------------+     |
|                                                 | require_human_approval  |     |
|                                                 | (Kestra Pause: PT24H)   |     |
|                                                 +-------------------------+     |
+---------------------------------------------------------------------------------+

Hacktoberfest 2026 Track 02 Compliance Checklist

  • Runnable Tasks (>= 3):
    1. setup_evaluation_context (io.kestra.plugin.core.log.Log)
    2. run_opik_evaluation (io.kestra.plugin.scripts.python.Commands)
    3. notify_slack_passed (io.kestra.plugin.notifications.slack.SlackIncomingWebhook)
    4. log_evaluation_passed (io.kestra.plugin.core.log.Log)
    5. log_evaluation_failed (io.kestra.plugin.core.log.Log)
    6. notify_slack_alert (io.kestra.plugin.notifications.slack.SlackIncomingWebhook)
  • Flowable Tasks (>= 1):
    1. check_evaluation_guardrail (io.kestra.plugin.core.flow.If)
    2. require_human_approval (io.kestra.plugin.core.flow.Pause)
  • Triggers (>= 1):
    1. ci_cd_webhook_trigger (io.kestra.plugin.core.trigger.Webhook)
    2. nightly_regression_sweep (io.kestra.plugin.core.trigger.Schedule)
  • Non-core Plugins (>= 1):
    1. plugin-scripts-python (io.kestra.plugin.scripts.python.Commands)
    2. plugin-notifications (io.kestra.plugin.notifications.slack.SlackIncomingWebhook)
  • Extend Metadata and Tags:
    • extend.tags: ['AI', 'Data'] (Valid per .github/workflows/validate-blueprint-tags.yml, <= 3 allowed tags).
    • Comprehensive documentation included in extend.description with ASCII diagram, Mermaid chart, inputs table, runbook, extension ideas, and plugin links.
  • Syntax and Formatting:
    • 100% valid YAML verified with PyYAML.
    • 0 tab characters.
    • 100% pure ASCII output (strictly emoji-free).

Verification

The blueprint was verified locally:

  1. validate-blueprint-tags.yml script pass verified with Python YAML parser.
  2. The Python evaluation script was executed with synthetic RAG benchmarks, validating proper calculation of answer_relevance, hallucination_score, and is_json_valid, trace emission to Opik, and proper Kestra ion output formatting ::{"outputs": ...}::.
  3. Flow conditional evaluation and pause behavior verified.

@kestrabot kestrabot Bot added this to Pull Requests Oct 4, 2026
@github-project-automation github-project-automation Bot moved this to To review in Pull Requests Oct 4, 2026
@MilosPaunovic MilosPaunovic added the kind/external Pull requests raised by community contributors label Oct 5, 2026
@MartinRst
MartinRst merged commit b39cf44 into kestra-io:main Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MartinRst MartinRst MartinRst approved these changes

Assignees

No one assigned

Labels

kind/external Pull requests raised by community contributors

Projects

Pull Requests
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@somuai @MartinRst @MilosPaunovic