Skip to content

Hackathon [Feature]: Add Prometheus data-source connector #5424

Open
topoteretes/cognee-community
#397

Description

@itssaharsh

Update: implemented in topoteretes/cognee-community#397. Its description lists where the code differs from this plan.

Problem Statement

cognee has five data-source connectors (confluence, gmail, google-drive, notion, slack) and open issues for Grafana (#4779), Datadog (#4549), New Relic (#4780) and Honeycomb (#4781). There is nothing for Prometheus. A search of issues and PRs across cognee, cognee-community and cognee-integrations turns up no connector, issue or proposal for it.

A Prometheus server holds a small layer that people write by hand: alerting rules with their summary, description and runbook_url annotations, and recording rules. That is where a team writes down what "broken" means for each service. An agent using cognee can't see any of it today.

This proposal ingests no samples and runs no queries. It is also unrelated to cognee's own OpenTelemetry export, which sends cognee's telemetry out. This reads a team's rules in.

Proposed Solution

Add a prometheus connector under packages/connector/prometheus/ in cognee-community, following the Notion connector's shape and built on dlt's declarative rest_api source.

  • Auth: none by default, since a stock Prometheus has none. Optional basic auth, bearer token and extra headers for servers behind a proxy. The base URL may include a path prefix.
  • Ingest: alerting and recording rules from GET /api/v1/rules, one document per rule: name, group, file, expression, for, labels and annotations. Metric help text from /api/v1/metadata is opt-in and off by default.
  • Incremental sync: Prometheus has no change feed or modified-since for rules, so each run reads the whole rule list in one request. Unchanged rules keep the same content hash and cognee does not re-cognify them. This is the full-snapshot model Hackathon [Feature]: Add incremental sync to the Notion connector #4820 describes for Notion, without its read cost, because the rule list is a single call.
  • Forget-on-delete: write_disposition="replace". A rule removed from the server drops out of the snapshot and orphan cleanup forgets it.

Watch out for:

  • The rules response mixes definitions with runtime state. Two reads of https://prometheus.demo.prometheus.io/api/v1/rules a few seconds apart differ only in evaluationTime and lastEvaluation. state and alerts change when an alert fires, and health and lastError when an evaluation fails. Rows are built from an allowlist of definition fields, so an unchanged server produces identical rows.
  • Rules have no id, and a rule name can repeat inside a group. On that demo server five alert names appear twice in the node-exporter group, once per severity. The row id is a hash of file, group, type, name and labels.
  • Under replace, a failed or partial response would make cognee forget real rules. Any non-2xx status, non-JSON body or status other than success aborts the run.

Alternatives Considered

-Samples or PromQL query results. Out of scope: #4549 already says raw metrics "answer nothing a knowledge graph is good at".

  • Active alerts from /api/v1/alerts. The endpoint returns only what is pending or firing right now, so every sync would rewrite it.
  • Getting rules through the Grafana connector in Hackathon [Feature]: Add Grafana data-source connector #4779. That issue covers dashboards, panel titles and annotations through Grafana's API. Prometheus rules live on the Prometheus server

Use Case

My upstream work is in metrics code: a merged change in OpenTelemetry Go's Prometheus exporter (PR 7866 in open-telemetry/opentelemetry-go) and one in kubernetes-sigs/resource-state-metrics (PR 102), which generates Prometheus metrics for Kubernetes custom resources. The question that comes up around alerts is what they mean: what a given alert checks, what its runbook says, which service it pages for, and which rules use a given metric. The answers are written in rule annotations and expressions, and this connector would put them where an agent can recall them.

Implementation Ideas

packages/connector/prometheus/
├── README.md
├── cognee_community_connector_prometheus/
│   ├── __init__.py
│   └── prometheus.py
├── examples/
├── tests/
└── pyproject.toml
  • One prometheus_source(base_url, ...) factory returning a rest_api source: single_page paginator, data_selector: "data.groups", and a processing step that emits one row per rule with the group's name and file copied in.
  • Rows carry id, title and content, with DOCUMENT_SOURCE_ATTR set, so each rule goes through cognify as a document.
  • Offline tests on recorded responses: two captures of an unchanged server give identical rows, an edited rule changes exactly one row, a removed rule removes its row, and a 503 or truncated body raises.
  • The first version claims Prometheus itself. Mimir, Thanos and VictoriaMetrics serve a compatible rules API with their own differences (path prefix, tenant header, missing parameters). I'd list them as untested.

Additional Context

Acceptance criteria, in the shape of the other connector issues:

  • A user points the connector at a Prometheus base URL, with optional credentials, and selects what to ingest
  • Rules are ingested and searchable in cognee
  • A re-sync re-processes only rules whose definition changed
  • A rule removed from Prometheus is removed from the graph on the next sync
  • README with setup steps and a runnable example under examples/
  • Tests covering the ingest path, determinism and removal

Testing against a real server: a Prometheus I run locally in Docker (prom/prometheus) with a rules file I control, so I can edit a rule, remove a rule and reload between syncs. For a larger real rule set I'd also read the public demo server at prometheus.demo.prometheus.io, read-only.

@NMZivkovic I'd like to build this for Mergetober. If it fits, could you add the INTEGRATIONS and hackathon labels? My assignment request is in the first comment.

Pre-submission Checklist

  • I have searched existing issues to ensure this feature hasn't been requested already
  • I have provided a clear problem statement and proposed solution
  • I have described my specific use case

Activity

  1. itssaharsh commented on Oct 6, 2026

    @itssaharsh
    Author

    Hi @NMZivkovic, I'd like to work on this one for Mergetober.

    How I'd build it:

    • Source: the Prometheus HTTP API through dlt's rest_api source. One request, GET /api/v1/rules, with /api/v1/metadata as an opt-in. No auth by default, optional basic or bearer.
    • Data: one document per alerting or recording rule, built from an allowlist of definition fields (name, group, file, expression, for, labels, annotations) so runtime fields never change the content hash. Full snapshot with replace, so a removed rule is forgotten on the next sync.
    • Testing: a Prometheus I run locally in Docker (prom/prometheus) with a rules file I control, so I can edit a rule, remove a rule and reload between syncs. For a larger real rule set I'd also read the public demo server at prometheus.demo.prometheus.io, read-only.

    Could you assign it to me?

  2. dexters1 commented on Oct 9, 2026

    @dexters1
    Collaborator

    Accepted as part of the hackathon. Thanks!

  3. pooja-bhavani commented on Oct 9, 2026

    @pooja-bhavani

    Hi @NMZivkovic!

    I would like to be assigned to build the Prometheus data-source connector as I am participating in Mergetober (#5424).

    Proposed Technical Architecture

    1. Connector Setup & Location: Build the connector under packages/connector/prometheus/ in cognee-community using dlt's declarative rest_api source.
    2. API Endpoint & Ingestion: Ingest alerting and recording rules via GET /api/v1/rules (with /api/v1/metadata opt-in). Support unauthenticated instances by default, with optional basic auth or bearer tokens.
    3. Data Normalization & Hash Stability: Filter rules using an allowlist of definition fields (name, group, file, expression, for, labels, annotations) and construct deterministic row IDs via hash(file, group, type, name, labels). This prevents runtime evaluation changes (state, lastEvaluation) from modifying content hashes.
    4. Snapshot & Safety: Use a full snapshot model (write_disposition="replace") so deleted rules drop from the graph. Abort synchronization immediately on non-2xx HTTP status or invalid JSON to safeguard existing graph nodes.
    5. Testing Strategy: Add pytest unit coverage over recorded API payloads, and validate real-world sync behavior against a local Docker Prometheus container (prom/prometheus) and [suspicious link removed].

    I will wait for official assignment before opening the Pull Request. Could you please assign this issue to me? Thanks!

  4. PratikPawar1401 commented on Oct 9, 2026

    @PratikPawar1401

    Hi @dexters1 and @NMZivkovic,

    I’d like to work on #5424 — the Prometheus data-source connector

    I reviewed the Building a cognee Integration guide. I’ll implement this as a Path B scheduled-pull integration under packages/connector/prometheus/, following the community connector structure:

    • Build a plain dlt source and pass it through cognee.remember().
    • Set DOCUMENT_SOURCE_ATTR so rules follow cognee’s document ingestion path.
    • Use a dedicated dataset, stable primary keys, write_disposition="replace", and self_improvement=False.
    • Keep the standard resolve_dlt_sources flow so deletion cleanup works correctly.

    The connector will ingest alerting and recording rules from GET /api/v1/rules?exclude_alerts=true. It will support unauthenticated servers, basic or bearer authentication, proxy headers, path-prefixed URLs, and Prometheus rule filters.

    Each rule will become a deterministic document containing its PromQL expression, group, file, duration, labels, and annotations. Runtime fields such as health, state, active alerts, and evaluation timestamps will be excluded.

    The complete response will be validated and buffered before any rows are emitted. API errors, warnings, malformed responses, or incomplete snapshots will abort the sync without deleting existing cognee data. A successfully removed rule will disappear through full-snapshot replacement and orphan cleanup.

    Tests will cover authentication, filtering, deterministic output, duplicate rule names, updates, removals, failure-safe snapshots, searchability, and graph cleanup. I’ll also live-test against a controlled Prometheus Docker instance and publish the required Medium post showing a real alert-rule search and deletion workflow.

    If this approach matches the integration guidelines, could you please assign the issue to me? I’ll wait for assignment before starting.

  5. szed0 commented on Oct 9, 2026

    @szed0

    Hi @NMZivkovic, I'd like to build #5424 for Mergetober if it is still available.

    I'll add prometheus_source under packages/connector/prometheus/ using dlt's declarative REST source. Selected alerting/recording rules become document-mode records containing their expressions, labels and runbook annotations. Authentication and path-prefixed servers are configurable; runtime evaluations and active alerts stay out of content hashes.

    The rules API now supports group pagination, whose results are not consistent during concurrent changes. I'll default to a complete unpaginated read, validate the response envelope and reject unexpected truncation or paging before reconciliation. Paginated support will need an explicit completeness policy. IDs distinguish repeated names using file, group, type and canonical labels.

    Publish complete scoped replace snapshots, retaining unchanged documents, with matching dlt_config in the cognee.remember() example. Failed extraction or downstream ingestion must preserve retryable work.

    Acceptance checks: runtime-only changes cause no reprocessing; an expression edit changes the intended document; removing the final rule clears Data, graph and vectors; malformed/error/partial responses preserve existing memory. I'll validate on a local Prometheus monitoring my Cognee example and write the separate blog from the observed workflow. Those runs are planned, not completed.

    Include offline fixtures, setup/example and signed-off commits. Could you assign #5424 to @szed0? I'll start implementation after assignment.

  6. Atharva-Ramawat commented on Oct 9, 2026

    @Atharva-Ramawat

    Hi @NMZivkovic and @dexters1,

    I see several excellent proposals have already been submitted, including from the original author (@itssaharsh). If they are unable to take this on for Mergetober, I would love to step in as a backup to build the Prometheus connector.

    To ensure we maintain a stable, human-readable graph without generating redundant updates or missing rules during transient network issues, here is my implementation design:

    • Source & Auth: Utilizing dlt's declarative rest_api source against GET /api/v1/rules (with /api/v1/metadata opt-in). I will fully support unauthenticated instances, Basic Auth, Bearer tokens, proxy headers, and custom path prefixes.
    • Deterministic Hashing: The rules API mixes static definitions with runtime state (e.g., evaluationTime, active alerts). I will enforce a strict allowlist of static fields (name, group, file, expression, for, labels, annotations) so transient runtime changes never alter the document content hash.
    • Stable Identifiers: Because rules lack native IDs and names can repeat within groups, I will generate a stable row ID by hashing the file, group, type, name, and labels. This ensures DOCUMENT_SOURCE_ATTR correctly anchors nodes in the graph.
    • Snapshot Safety & Deletions: I will use a full snapshot model with write_disposition="replace" to handle deletions via orphan cleanup. Crucially, the connector will buffer and validate the full API response before emitting any rows. Any non-2xx status, truncated body, or unexpected pagination failure will immediately abort the run, preventing Cognee from accidentally deleting existing rules due to a transient server failure.
    • Testing Strategy: Comprehensive pytest coverage using recorded offline API responses to validate identical hashes across unchanged reads, edit propagation, rule deletion via snapshot replacement, and fail-safe aborts on 503s. I will also include a live Docker setup (prom/prometheus) in example.py for end-to-end validation.

    Please let me know if I can help take this across the finish line!

  7. somuai commented on Oct 10, 2026

    @somuai

    Hi @dexters1 and @NMZivkovic,

    I would like to be considered for #5424 (Prometheus data-source connector) for Mergetober.

    As proof of execution on the cognee-community connector architecture, I have already designed, implemented, and tested two complete dlt-backed Cognee data-source connectors during this hackathon:

    Below is our production-ready architectural specification for packages/connector/prometheus/ that addresses deterministic rule hashing, composite identity resolution, fail-safe full-snapshot replacement, secret scrubbing, and optional operational context enrichment.


    1. End-to-End Architecture & Data Flow

    flowchart LR
        subgraph Prometheus["Prometheus HTTP API v1"]
            R["GET /api/v1/rules"]
            M["GET /api/v1/metadata (Opt-In)"]
            T["GET /api/v1/targets (Opt-In)"]
            A["GET /api/v1/alerts (Opt-In)"]
            Q["GET /api/v1/query & /api/v1/query_range (Opt-In)"]
        end
    
        subgraph Connector["packages/connector/prometheus"]
            Client["PrometheusClient (Exponential Backoff 429/5xx)"]
            Buffer["Atomic Snapshot Validator & Watermark Cursor"]
            Scrubber["Regex Secret & Credential Scrubber"]
            Projector["Deterministic Semantic Markdown Projector"]
            DLT["@dlt.source (DOCUMENT_SOURCE_ATTR = 'prometheus')"]
        end
    
        subgraph Cognee["Cognee Memory Engine"]
            Add["cognee.add(dataset_name=..., write_disposition='replace', data_per_batch=...)"]
            Orphan["resolve_dlt_sources + orphan_cleanup"]
            Cognify["cognee.cognify()"]
            Memify["cognee.memify()"]
        end
    
        R --> Client
        M --> Client
        T --> Client
        A --> Client
        Q --> Client
        Client --> Buffer
        Buffer --> Scrubber
        Scrubber --> Projector
        Projector --> DLT
        DLT --> Add
        Add --> Orphan
        Orphan --> Cognify
        Cognify --> Memify
    
    Loading

    2. Package Structure

    Following the canonical cognee-community/packages/connector/ layout established in our #348 and #351 PRs:

    packages/connector/prometheus/
    |-- README.md
    |-- pyproject.toml
    |-- cognee_community_connector_prometheus/
    |   |-- __init__.py
    |   |-- config.py          # Pydantic BaseModel (PrometheusConfig)
    |   |-- connector.py       # PrometheusClient, dlt source, DOCUMENT_SOURCE_ATTR = "prometheus"
    |   `-- ingest.py          # Pipeline orchestration: cognee.add -> cognify -> memify
    |-- examples/
    |   `-- example.py         # Runnable Docker prom/prometheus + Cognee search & delete demo
    `-- tests/
        `-- test_prometheus.py # 100% offline pytest suite with recorded JSON fixtures
    

    3. Configuration Schema (PrometheusConfig) & Endpoint Coverage

    We define a strict Pydantic BaseModel configuration supporting stock unauthenticated Prometheus instances as well as authenticated/proxied enterprise deployments (Thanos/Mimir/VictoriaMetrics path prefixes and tenant headers):

    from typing import Literal
    from pydantic import BaseModel, Field, SecretStr
    
    class PrometheusConfig(BaseModel):
        base_url: str = Field(..., description="Prometheus base URL (supports custom path prefixes)")
        auth_type: Literal["none", "basic", "bearer"] = "none"
        username: str | None = None
        password: SecretStr | None = None
        bearer_token: SecretStr | None = None
        extra_headers: dict[str, str] = Field(default_factory=dict)
        rule_type: Literal["alert", "record", "all"] = "all"
        file_filter: list[str] | None = None
        group_filter: list[str] | None = None
        include_metadata: bool = False
        include_targets: bool = False
        include_active_alerts: bool = False
        promql_summary_queries: list[str] = Field(default_factory=list)
        last_evaluation_timestamp: float | None = Field(
            default=None,
            description="Optional watermark cursor tracking rule group evaluation progression",
        )
        timeout_seconds: float = 15.0
        max_retries: int = 5
        data_per_batch: int = 50
    Endpoint Default State Extracted Semantic Entities Volatility / Hash Policy
    GET /api/v1/rules [ENABLED] Alerting & recording rules: name, group, file, query (PromQL), duration (for), keep_firing_for, labels, annotations (summary, description, runbook_url) Strict static allowlist; volatile fields (evaluationTime, lastEvaluation, state, health, alerts) stripped from document body so unchanged rules keep identical content hashes
    GET /api/v1/metadata [OPT-IN] Metric type (counter, gauge, histogram, summary), help docstrings, and unit Enriches rules referencing the metric and yields standalone metric glossary documents
    GET /api/v1/targets [OPT-IN] Scrape pool job names, target labels, discovered service topology Scrubs target URL query parameters/credentials; strips volatile lastScrape/lastScrapeDuration
    GET /api/v1/alerts [OPT-IN] Point-in-time firing alert snapshot when explicitly requested for incident triage Isolated into separate resource table when enabled; excluded from static rule hashes
    GET /api/v1/query & /api/v1/query_range [OPT-IN] Pre-configured high-level SLI/SLO summary projections Only runs for user-supplied promql_summary_queries

    4. Core Engineering Safeguards

    1. Deterministic Composite Primary Key (Duplicate Rule Name Disambiguation):
      Prometheus rules do not have server-assigned UUIDs, and identical alert names frequently appear multiple times within the same group differentiated only by severity labels (e.g., NodeFilesystemAlmostOutOfSpace with severity="warning" vs. severity="critical"). We compute a deterministic primary key:
      canonical_labels = json.dumps(sorted((rule.get("labels") or {}).items()), separators=(",", ":"))
      rule_id = hashlib.sha256(
          f"{group_file}|{group_name}|{rule_type}|{rule_name}|{canonical_labels}".encode("utf-8")
      ).hexdigest()[:24]
    2. Hash Stability & Cursor/Watermark State (last_evaluation_timestamp):
      • By default, rule documents are projected strictly from the static definition allowlist (name, type, group, file, query, duration, labels, annotations). Two consecutive reads against a live server with changing evaluationTime, lastEvaluation, or alerts produce byte-for-byte identical Markdown and identical data_id hashes, preventing redundant cognify() LLM calls.
      • We also track last_evaluation_timestamp in dlt.current.resource_state() for auditability and optional incremental diagnostics without polluting the static content hash.
    3. Atomic Full-Snapshot Validation & Forget-on-Delete (write_disposition="replace"):
      • Because write_disposition="replace" instructs Cognee's resolve_dlt_sources and orphan_cleanup to purge any previously ingested document missing from the current batch, emitting a partial or empty stream on a failed request would catastrophically wipe the knowledge graph.
      • PrometheusClient buffers the complete response payload in memory and validates: (a) HTTP 200 status, (b) valid JSON structure, (c) payload.get("status") == "success", and (d) presence of data.groups list (rejecting unexpected group pagination truncation unless all page tokens are resolved). Any error raises before yielding a single row, preserving existing Cognee memory intact.
      • When a rule or scrape target is genuinely deleted upstream, it drops from the validated snapshot and orphan_cleanup removes its Data row, vector embeddings, and graph nodes.
    4. Defensive Regex Secret Scrubbing:
      Before any document is yielded to dlt or cognee.add(), a recursive sanitizer scrubs:
      • Authorization headers (Bearer ..., Basic ...)
      • Embedded URL credentials in scrape targets or runbook_url annotations (https://user:password@host/... -> https://[REDACTED]@host/...)
      • Label/annotation keys matching (?i)(token|secret|password|api_key|credential|auth_header)
    5. Resilient Transport (HTTP 429 & 5xx Backoff):
      PrometheusClient implements exponential backoff (0.5s * 2^(attempt-1)) honoring Retry-After headers across 429, 500, 502, 503, and 504 responses.
    6. Cognee Pipeline Integration (ingest.py):
      • Tags the @dlt.source object with DOCUMENT_SOURCE_ATTR = "prometheus" (imported from cognee.tasks.ingestion.dlt_utils where DOCUMENT_SOURCE_ATTR = "cognee_document_source").
      • Pipelines documents through await cognee.add(source, dataset_name=dataset_name, write_disposition="replace", data_per_batch=config.data_per_batch), followed by await cognee.cognify(datasets=[dataset_name]) and await cognee.memify(dataset=dataset_name).

    5. Offline pytest Verification Plan (tests/test_prometheus.py)

    All unit tests run 100% offline in CI using deterministic httpx.MockTransport fixtures:

    Test Case Verification Target Status Design
    test_static_rule_hash_stability Two snapshots differing only in evaluationTime, lastEvaluation, and active alerts produce identical id and content [READY]
    test_duplicate_rule_names_in_group Five alerts sharing the same name in node-exporter with distinct severity labels receive distinct deterministic SHA-256 IDs [READY]
    test_rule_edit_and_orphan_deletion Modifying a PromQL expression updates only that document; removing a rule under write_disposition="replace" triggers cleanup [READY]
    test_fail_safe_abort_on_503_or_malformed HTTP 503, truncated JSON, or status="error" raises prior to yielding, preventing false orphan deletion [READY]
    test_secret_scrubbing_in_targets_and_annotations Basic auth URLs, bearer tokens, and sensitive label values are replaced with [REDACTED] [READY]
    test_optional_metadata_and_targets_enrichment Opt-in /api/v1/metadata and /api/v1/targets enrich rule documents with metric type/help and target pool topology [READY]
    test_retry_backoff_on_429_and_502 Transient HTTP 429/502 responses retry with exponential backoff and succeed when upstream recovers [READY]

    If this architecture looks good to you, could you please assign #5424 to me? I am ready to open the PR immediately upon assignment.

  8. arindewangan commented on Oct 11, 2026

    @arindewangan

    Hi @NMZivkovic, I'd like to claim this issue for Mergetober and build the Prometheus data-source connector. Here's my approach:

    I'll implement the prometheus connector under packages/connector/prometheus/ in the topoteretes/cognee-community repo, modeled on the notion reference connector. Auth: none required by default, with optional basic/Bearer token configuration for proxied or protected servers.

    For ingest, I'll build a dlt declarative rest_api source on GET /api/v1/rules, covering both alerting and recording rules. Each rule becomes a document with name, group, file, expression, for, labels, and annotations (summary, description, runbook_url), tagged with DOCUMENT_SOURCE_ATTR so rules flow through cognee's document ingestion path. I'll use a full-snapshot model with write_disposition="merge" plus a _deleted hard-delete column, so a rule removed from the server is forgotten on the next sync. Non-2xx or malformed responses abort the run to protect existing data.

    Testing: unit tests covering the ingest path and determinism, plus a live integration test against the public demo server prometheus.demo.prometheus.io, as the issue prescribes. I'll include a README with a runnable example, sign all commits with DCO sign-off (git commit -s), and open the PR against the dev branch with a conventional title.

    Could you please assign this to me for Mergetober? I'll wait for assignment before opening the PR.

  9. somuai commented on Oct 11, 2026

    @somuai

    Hi @dexters1, @NMZivkovic,

    Following up on our detailed architectural proposal submitted on Oct 10 (Comment #6097268704), we wanted to provide a quick status update:

    The complete Prometheus data-source connector has already been fully built, linted, and tested on branch somuai/cognee-community:feat/connector-prometheus (commit 8311b33).

    Verification metrics:

    We are ready to open the upstream pull request immediately upon assignment.

  10. shiv-69-debug commented on Oct 12, 2026

    @shiv-69-debug

    Hi @NMZivkovic and @dexters1, I’d like to be considered for the Prometheus integration.

    My focus would be definition-only ingestion with a reproducible local lifecycle test. I recognize the existing proposals and published implementation branch, and would coordinate the preferred implementation direction after assignment.

    Scope

    I would use the standard community package layout and the requested dlt REST-source approach. Configuration would support the Prometheus base URL, path prefixes, optional authentication and headers, and selected rule types, groups, or files.

    Documents would preserve rule expressions, durations, labels, annotations, and group/file context. Metric metadata would be optional. Metric samples and PromQL query execution would remain outside the requested scope.

    Deterministic content

    I would explicitly map the API’s query and duration fields and exclude runtime evaluation state from indexed content.

    A key test would supply two responses with identical rule definitions but different evaluation timestamps, health, and timing values. They must produce identical document content.

    Safe synchronization

    Each replacement would require a complete, validated rule snapshot. Failed requests, malformed envelopes, or unexpected pagination would not authorize removal. Stable scoped identities would distinguish similarly named rules, with ambiguous duplicates reported rather than silently overwritten.

    Validation and delivery

    After assignment, I would use local Docker Prometheus to demonstrate ingestion, search, a definition edit, reload, deletion, and removal of the final rule. Tests would cover selection, authentication, deterministic rendering, incomplete responses, and recovery.

    I would verify actual Cognee cleanup alongside staging behavior and include setup documentation, a runnable example, package metadata, the required blog, signed-off commits, and DCO affirmation.

    I have not completed these tests yet and will wait for assignment before implementing

  11. SawhneySatvik commented on Oct 12, 2026

    @SawhneySatvik

    Hi @dexters1 and @NMZivkovic, I'd like to take the Prometheus connector for Mergetober.

    I'll follow the issue's design: a prometheus package under packages/connector/prometheus/, built on dlt's rest_api source and shaped like the Notion connector.

    • Read alerting and recording rules from GET /api/v1/rules, one document per rule, with rows built from an allowlist of definition fields so runtime state (health, lastEvaluation, alerts) never changes a row.
    • Row id is a hash of file, group, type, name and labels, since names repeat inside a group.
    • Optional basic auth, bearer token, extra headers and a base URL path prefix. Metadata ingest stays opt-in.
    • write_disposition="replace" for forget-on-delete, and any non-2xx, non-JSON or non-success response aborts the run so a failed fetch can't wipe the graph.
    • Tests on recorded responses (unchanged server gives identical rows, edit changes one row, removal drops one row, 503 and truncated body raise), plus a run against a local Prometheus in Docker.

    I'll open the PR with signed commits and the DCO statement. Could you assign it to me?

  12. Darshan3690 commented on Oct 12, 2026

    @Darshan3690

    Issue: #5424
    Author: @Darshan3690

    Hi @NMZivkovic @Vasilije1990 @dexters1 @siillee @thisisharsh7 sir,

    I'd like to take full ownership of the Prometheus connector for MergeTober. Following my work on the Confluence DLT connector (incremental sync + orphan cleanup), I am very familiar with Cognee's DLT ingestion path and snapshot reconciliation models.

    Below is my technical design proposal so we can align on the implementation approach before I submit the PR:


    Design Proposal: Prometheus Rules DLT Connector

    1. Motivation & Use Case

    Prometheus servers store manually authored operational knowledge: alerting and recording rules, expressions, severity levels, descriptions, and runbook URLs. When alerts fire or incidents escalate, engineers ask:

    • "What does the HighErrorRate alert actually measure, and what is its runbook URL?"
    • "Which service is paged by this rule, and what recording rules feed into it?"

    This connector ingests rule definitions and metadata (not time-series metrics/samples) into Cognee memory so an agent can reason over a team's operational SLOs and alerting logic.


    2. Package Architecture & Layout

    Following the standard cognee-community layout:

    packages/connector/prometheus/
    ├── README.md
    ├── pyproject.toml
    ├── cognee_community_connector_prometheus/
    │   ├── __init__.py
    │   └── prometheus.py
    ├── examples/
    │   └── example.py
    └── tests/
        ├── conftest.py
        └── test_prometheus.py
    

    3. Core Technical Design

    A. Authentication & Transport

    • Auth: Unauthenticated by default (standard Prometheus setup). Supports optional Basic Auth, Bearer Token, and custom HTTP headers for reverse-proxy / gateway deployments.
    • Base URL & Prefix: Transparently accommodates optional reverse-proxy path prefixes (e.g., https://monitor.internal/prometheus).
    • Endpoints:
      • GET /api/v1/rules: Ingests alerting and recording rules.
      • GET /api/v1/metadata: Optional/opt-in ingestion of metric HELP and TYPE text (include_metric_metadata=False by default).

    B. Ingestion & Content Modeling

    • Document Mode: Renders each rule into a human/LLM-readable card with DOCUMENT_SOURCE_ATTR = "prometheus" for seamless Cognee chunking and graph extraction:
      # Alert Rule: HostHighCpuLoad (Severity: warning)
      **Group:** system-alerts | **File:** /etc/prometheus/rules/system.yml
      **Expression:** 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
      **Duration:** 10m
      
      ## Annotations
      - **summary:** High CPU load on host
      - **description:** Host CPU load has exceeded 85% for more than 10 minutes.
      - **runbook_url:** https://wiki.internal/runbooks/high-cpu
      
      ## Labels
      - alertname: HostHighCpuLoad
      - severity: warning
      - service: infrastructure

    C. Deterministic Identity & Runtime State Stripping

    • Runtime State Exclusion: The Prometheus /api/v1/rules response mixes static definitions with volatile runtime fields (evaluationTime, lastEvaluation, state, health, lastError, alerts). The connector strips these dynamic fields, extracting only static definition fields so unchanged rules produce identical content hashes across syncs.
    • Deterministic Primary Key: Rules lack a native UUID and names can repeat across severities within the same group. The id is computed as a deterministic SHA-256 hash of (file, group, type, name, sorted_labels).

    D. Full-Snapshot Sync & Forget-on-Delete

    • Snapshot Model: Rule lists are small and fetched in a single HTTP GET request. The resource is configured with write_disposition="replace".
    • Forget-on-Delete: When a rule is modified or deleted from Prometheus configuration files, Cognee's orphan_cleanup purges the obsolete rule and its associated graph nodes/vectors during the replace sync.
    • Anti-Corruption Guardrail: Under replace, any partial response, non-2xx HTTP status, non-JSON payload, or response where status != "success" immediately raises an exception to abort ingestion and preserve existing graph memory.

    4. Usage Example

    import cognee
    from cognee_community_connector_prometheus import prometheus_source
    
    await cognee.remember(
        prometheus_source(
            base_url="https://prometheus.internal.net",
            # Optional: auth_token="bearer-token",
            # Optional: include_metric_metadata=False,
        ),
        dataset_name="prometheus_rules",
        primary_key="id",
        write_disposition="replace",  # Full snapshot with orphan cleanup
    )
    
    # Search Prometheus alerts and runbooks
    results = await cognee.search(
        query_text="What is the runbook procedure and threshold for HostHighCpuLoad?",
        query_type=cognee.SearchType.GRAPH_COMPLETION,
        datasets=["prometheus_rules"],
    )
    print(results)

    5. Acceptance Criteria & Test Coverage

    • Ingestion of alerting and recording rules from GET /api/v1/rules with optional authentication.
    • Filtering out volatile evaluation states for deterministic content hashing.
    • Stable SHA-256 primary key generation across identical rules with differing labels.
    • Snapshot-based orphan cleanup via replace write disposition.
    • Strict anti-corruption failure abort protecting graph integrity on failed or partial responses.
    • Offline unit test suite with recorded API payloads in tests/test_prometheus.py.
    • Comprehensive documentation and runnable demo in README.md and examples/example.py.
  13. shanaldo7 commented on Oct 13, 2026

    @shanaldo7

    Hi @NMZivkovic and @dexters1,

    I’d like to contribute to #5424 — Add Prometheus data-source connector as part of Mergetober, if the issue is still available for assignment.

    My implementation plan would be to follow the existing Notion connector and Cognee community integration guidelines:

    • Data ingestion: Use dlt’s declarative REST API source to fetch alerting and recording rules from Prometheus’s /api/v1/rules endpoint, creating one document per rule.
    • Stable documents: Include rule definitions, expressions, labels, annotations, and runbook URLs while excluding changing runtime fields so unchanged rules do not generate unnecessary updates.
    • Safe synchronization: Use deterministic rule identifiers and full-snapshot synchronization, ensuring failed or malformed API responses cannot accidentally remove existing data.
    • Configuration: Support unauthenticated Prometheus instances by default, with optional authentication and custom headers where needed.
    • Testing and documentation: Add tests for unchanged, modified, and deleted rules, API failures, and deterministic output, along with a README and runnable example.

    I’ll first review the existing connector patterns and coordinate with the maintainers to avoid duplicating any implementation already in progress. I’ll wait for assignment before starting development and follow the required testing and DCO guidelines.

    Could you please let me know if this issue is still available and assign it to me if appropriate?

    Thank you!

  14. abhinayteja commented on Oct 13, 2026

    @abhinayteja

    Hi @NMZivkovic! I'd like to contribute the Prometheus data-source connector as part of the open-source integration challenge.

    My proposed implementation is to create a connector in cognee-community that fetches Prometheus alerting and recording rules through its HTTP API and converts them into structured documents compatible with Cognee's ingestion pipeline.

    I'll aim to include:

    • A dlt-based source connector following the repository's existing conventions.
    • Authentication and API error handling.
    • Tests for rule parsing, stable document IDs, and repeated synchronization.
    • Installation instructions and a working usage example.

    Before starting implementation, I'll review the existing connector patterns and contribution guidelines to ensure the solution fits the project.

    Could you please confirm whether this issue is available and assign it to me?

    Thank you!

  15. dexters1 commented on Oct 13, 2026

    @dexters1
    Collaborator

    Issue assigned to @itssaharsh

  16. added a commit that references this issue on Oct 15, 2026
    da96678
  17. mahilsaily306-max commented on Oct 16, 2026

    @mahilsaily306-max

    @NMZivkovic I'd like to work on this Prometheus connector integration for Mergetober. I have experience building monitoring integrations and time-series data connectors. My approach would be to:

    1. Study the existing connectors for patterns
    2. Implement the prometheus connector using the Prometheus HTTP API
    3. Handle authentication (API key or basic auth as needed)
    4. Implement incremental sync using timestamps for metrics collection
    5. Ensure proper labeling and linking of metrics data
    6. Create a runnable example and comprehensive tests

    I'll start by reading the notion connector as reference, then implement the connector following the proposed structure. Please assign me to this issue so I can begin working on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions