Skip to content

feat: add datahub-ml-impact skill - #66

Open
cnpierrepapi wants to merge 2 commits into
datahub-project:mainfrom
cnpierrepapi:feat-ml-impact-skill
Open

cnpierrepapi wants to merge 2 commits into
datahub-project:mainfrom
cnpierrepapi:feat-ml-impact-skill

Conversation

@cnpierrepapi

Copy link
Copy Markdown

Adds a sixth catalog interaction skill, for the case where a model is on one end
of a lineage question, and registers it in the using-datahub routing table.

Why a separate skill rather than more of /datahub-lineage

/datahub-lineage is good and general, and the moment a model is involved a
generic walk starts producing confident wrong answers. Four things about ML
lineage are different, and each one fails silently rather than raising:

  1. Tags live on the sibling you did not walk. dbt holds the governance tags,
    the warehouse entity holds none, and traversal returns the warehouse one. Ask
    it whether a column is protected and the answer is no. Nothing errors, and a
    check written that way reports all clear forever.

  2. Filtering a column walk by column name drops the origin. A warehouse
    renames as it cleans. In my case the column was public_coverage_flag in the
    marts and pubcov in the raw extract, and the name filter turned a seven hop
    answer into a five hop one that read as complete.

  3. Column edges stop at the feature table. dbt emits column level edges. ML
    platforms emit a table level edge into the model, because a training run
    consumes a table. A walk that does not cross that boundary reports the column
    reaches no model, which is the exact question that was asked.

  4. The environment in a URN is not a deployment status. PROD is the fabric
    the entity lives in. Read the stage from the registry, or every archived
    experiment counts as live.

All four are written into the skill as explicit traps with the workaround, so an
agent following it gets the answer rather than the plausible version.

Scope

The workflow uses only the DataHub CLI and the MCP server, so it stands alone
and pulls in no external dependency. It declares its own boundary: if nothing
downstream is a model it routes back to /datahub-lineage, which is simpler and
correct.

Changes

  • skills/datahub-ml-impact/SKILL.md, new
  • skills/using-datahub/SKILL.md, routing row, a Lineage vs. ML Impact
    disambiguation rule, count 5 to 6, and the CLI attribution example

prettier and markdownlint-cli2 both pass against the repo config.

Found while building a lineage-grounded governance tool on DataHub Core
(GMS v1.5.0.6, MLflow, dbt, Postgres). Happy to adjust tone or structure to
match the house style more closely.

@cnpierrepapi

Copy link
Copy Markdown
Author

Bumping, though I can see there are over a hundred open PRs here at the moment, so this is a queue problem rather than anyone ignoring it.

One thing that may make it easier to place. This skill answers "what breaks if I change this, and did anything protected reach a deployed model", which the existing five do not cover. The failure modes written into it are not hypothetical: column edges stop at the feature table so a column-only walk reports that nothing reaches a deployed model, tags land on the dbt sibling rather than the warehouse entity so a naive check reports all clear, and the graph index is asynchronous for roughly forty seconds after an ingest so a check that runs immediately reports all clear against an empty graph. Each one returns a confident wrong answer rather than an error, which is the reason they are in the skill body.

Related upstream, in case it is useful context for whoever picks this up: datahub-project/datahub#18999 and #18685 came out of the same work, and I have taken on closing the mlModel and mlFeature incident gap in the main repo.

Happy to rebase or adjust to house style on request.

The routing was one way. using-datahub sends ML questions to the new skill,
but datahub-lineage still claimed "what breaks if I change X?" without a
carve out, so anyone landing on it directly never got redirected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant