Repository navigation
feat(plugin-catalog): add skill-lineage - #142381
Conversation
alt-glitch
added
comp/plugins
…ion to the code sha 96ffb13 -> cbb85a9 (exact link types, related work, CLI no longer creates or resets the hashing key). description: name the anchor tokens the code actually uses (a URL or host, a shell command with its first argument, a package install, a file path, an environment variable, an IP or e-mail address) instead of identifiers and quoted strings; state the measured counts on Hermes 3166d61 (3.14: 56 passed; 3.11: 48 passed, 1 skipped) instead of the older 46; acknowledge prior work on memory lineage. Drop provides_commands, which the catalog schema does not define.
… result sha cbb85a9 -> 37132da (labelled evaluation, boilerplate filtering, sharper anchor extraction). Measured counts on Hermes 3166d61 are now 3.14: 59 passed; 3.11: 51 passed, 1 skipped. Adds the held-out evaluation: precision 0.988 for the default link set, recall 1.00 on verbatim, paraphrase-with-operative-token and two-hop-via-curator cases, against 0.385 when every link is recorded. States the limits in the same breath: the paraphrases keep their operative token by construction, and a rewrite by a real model is not yet measured.
…l result The pin moves from 37132da to 8df4bcb. The description no longer says a rewrite by a real model is unmeasured: two batches of 60 runs each are reported, with pre-registered false links falling from 53/60 to 6/60 and clean ones from 10/60 to 2/60. Test counts follow the new build.
The entry points at the README for the evaluation instead of restating precision and p-values; a catalog description should say what the plugin reads and writes.
teknium1
left a comment
teknium1
left a comment
There was a problem hiding this comment.
Thanks @simin-yuan! We reviewed skill-lineage at 554a951a. It watches skill writes and keeps a tamper-evident, hash-only record of which session sources (user text, tool results, skills read) each new skill version drew on. The code is clean: no network, no credentials, hashes only, and it never blocks a tool call. One availability bug holds it.
Before it can list:
- The e-mail regex is quadratic on hostile input (textfeat.py:20).
anchor_strings(textfeat.py:80) runs it on every tool result insidepost_tool_call(hooks.py:150-152, :172), and results are capped at 2 MB. We timed the pattern alone: 40 KB of"a."takes about 2 s, 200 KB about 51 s, and a 2 MB page would run for over an hour. Python'sreholds the GIL in one call, so Hermes' hook timeout can't interrupt it, and a prompt-injected web page or file can freeze the agent's turn. A bounded pattern such as(?<![\w.+-])[\w.+-]{1,64}@[\w-]+(?:\.[\w-]+)+\bruns 2 MB in about 0.02 s here and still matches[email protected]. Please add a timing test that feedsanchor_strings"a." * 1_000_000plus a one-line pathological input for each pattern.
Recommended, not blocking:
- skills.py:27-29 reads
HERMES_HOMEfrom the environment. When one Hermes process serves several profiles, that is the launch profile's home, so writes in another profile's session go unrecorded.hermes_constants.get_hermes_home()is profile-aware.
We'll add a disclosure line at merge.
Push the fix, re-pin sha: here and comment when it's in; we'll review the delta from 554a951a.
anchor_strings runs on every tool result, and the unbounded local part in the e-mail pattern re-scanned a run of word characters from every offset: 40 KB of "a." took 4.1 s and 200 KB 104 s, while tool results are capped at 2 MB and re cannot be interrupted. The local part is now bounded to 64 octets (RFC 5321) and anchored with a lookbehind, so 2 MB costs 0.03 s in the pattern alone and 0.55 s through anchor_strings. tests/test_textfeat_perf.py gives every pattern one pathological input and runs the 2 MB reproducer through anchor_strings, each case in its own process with a hard timeout, so a regression fails the run instead of hanging it. skills.py takes the home from hermes_constants.get_hermes_home() — a public surface for plugins — resolved at call time, with an HERMES_HOME fallback for older releases. Skill lookup in a second profile's session no longer reads the launch profile's tree. Tests: 105 passed, 1 skipped on Python 3.11.16 and 3.14.7; hermes plugins validate reports no failures or warnings. Reported by @teknium1 in review of NousResearch/hermes-agent#142381.
0.1.5 bounds the e-mail pattern in anchor_strings (quadratic on hostile input: 40 KB of "a." took 4.1 s, 200 KB 104 s; now 2 MB takes 0.03 s) and adds per-pattern timing tests that run in a subprocess under a hard timeout. It also takes the Hermes home from hermes_constants.get_hermes_home(), so skill lookup follows the session's profile instead of the launch profile.
0.1.6 sets the 2 MB timing guard's bar where the hardware can actually meet it and records the second behavioural difference of the bounded e-mail pattern.
Found in a self-audit after the same class was reported against a sibling plugin (NousResearch/hermes-agent#142381). Both patterns had two adjacent unbounded whitespace runs that were retried from every backtrack position: _KEY_VALUE "key" + 1 MiB of spaces 0.67 s -> 0.003 s _TRAILING_PAREN "a" + 1 MiB of spaces + "b" >30 s -> under 0.001 s Both rewrites use possessive quantifiers, so the plugin now needs Python 3.11+. What is flagged does not change: the key may still contain spaces and no longer captures trailing whitespace (the caller strips it), and the parenthesis rule leaves the whitespace it used to absorb to the caller's rstrip(). tests/test_performance.py gains the four whitespace shapes, each in its own process under a 20 s timeout, and three whitespace entries in the end-to-end adversarial set. Tests: 733 passed, 2 skipped, 6 xfailed; corpora 213/213 blocked 0 and 174/174 expected outcomes met.
0.1.7 fixes the end-to-end failure introduced in 0.1.4 (a user turn is the origin of text the user dictated), so the hermes-tools job is green again; 0.1.8 keeps one address one anchor and adds hostile-input timings for all eight patterns.
|
Fixed and re-pinned: sha c21178d4578680297b6100638fff59b75d1b572e (0.1.8). Thanks for catching this, and for the bounded pattern. It's at textfeat.py:28 exactly as you tests/test_textfeat_perf.py runs that input plus hostile inputs for each of the eight patterns, Behaviour: local parts over 64 characters no longer match. The new pattern can also start on a HERMES_HOME: skills.py now calls hermes_constants.get_hermes_home() on each call, falling back One more thing in the delta from 554a951a. CI on 0.1.4 was red on two e2e tests: an earlier I also ran the same checks over provenance-gate and found two patterns in claims.py with the 131 passed, 1 skipped on Python 3.11 and 3.14. |
teknium1
left a comment
teknium1
left a comment
There was a problem hiding this comment.
Re-reviewed at c21178d4: the e-mail regex is bounded (with timing tests) and the home lookup follows the active profile via get_hermes_home(). Thanks! Approved. Our merge chores (reviewed disclosure line and any requires_hermes floor) are pushed to this branch, and it goes on today's merge train once CI is green.
|
Thanks for the review and the merge. The provenance-gate re-pin is now #144339: 0.1.3, with the two patterns from 0.1.1, one more found afterwards, and timing tests for all three. |
What it does
Records where each skill version's content came from, and keeps that record
verifiable afterwards.
For every skill write — a foreground
skill_managecall, a background-reviewedit, or a curator consolidation — the plugin stores the content fingerprint
before and after, the sources present earlier in the same session (user text,
tool results, skills that were read), the features that link a source to the
added lines, and a
seq -> prev_hash -> hashchain over the entries. Text isnever stored; hashes are keyed per install and the key stays in
plugin-data/skill-lineage.hermes skill-lineage trace <skill>/descendants <url|skill>/report/verify/headanswer where a version came from, what a source produced,which versions carry lines of no traceable origin, and whether the chain (or a
recorded head value) still matches. A key reset is recorded in the chain and
shown in
statusrather than happening silently.Hermes surfaces used
Four hooks, all read-only observers:
on_session_start,pre_llm_call,pre_tool_call,post_tool_call. One CLI command group(
hermes skill-lineage ...). No tools, no middleware, no env vars, no coreoverride. Capabilities in the entry match what the pinned commit registers.
Disclosure (rule 13)
Reads tool results, user messages and skill files to compute keyed hashes.
Stores hashes (never text) for 7 days per session, plus the lineage chain
(hashes and skill names) under
plugin-data/skill-lineage. No network calls, nocredentials, never blocks or changes a tool call.
Tests
python -m pytest -qon the pinned commit (c21178d, Python 3.11.16 and3.14.7): 131 passed, 1 skipped.
hermes plugins validate .: passes, nowarnings. The skip is the end-to-end suite, which needs a Hermes importable by
the running interpreter and skips with a printed reason rather than passing
silently.
Manual end-to-end smoke run on Hermes
e06df9c2ea(2026-10-21): installed intoan isolated
HERMES_HOME, enabled, and driven by one real session that createda skill through
skill_manage;status,trace,descendants,report,verifyandheadall returned as documented. Raw output intests/smoke-2026-10-21.txt.
Relation to existing work
revision, files, install path). That is where a skill came from at install
time; this records which session evidence a skill version's content was
derived from. Complementary, not overlapping.
skill-integrity(catalog) detects that installed skill files changed.This records origin: which source, which lines, whether the record is
intact. A drift detector and an origin record answer different questions.
$HERMES_HOME/skillsinsteadof importing
tools.skills_tool(rule 9).Owner submission (rule 5): I am the repository owner.