Skip to content

HeuristicScorer only checks frontmatter description for triggers — ignores SKILL.md body #223

Description

@diberry

Bug

HeuristicScorer.Score() in internal/scoring/scoring.go checks for trigger patterns (USE FOR:, DO NOT USE FOR:, **UTILITY SKILL**, etc.) only in sk.Frontmatter.Description. It never inspects the SKILL.md body text. Skills that place these routing sections in the body — which is the natural authoring pattern — are scored as Low compliance with "missing triggers" even though the sections are present.

Expected behavior

waza check should detect USE FOR:, DO NOT USE FOR:, and routing clarity markers (**UTILITY SKILL**, INVOKES:, etc.) wherever they appear in SKILL.md — both the frontmatter description and the body.

Actual behavior

waza check reports Low compliance and recommends adding sections that already exist:

Compliance Score: Low
  Needs significant improvement. Description too short or missing triggers.

Next Steps:
  1. Add a 'USE FOR:' section with 3-5 trigger phrases
  2. Add a 'DO NOT USE FOR:' section
  3. Add routing clarity (e.g., **UTILITY SKILL**, INVOKES:)

Root cause

In internal/scoring/scoring.go, lines ~130-140:

desc := sk.Frontmatter.Description
// ...
trimmedDesc := strings.TrimSpace(desc)

result.HasTriggers = containsAny(trimmedDesc, triggerPatterns)
result.HasAntiTriggers = containsAny(trimmedDesc, antiTriggerPatterns)
result.HasRoutingClarity = containsAny(trimmedDesc, routingClarityPatterns)

All three checks use trimmedDesc which is only the frontmatter description field. The body text (sk.Body or equivalent) is never searched.

The computeLevel() function then gates on these booleans:

func computeLevel(r *ScoreResult) AdherenceLevel {
    if r.DescriptionLen < 150 || !r.HasTriggers {
        return AdherenceLow   // <-- always hits this
    }
    // ...
}

Minimal reproduction

Create a SKILL.md with triggers in the body (not the description):

---
name: test-skill
description: A test skill that demonstrates the scoring bug where triggers in the body are not detected by the heuristic scorer compliance check.
allowed-tools: []
---

UTILITY SKILL — Test skill for reproduction.

USE FOR: testing waza compliance scoring, reproducing trigger detection bugs.

DO NOT USE FOR: production use, non-test scenarios.

## References

- [references/example.md](references/example.md) — Example reference

Then run:

waza check test-skill/

Result: Compliance Score: Low, recommends adding USE FOR / DO NOT USE FOR / routing clarity — all of which are present in the body.

Workaround: Stuff all triggers into the frontmatter description string (as skills/waza/SKILL.md does), but this hurts readability and conflicts with the natural body-section authoring pattern.

Suggested fix

In HeuristicScorer.Score(), search both the description and the body:

searchText := trimmedDesc + "\n" + strings.TrimSpace(sk.Body)

result.HasTriggers = containsAny(searchText, triggerPatterns)
result.HasAntiTriggers = containsAny(searchText, antiTriggerPatterns)
result.HasRoutingClarity = containsAny(searchText, routingClarityPatterns)

Or alternatively, parse the body separately and merge the results so trigger counts remain accurate.

Environment

  • waza version: latest (installed via azd extension install waza)
  • OS: Windows 11
  • Discovered while running waza check on 6 Azure SDK review skills that all have USE FOR / DO NOT USE FOR / UTILITY SKILL in the body

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions