[FEA]: Name the Tier 1 token count an estimate, and offer a tokenizer or a set ratio #461

Problem to solve

Thank you for building SkillEvaluator. A token is not a shared unit: each model family counts with its own tokenizer, so a tool meant for every agent has good reason to estimate with characters divided by four. I think that choice is reasonable. What I found is smaller: the number is presented as measured, and the Efficiency score judges by it.

What I saw, running quality-check from the v0.8.0 tag on a public skill of mine:

• src/skillevaluator/validators/quality_score.py sets qs.total_tokens = len(content) // 4 (line 878 at v0.8.0; unchanged on main at 43a66115, 2026-11-11, where I read it but did not re-run). A comment above it says "Token estimates", and the CHANGELOG entry for fix(quality): derive instruction tokens from parsed frontmatter #356 (on main, merged after the tag) says "Estimate". The JSON key (metrics.total_tokens), the "Large skill (N tokens ...)" finding text and docs/tier1-validation.mdx ("Token budget", "the 5,000-token budget") say tokens with no word that it is an estimate.
• For a dense SKILL.md (tables, identifiers, code spans, long lines) characters per token is well below 4, so the estimate comes out low. My file, SKILL.md of the public skill TheColliery/CoalFace at tag v0.15.3, is 17,756 characters, so it reports 17,756 // 4 = 4,439 tokens. That is under the 5,000 recommendation: no "Large skill" finding, Efficiency 100.
• Claude Code's own plugin details for the installed copy (byte-identical at the previous patch version) list about 6.8k tokens on invoke, about 1.5 times the reported figure and over 5,000. That figure is itself an estimate, and I have not checked how Claude Code derives it, so I offer it as a second estimate, not as the truth.

So for a dense file the two estimates sit on opposite sides of the recommendation, and the report does not tell the reader that the 4,439 is a rough count.

Reproduction, with the public repositories:

git clone --depth 1 --branch v0.8.0 https://github.com/NVIDIA/SkillEvaluator.git
git clone --depth 1 --branch v0.15.3 https://github.com/TheColliery/CoalFace.git
cd SkillEvaluator
uv run --python 3.13 skillevaluator quality-check ../CoalFace/plugin/skills/coalface -r cli -r json -o reports/coalface

1. Open reports/coalface/skillevaluator-quality.json and read metrics.total_tokens: 4439. Efficiency is 100.0 and there is no "Large skill" finding.
2. Count the characters: python -c "from pathlib import Path; print(len(Path('../CoalFace/plugin/skills/coalface/SKILL.md').read_text(encoding='utf-8')))" prints 17756; 17756 // 4 is 4439.
3. Compare with any tokenizer you trust for the model you target.

Proposed approach

Offered for you to weigh, not prescribed:

• Name it an estimate where the reader meets it: the report and the docs say "estimated tokens (characters / 4)", and perhaps the JSON key follows (for example estimated_tokens). This is the smallest change and the one I would ask for.
• Optionally, a way to do better for people who want it: a real tokenizer behind an optional extra (falling back to the estimate when it is not installed), or a configurable characters-per-token ratio, in the way SKILLEVALUATOR_SCHEMA_ALLOWED_DIRS already lets a repository adjust a default.
• Or leave the estimate as is and state the assumption in the docs.

Additional context

• I do not know which tokenizer the 5,000 figure was meant for, and I have not run a published tokenizer over my file.
• I have tested one dense skill only, so I do not know how common the gap is. Prose-heavy skills may be close to 4.
• Since fix(quality): report the 5,000-token skill budget as advisory #272 made the over-budget finding advisory, the practical effect is a missed MEDIUM finding and a 15-point Efficiency deduction that never applies, so the composite score is higher than a tokenizer would give. It does not change a gate by itself.
• Environment: Windows 11, uv with Python 3.13, uv sync --frozen from the tag, base dependencies only. The check has no model in it, so a repeat gives the same output.

Before submitting

• I agree to follow this project's Code of Conduct
• I have searched the open issues and found no duplicate of this request