test: reduce fake-bin fixture coverage#240
Merged
zeitlinger merged 3 commits intomainfrom Apr 27, 2026
Merged
Conversation
Signed-off-by: Gregor Zeitlinger <[email protected]>
Signed-off-by: Gregor Zeitlinger <[email protected]>
Signed-off-by: Gregor Zeitlinger <[email protected]>
zeitlinger
force-pushed
the
codex/reduce-fake-bin-test-confidence
branch
from
f68c7b7 to
9a42cee
Compare
martincostello
approved these changes
Apr 27, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves test-suite confidence by reducing reliance on synthetic fake_bins fixtures and shifting coverage toward deterministic unit tests and real-tool fixture executions (notably rumdl and ryl), while keeping targeted fakes only where they validate Flint-owned orchestration behavior.
Changes:
- Stabilize and tighten E2E harness behavior (CI env normalization, stricter fake-biome contract, normalize environment-dependent mise pin warnings).
- Replace a removed
linterstable E2E fixture with deterministic unit coverage by refactoring table rendering into a pure function. - Replace several
fake_binsfixture cases with real-tool fixtures (pinning tool versions viamise.toml) and add unit-test seams for mise pin resolution behavior.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
tests/e2e.rs |
Stabilizes E2E output across environments and tightens remaining fake-bin contracts; adds normalization for mise pin fallback warnings. |
tests/cases/ryl/** |
Converts multiple ryl fixtures from fakes to real-tool runs using real .yamllint.yml configs and adds new regression cases. |
tests/cases/rumdl/** |
Converts rumdl fixtures from fakes to real rumdl 0.1.78 executions and updates assertions accordingly. |
tests/cases/general/list/** |
Removes the prior fake-bin-heavy linters fixture cluster (replaced by unit tests in src/main.rs). |
tests/cases/general/init-rust/test.toml |
Removes fake mise contract from init fixture expectations (mise pin behavior now covered via unit seams). |
tests/cases/general/init-idempotent/test.toml |
Drops fake mise usage from idempotent init fixture. |
tests/cases/general/fast-only-explicit-override/test.toml |
Tightens the remaining synthetic case to assert invocation via a sentinel file. |
src/runner.rs |
Adds a unit test asserting ryl config injection ordering in constructed invocations. |
src/main.rs |
Refactors linter table printing into a pure render_linters_table + adds deterministic unit coverage for the table output and status logic. |
src/init/mise_tools.rs |
Introduces injectable runner seams for mise use --pin flows and adds unit tests to verify “resolved pin” behavior is preserved. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Merged
zeitlinger
added a commit
that referenced
this pull request
Apr 27, 2026
### Added - add flint setup preflight migrations ([#233](#233)) - support managed linter configs and aqua-backed tool keys ([#228](#228)) - switch xmloxide to github backend ([#229](#229)) - add taplo formatter check ([#224](#224)) - *(init)* configure line length during init ([#218](#218)) ### Fixed - support renovate v43.136.3 ([#223](#223)) - switch yaml-lint to ryl ([#226](#226)) - *(ruff)* install via github releases ([#227](#227)) - resolve init rollout regressions ([#220](#220)) ### Other - reduce fake-bin fixture coverage ([#240](#240)) - *(deps)* update dependency npm:renovate to v43.141.1 ([#221](#221)) - guard fixture case groups ([#238](#238)) - *(deps)* update dependency grafana/flint to v0.20.4 ([#235](#235)) - *(deps)* lock file maintenance ([#237](#237)) - *(deps)* update dependency mise to v2026.4.19 ([#236](#236)) - *(deps)* update rust crate json5 to v1 ([#232](#232)) - *(deps)* update taiki-e/install-action digest to 787505c ([#219](#219)) - *(deps)* update dependency grafana/flint to v0.20.3 ([#225](#225)) > [!IMPORTANT] > Close and reopen this PR to trigger CI checks. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gregor Zeitlinger <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
This reduces the use of confidence-reducing fake binaries in the test suite and replaces several of them with more direct or higher-confidence coverage.
Key changes:
flint linterstable coverage out of a fake-bin fixture and into deterministic unit testsmiseusage frominit-idempotentinit-rustfake-misecontract with an injectable Rust seam plus unit coverage for resolved pin behaviorrumdlfixture cases to use realrumdl 0.1.78rylfixture cases to use the real tool and real.yamllint.ymlconfig, including newpartial-after-fixandwarning-only-auto-fixcasesWhy it changed
Several tests were simulating upstream CLI behavior rather than testing Flint itself. That reduced confidence because the suite could pass even if the real tools behaved differently.
This change keeps synthetic tests where they are valuable for Flint-owned orchestration, but replaces the weaker cases with real-tool fixtures or direct unit tests.
Impact
init,rumdl, andrylcoveragerylwarning-only--fixbehavior explicit with a real regression casegeneral/list)No production behavior change was confirmed by this work; this is primarily a test-confidence cleanup.
Validation
cargo testmise run lint:fixmise run lintrumdl,ryl,init, and the remaining synthetic cases