feat: add content-based file type matching by jdx · Pull Request #416 · jdx/hk

jdx · 2025-11-13T18:11:14Z

Summary

Adds support for matching files by their content type (extension, shebang, or magic number) rather than just glob patterns. This addresses the feature request in #413 for matching scripts without file extensions.

Problem

Users previously had to use complex glob patterns with excludes to match files, making it difficult to handle scripts without file extensions. For example, matching Python scripts that use #!/usr/bin/env python3 but don't have a .py extension required error-prone regex patterns.

Solution

Added a new types field to the Step configuration that uses OR logic (file matches ANY of the specified types):

["black"] {
    types = List("python")  // Matches .py files AND Python scripts by shebang
    fix = "black {{files}}"
}

["shellcheck"] {
    types = List("shell")  // Matches shell files by extension & shebang
    check = "shellcheck {{files}}"
}

Implementation

File type detection in new src/file_type.rs module with caching:
- Extension-based detection (e.g., .py → python)
- Shebang parsing for scripts (e.g., #!/usr/bin/env python3)
- Magic number detection using infer crate for binary files
- Special filename detection (e.g., Dockerfile)
Supported types include:
- Languages: python, javascript, typescript, ruby, go, rust, etc.
- Shells: shell, bash, zsh, fish
- Data formats: json, yaml, toml, xml
- Special: text, binary, executable, dockerfile
Updated builtins (black, shfmt) to use the types field

Test Plan

✅ Added unit tests in src/file_type.rs for all detection methods
✅ Added comprehensive bats integration tests in test/types.bats:
- Extension-based matching
- Shebang-based matching
- OR logic with multiple types
- Combining types with glob patterns
✅ All existing tests pass
✅ Updated documentation with examples and supported types

Migration

This is fully backwards compatible - existing glob patterns continue to work. The types field is optional and can be used alongside glob patterns for more precise filtering.

🤖 Generated with Claude Code

Note

Introduces file type detection (extension, shebang, magic) and a new Step.types filter, with docs/tests and infer dependency.

Core/Engine:
- Add src/file_type.rs with cached detection by extension, shebang, and magic numbers (via infer); handles symlinks/executables.
- Extend Step with types: List<String> and filter files accordingly; update skip logic; wire module in src/main.rs.
Config/PKL:
- Add types field to pkl/Config.pkl Step schema.
Docs:
- Document <STEP>.types in docs/configuration.md with examples and supported types.
Tests:
- Add unit tests in src/file_type.rs and integration tests in test/types.bats.
Dependencies:
- Add infer (plus transitive cfb, uuid) in Cargo.toml/Cargo.lock.

^{Written by Cursor Bugbot for commit 74ac521. This will update automatically on new commits. Configure here.}

Add support for matching files by their content type (extension, shebang, or magic number) rather than just glob patterns. This addresses user requests for matching scripts without file extensions. **Problem:** Users want to match files based on their content type (e.g., Python scripts without .py extension, shell scripts identified by shebang) rather than relying solely on filename patterns. **Solution:** - Added `types` field to Step configuration (OR logic - matches ANY type) - Implemented file type detection in new `src/file_type.rs` module: - Extension-based detection (e.g., .py → python) - Shebang parsing for scripts (e.g., #!/usr/bin/env python3) - Magic number detection using `infer` crate for binary files - Special filename detection (e.g., Dockerfile) - Results are cached for performance **Supported types include:** - Languages: python, javascript, typescript, ruby, go, rust, etc. - Shells: shell, bash, zsh, fish - Data formats: json, yaml, toml, xml - Special: text, binary, executable, dockerfile **Example usage:** ```pkl ["black"] { types = List("python") // Matches .py files AND Python scripts fix = "black {{files}}" } ["shellcheck"] { types = List("shell") // Matches shell files by extension & shebang check = "shellcheck {{files}}" } ``` **Changes:** - Added `infer` crate dependency for magic number detection - Created `src/file_type.rs` module with detection logic and tests - Updated `pkl/Config.pkl` schema with `types` field - Updated `src/step.rs` to apply type filtering after glob/regex - Updated builtins (black, shfmt) to use types field - Added comprehensive bats tests in `test/types.bats` - Updated documentation with examples and supported types Fixes: #413 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

cursor · 2025-11-13T18:13:39Z

pkl/builtins/shfmt.pkl

-    stage = glob
+    // Matches shell files by extension and shebang (including extensionless scripts)
+    types = List("shell")
+    glob = List("**/*.bats")  // Also include .bats files which may not be detected by shebang


Bug: Misclassified .bats files excluded by filters.

The types and glob filters combine with AND logic, but .bats files won't match types = List("shell") since the .bats extension isn't recognized as shell in get_types_by_extension and the #!/usr/bin/env bats shebang isn't handled in detect_shebang. This causes .bats files to be excluded despite matching the glob pattern, breaking shfmt for bats test files.

- Use next_back() instead of last() for DoubleEndedIterator - Remove unnecessary trim() before split_whitespace() - Replace vec! with slices in tests

socket-security · 2025-11-13T18:14:48Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/infer@0.16.0

View full report

Keep builtins using glob patterns for backwards compatibility. Users can opt-in to types in their own configurations.

src/file_type.rs

**Problem:** The function unconditionally inserted "binary" after infer detected a file type, preventing the fallback text-vs-binary detection from executing. Files without recognized magic numbers wouldn't get proper text/binary classification. **Root Cause:** 1. Line 171 inserted "binary" immediately after infer returned a result 2. This made types non-empty, so the condition at line 212 (if types.is_empty()) would never be true 3. The fallback null-byte scanning never ran for unrecognized file types 4. The file variable was opened but unused since infer::get_from_path opens the file independently **Solution:** - Early return after magic number detection succeeds - Only open file once for the null-byte scanning fallback - Now files without magic numbers properly fall through to text/binary detection **Tests:** - Added test_text_file_without_extension() to verify plain text detection - Added test_binary_file_detection() to verify null-byte detection - All existing tests still pass

src/file_type.rs

**Problem:** Symlinks were detected and returned early with only the "symlink" type, preventing them from being matched by content-based type filters. When allow_symlinks is enabled and a user specifies types = List("python"), symlinks pointing to Python files wouldn't match because get_file_types never checked the target's extension, shebang, or content. **Root Cause:** The early return at line 26 bypassed all subsequent type detection logic. Symlinks were classified as {"symlink"} only, losing information about their target file type. **Solution:** 1. Don't return early for symlinks - continue detection 2. Read the symlink target path when checking filename and extension 3. Use std::fs::metadata (follows symlinks) for executable check 4. Keep symlink_metadata only for detecting the symlink itself **Result:** Symlinks now get both "symlink" type AND their target's types: - Link to .py file: {"symlink", "python", "text"} - Link to shell script: {"symlink", "shell", "bash", "text"} **Tests:** - test_symlink_to_python_file: Verifies symlinks get target types - test_symlink_matches_target_type: Verifies type filtering works - All 114 tests pass

@scop

cursor bot reviewed Nov 13, 2025

View reviewed changes

fix: address clippy warnings in file_type.rs

575dc95

- Use next_back() instead of last() for DoubleEndedIterator - Remove unnecessary trim() before split_whitespace() - Replace vec! with slices in tests

revert: remove modifications to builtins

6584402

Keep builtins using glob patterns for backwards compatibility. Users can opt-in to types in their own configurations.

cursor bot reviewed Nov 13, 2025

View reviewed changes

src/file_type.rs Show resolved Hide resolved

cursor bot reviewed Nov 13, 2025

View reviewed changes

src/file_type.rs Show resolved Hide resolved

jdx merged commit 1d069f3 into main Nov 13, 2025
14 checks passed

jdx deleted the feat/content-based-file-matching branch November 13, 2025 20:16

jdx mentioned this pull request Nov 13, 2025

chore: release v1.21.0 #384

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

feat: add content-based file type matching#416

feat: add content-based file type matching#416
jdx merged 5 commits intomainfrom
feat/content-based-file-matching

jdx commented Nov 13, 2025 •

edited by cursor bot

Loading

Uh oh!

cursor bot Nov 13, 2025

Uh oh!

socket-security bot commented Nov 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments

Conversation

jdx commented Nov 13, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Implementation

Test Plan

Migration

Uh oh!

cursor bot Nov 13, 2025

Choose a reason for hiding this comment

Bug: Misclassified .bats files excluded by filters.

Uh oh!

socket-security bot commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jdx commented Nov 13, 2025 •

edited by cursor bot

Loading

Bug: Misclassified `.bats` files excluded by filters.

socket-security bot commented Nov 13, 2025 •

edited

Loading