i18n-bugbench

Real internationalization and portability bug fixes from open-source projects, packaged so that an AI coding agent can be asked to reproduce each fix and be checked against the project's own tests.

19 tasks from 15 repositories in Python, TypeScript/JavaScript, Rust, Go and dictionary data. 12 of the fixes are merged upstream and 7 are open pull requests. All 19 tasks are validated: the hidden tests fail at the base commit and pass with the reference fix, on Windows 11 (18 tasks), on Linux (9 tasks), or both.

Why

Agent benchmarks such as SWE-bench mostly come from Linux CI with a UTF-8 locale, UTC and English text. A whole class of bugs never shows up there:

• Windows paths: drive letters read as URL schemes, \ vs /, \\?\ verbatim paths, MAX_PATH.
• Legacy code pages: text read or written with cp932, cp936 or cp950 instead of UTF-8.
• CRLF line endings.
• Multibyte text: byte lengths vs character lengths, cutting UTF-8 in the middle of a character.
• CJK input: numerals, dates, regional vocabulary.
• Percent-encoded non-ASCII text in URLs and forms.

These bugs are common, and they are easy to miss in review because the project's own CI usually passes. Fixing one usually means reproducing the environment first, not only reading code. The tasks here are bugs of this kind that were found, fixed and accepted, or at least submitted, upstream.

Tasks

Task	Upstream PR	Language	Category	Bug shows on	Upstream	Hidden tests	Split
chrono-682	wanasit/chrono#682	typescript	cjk-parsing	any	open	upstream-pr	pending
clack-627	bombshell-dev/clack#627	typescript	windows-path	any	open	upstream-pr	pending
dash-4074	plotly/dash#4074	python	url-encoding	any	open	upstream-pr	pending
edge-tts-506	rany2/edge-tts#506	python	multibyte-text	any	open	benchmark	pending
fumadocs-3752	fuma-nama/fumadocs#3752	typescript	windows-path	any	merged	upstream-pr	main
git-cliff-1657	orhun/git-cliff#1657	rust	windows-path	windows	merged	benchmark	main
glamour-651	charmbracelet/glamour#651	go	line-endings	any	open	upstream-pr	pending
isort-2744	PyCQA/isort#2744	python	windows-path	windows	open	upstream-pr	pending
lsd-1248	lsd-rs/lsd#1248	rust	windows-path	windows	open	upstream-pr	pending
monolith-541	Y2Z/monolith#541	rust	windows-path	windows	merged	upstream-pr	main
mycli-2389	dbcli/mycli#2389	python	locale-encoding	any	merged	upstream-pr	main
opencc-1509	BYVoid/OpenCC#1509	data	locale-terminology	any	merged	upstream-pr	main
opencc-1510	BYVoid/OpenCC#1510	data	locale-terminology	any	merged	upstream-pr	main
opencc-1511	BYVoid/OpenCC#1511	data	locale-terminology	any	merged	upstream-pr	main
opencc-1512	BYVoid/OpenCC#1512	data	locale-terminology	any	merged	upstream-pr	main
opencc-1514	BYVoid/OpenCC#1514	data	locale-terminology	any	merged	upstream-pr	main
pdfkit-1828	foliojs/pdfkit#1828	javascript	windows-path	windows	merged	benchmark	main
purgecss-1505	FullHuman/purgecss#1505	typescript	windows-path	any	merged	upstream-pr	main
vcrpy-1066	kevin1024/vcrpy#1066	python	url-encoding	any	merged	upstream-pr	main

"Bug shows on" is where the hidden tests fail at the base commit. Five tasks only fail on Windows; on Linux and macOS their tests already pass before the fix, so a run there says nothing about the agent.

Categories

Category	Tasks	What goes wrong
windows-path	8	Drive letters, backslashes, verbatim paths, MAX_PATH, paths passed where a URL is expected
locale-terminology	5	zh-CN to zh-TW conversion data turns ordinary words into the wrong Taiwan term
url-encoding	2	Percent-encoded names and paths compared without decoding
locale-encoding	1	A bundled UTF-8 file opened with the locale's code page (cp932)
multibyte-text	1	A byte limit enforced on text without spaces, splitting characters
line-endings	1	CRLF input rendered differently from LF
cjk-parsing	1	Chinese numerals for days 21-31 cut off

By language: Python 5, TypeScript 4, JavaScript 1, Rust 3, Go 1, dictionary data (OpenCC) 5.

Splits

• main (12): merged upstream, validated.
• pending (7): the upstream pull request is still open, validated. The fix may still change in review.
• unvalidated (0): tasks not yet validated anywhere. purgecss-1505 could only be validated on Linux: purgecss's own Jest config can't find or transform its tests on Windows.

A task counts as validated when validation/<task>.json has a record in which the hidden tests fail at the base commit, pass with the reference fix, and the regression command (existing tests) passes in both runs. Each record lists the OS, locale encoding, time zone and tool versions.

What a task contains

tasks/monolith-541/
  task.json     repository, base commit, environment, setup and test commands
  problem.md    the bug report given to the agent, written without hints about the fix
  gold.patch    the reference fix (the upstream change minus tests and changelogs)
  test.patch    hidden tests taken from the upstream PR

Three tasks have no usable upstream test. git-cliff-1657 and pdfkit-1828 use a black-box script (check_*.py, check_*.mjs) that runs the built program the way the bug report describes. edge-tts-506 has a small pytest file written for this benchmark that checks the function's documented contract. Where an upstream test only exercised a helper the fix introduced (monolith-541), it is left out, so the hidden tests don't require the agent to pick the same function name.

Fields in task.json:

Field	Meaning
repo, base_commit	Where the agent starts. For merged PRs this is the commit the fix was merged onto (for merge commits, the PR's merge base).
upstream_pr, upstream_status, merge_commit	Provenance
environment.reproduces_on	windows, or any
environment.locale, tz, toolchain, notes	What the run needs, and anything unusual about it
setup	Commands run after the patches are applied (install, build)
test_command	The hidden tests; a list of commands when there are several. All must exit 0.
regression_command	Existing tests that must keep passing
hidden_tests	Where the tests come from (upstream-pr or benchmark) and which ones run
license	The upstream project's license, which applies to the patches

Commands are argv lists, not shell strings. {python}, {workdir}, {task_dir} and {bench} are replaced at run time.

Running

The harness needs Python 3.9 or newer and Git, and nothing else; each task needs its own toolchain (Node.js with corepack, Rust, Go, or Python). Windows-only tasks need Windows.

python -m bugbench list
python -m bugbench prompt monolith-541 > prompt.txt

Give the prompt to the agent together with a checkout of the repository at the base commit, and save the agent's changes as a unified diff (git diff > candidate.patch). Then:

python -m bugbench run monolith-541 --patch candidate.patch

To score several tasks, write one JSON object per line, in the same format as SWE-bench predictions:

{"task_id": "monolith-541", "model_patch": "diff --git a/src/core.rs b/src/core.rs\n..."}

python -m bugbench evaluate predictions.jsonl --name my-agent --out results/my-agent

This writes results.json, the logs, and summary.md with one row of the results table below. Runs on a platform where a task's bug doesn't reproduce are reported but not counted.

A run does this:

1. Clone the repository once (a bare partial clone under .work/repos, or $BUGBENCH_WORK) and check out the base commit in a worktree. core.autocrlf is turned off for the clone so that patches see the repository's bytes.
2. Apply the candidate patch with git apply.
3. Put back any file the hidden tests own, so a candidate can't edit or replace them, then apply test.patch.
4. Run setup, then test_command, then regression_command. The task is resolved when every command exits 0.

python -m bugbench validate [task ...] runs the base commit and the reference fix and writes the validation record. python -m bugbench split-diff splits an upstream diff into gold.patch and test.patch when adding a task.

Candidate patches and the projects' build scripts run arbitrary code on your machine. Run evaluations in a VM or a disposable container.

Results

No agent has been evaluated yet. Template:

Agent	Date	Platform	main (12)	pending (7)	Windows-only, of those (5)
name, version	YYYY-MM-DD	windows / linux	x / 12	x / 7	x / 5

Report the platform: on Linux or macOS only 9 of the 12 main tasks and 5 of the 7 pending tasks can be counted.

Validation environment

• Windows (18 tasks, every task except purgecss-1505): one machine with Windows 11 Pro (build 26200), Japanese system locale (code page cp932), time zone Asia/Tokyo, Python 3.11.9, Node.js 24.15.0, Rust 1.99.1 (x86_64-pc-windows-gnu), Go 1.27.3, Git 2.51.2.
• Linux (9 tasks: purgecss-1505 and 8 of the tasks marked any): GitHub Actions ubuntu-latest (UTF-8 locale, UTC), Python 3.11, Node.js 24, Go 1.27, through the validate workflow.

macOS has not been used for validation.

Building this on a cp932 machine turned up more bugs of the same kind:

• The harness itself crashed with UnicodeEncodeError when it printed a test log on the cp932 console. The black-box script for git-cliff-1657 crashed the same way on the emoji in the changelog, which made the base commit "fail" for the wrong reason. Both are fixed, and every validation log was read to check that the base commit fails for the reason the task describes.
• purgecss's Jest config doesn't find or transform its tests on Windows (testMatch and transformIgnorePatterns assume /).
• Some existing Dash Pages tests and clack's path.test.ts fail on Windows at the base commit; fumadocs still writes some generated paths with \ after the fix. These tests are left out of the regression commands, and the task's notes say so.
• lsd's broken-symlink test needs Developer Mode on Windows.

Limitations

• Small. 19 tasks, and the categories are uneven: 8 Windows-path tasks and 5 OpenCC vocabulary tasks, but only one task each for code pages, CRLF, multibyte text and CJK parsing. There are no time zone/DST or Turkish-locale tasks yet.
• One author. Every fix is a pull request by the same contributor (Wei-Lun Li), written with an AI coding agent (Claude Code), and the problem statements were written by the same person. Agents may find these fixes more "natural" than average, and the selection reflects what one person happened to find in October 2026.
• Contamination. All the pull requests, issues and patches are public. A model trained on data from after October 2026 may have seen the fixes. Treat scores as an upper bound for such models.
• Not the upstream test setup everywhere. The OpenCC tasks run the upstream test cases with the PyPI opencc 1.4.2 engine and text dictionaries built from the checkout (the same version as the base commits) instead of a C++ build, and they don't run upstream's C++ dictionary checks (sort order, reverse-mapping completeness). dash-4074 takes Dash's generated component files from the 4.4.1 wheel because the base commit (4.5.0rc2) would need an npm build of the whole monorepo.
• Hidden tests can be stricter than the bug report. Where the expected output is a matter of judgment (the OpenCC vocabulary), the problem statement lists the expected conversions.
• Windows-only tasks need Windows. Five tasks only reproduce there, and the code page can't be switched per process on Windows, so the locale-encoding task detects the problem through Python's EncodingWarning instead.
• Network and drift. Dependencies are installed from npm, PyPI, crates.io and the Go proxy at run time. Lockfiles pin most of them, but nothing is vendored.
• Few validation environments. One Windows machine and GitHub's Linux runner. fumadocs-3752 and four of the OpenCC tasks are marked any but were only validated on Windows; nothing was validated on macOS.

License

The harness, problem statements, task metadata and the tests written for this benchmark are under the MIT License. The gold.patch and test.patch files are excerpts of the upstream projects and stay under each project's license; see THIRD_PARTY.md. This project is not affiliated with the upstream projects.