Repository navigation
Releases
7.7.0
Big release: raising max_bytes to cover a huge input no longer makes detect() slow, every answer now decodes the window it was detected from, and markup declarations are read the way browsers read them. A lot of this came from new contributors, so thank you all!
Performance
- Raising
max_byteson large inputs no longer gets slow. On compiled wheels,detect(data, max_bytes=len(data))on 272 MiB takes about 0.2s for cp1252 and 0.5s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, and the other stages stop at a 256 KiB evidence cap (ADR-0006). The default 200 KB cap is still the fastest path, and default calls are unchanged. - The pure-Python wheel on CPython scores bigram models through
operator.itemgetterinstead of a Python loop, makingdetect()about 1.25x faster anddetect_all()about 1.35x faster. (#404, thanks @deadstrobe5) - Fixed three scans that went quadratic on hostile input: the UTF-7
+scan on line-wrapped base64 (128 KiB went from 5.4s to 1.2ms), unclosed<metaand<?xmltags in the markup scan, and the EBCDIC charset-declaration regex. (#403, thanks @Samin061)
Bug Fixes
- Every answer now decodes the examined window under the name you get back. BOMs, BOM-less UTF-16/32 null patterns, ISO-2022/HZ escapes, and
<meta charset>declarations used to be trusted without decoding the rest of the window, so a confident answer could still raise further in. The 5.x/6.x display names (EUC-JP,SHIFT_JIS,Big5,ISO-2022-JP) are only used when those narrower codecs decode it too. (#394 and #396, thanks @Samin061) - Markup declarations follow the WHATWG prescan: declarations inside comments are ignored, a
charset=inside another attribute (like a refresh URL) no longer beats a real<meta charset>, quoted or spacedcontentcharsets work, and an ASCII page declaringutf-16leno longer comes back as UTF-16-LE. (#389 and #398, thanks @SashaMIT and @xiehuanyi) - PEP 263 cookies resolve the way
tokenize.detect_encodingresolves them, including only counting a line-2 cookie when line 1 is blank or a comment. (#391, thanks @xiehuanyi) - Fixed BOM-bearing UTF-32 coming back as UTF-16 when
max_bytescuts a code unit. (#405, thanks @AryanRG) - Fixed language labels on Unicode text tilting toward accent-heavy languages, so a French paragraph with a few accents no longer comes back as Hungarian. Mislabels on the test corpus's Unicode files dropped from 140 to 104.
- Fixed a mostly-ASCII Windows-1252 file with a single
Ödetecting as HP-Roman8, and English with a loneÉdetecting as MacRoman. (#383, thanks @aadsm) - Fixed null-separated records like
find -print0output coming back as UTF-16-LE with Chinese as the language. prefer_superset=Trueno longer swaps in a code page that can't decode the data, and now respectsinclude_encodings/exclude_encodings. (#393, thanks @00200200)chardetectandUniversalDetectorno longer mistake input cut exactly at the byte cap for the whole stream. (#392, thanks @00200200)lookup_encoding's cache is capped at 1024 entries, since charset names come from untrusted markup. (#390, thanks @Samin061)- Plus smaller fixes to
detect_all()ranking, EBCDIC XML MIME types, and the deprecatedchardet.equivalencesshim. The changelog has the full list.
Improvements
- New
-m/--mime-typeflag forchardetectshows the MIME type next to the encoding, likefile -i. (#384, thanks @smi11 for the request) - Visual-order Hebrew is now detected, and cp864 Arabic is trained on contextually shaped text in visual order, the way real cp864 systems stored it.
EncodingInfo.is_multibyteis nowis_multibyte_cjk. The old name still works with aDeprecationWarningand goes away in 8.0.pipeline/markup.pyis now compiled by mypyc (15 modules total), and CI enforces 100% test coverage.
Full Changelog: 7.6.0...7.7.0
7.6.0
Big release: a Cython scoring kernel joins mypyc in compiled wheels, every model retrained on a deduplicated corpus, UTF-7 fixed in both directions, and a guarantee that detect() never returns an encoding that can't decode your complete input.
Performance
- Compiled wheels now score bigram profiles through a small Cython kernel alongside mypyc, and the pair is 4.7x faster than the pure wheel on CPython 3.14.
_kernel.pystays plain Python (PyPy and pure wheels run it interpreted, unchanged), and detection output is bit-identical. The kernel declares itself safe without the GIL, so free-threaded CPython scales instead of silently re-enabling the GIL on import: 3.14t runs the whole suite in ~340ms across 8 threads, the fastest configuration measured. Compiled builds now need both hooks:HATCH_BUILD_HOOK_ENABLE_MYPYC=true HATCH_BUILD_HOOK_ENABLE_CUSTOM=true. - Added support for CPython 3.15, including the free-threaded build. No code changes were needed.
Bug Fixes
detect()no longer returns an encoding that cannot decode the input it was given (#380, thanks @yarikoptic). When the whole input has been examined and the winner's only multi-byte evidence is an incomplete trailing sequence, the best candidate that decodes the input completely wins instead. Genuinely truncated data keeps its answer.- Fixed delimited ASCII data like
|NAME,+LAY|misdetecting as UTF-7 (#371 follow-up, thanks @agreenburg). The whole buffer must now actually decode as UTF-7, and a lone shifted character must land in a plausible script range. - Signed UTF-7 no longer reads as ASCII: the BOM stage recognizes the four UTF-7 signature prefixes when the rest of the buffer decodes as UTF-7.
- Fixed short apostrophe-heavy English being labeled Scottish Gaelic or Breton: a rare-language label on an input under 128 bytes now needs a 0.03 lead over the best mainstream language (ADR-0005).
- Fixed Hungarian text losing to a Czech reading in confusion rescoring; tied pairs are compared only under language models both encodings have.
- Fixed space-padded text matching a degenerate Serbian model at high confidence; statistical scoring now skips repeated-whitespace bigrams. This also fixes windows-1251 logs misdetecting as windows-1250 (#379).
- Fixed EBCDIC text being invisible to the early pipeline stages, and the last two EBCDIC sibling misdetections.
- Fixed training normalization gaps that starved ISO-8859-16 and the 26 pre-euro encodings at exactly their distinguishing bytes.
Improvements
- Retrained every bigram model on a refreshed, deduplicated corpus with training provenance now recorded per model.
- New ANSI-art model for cp437, trained on 16,621 text-mode art files from 16colo.rs.
- Rare-language arbitration (ADR-0005): low-confidence statistical winners from languages with no documented legacy-encoding population yield to near-tied mainstream candidates.
- Confusion-group resolution is context-aware: per-occurrence votes, word-shape demotions, art-model exemption.
- Statistical dead heats no longer resolve by candidate enumeration order.
- Training pipeline hardening after a cache-loss post-mortem.
Full Changelog: 7.5.1...7.6.0
7.5.1
Patch release: three detection fixes found while benchmarking against charset-normalizer's char-dataset.
Bug Fixes
- Fixed markup-declared encodings being reported under a name that can't decode the input. A page declaring
Shift_JISbut using CP932 extension characters (like ①) came back asSHIFT_JIS, which fails.decode()on those same bytes. Superset promotion (CP932,CP949) now always fires when the reported name can't decode the data but the superset can. - Fixed a lying charset declaration beating genuine UTF-8 content. A UTF-8 page declaring
<meta charset="iso-8859-1">came back as ISO-8859-1, which decodes to mojibake. Valid multi-byte UTF-8 now wins over a conflicting declaration. - Fixed BOM-less UTF-16 byte-order detection for pure-CJK text: short Chinese UTF-16 samples came back with reversed endianness at full confidence. Byte order is now chosen by decoding both ways and comparing text quality.
Full Changelog: 7.5.0...7.5.1
7.5.0
Accuracy and speed release: truncation-proof byte validity, statistical pruning worth ~2.9x, and half the peak memory.
Bug Fixes
- Fixed multi-byte encodings being eliminated when the input ends in an incomplete character. Byte-validity filtering used a one-shot strict decode, which cannot tell a truncated tail from corrupt data, so a single dangling lead byte dropped every CJK candidate and the result came down to input-length parity. Also reachable on complete files through chardet's own
max_bytesand_SCAN_LIMITslicing. Validity checks now decode incrementally withfinal=False. (#376, thanks @aadsm) - Fixed
compat_names(the default) leaking internal Python codec names for seven encodings (ISO-8859-2,ISO-8859-6,ISO-8859-13,Windows-1250,Windows-1256,Windows-1257,CP874). (#374, thanks @aadsm) - Fixed
compat_namesleaking the internalcp932codec name;detect()now returnsCP932. (#375, thanks @uttam12331)
Performance
- Statistical scoring now skips single-byte models that provably can't beat the current runner-up, using per-model row-maximum tables. Results are bit-identical. Mean detection time dropped ~2.9x with mypyc, with the largest gains on legacy CJK (p99 from 9.5ms to 3.3ms).
- Model tables are now
bytes(native array indexing under mypyc) and the model blob is decompressed in chunks: peak process memory dropped from 53.9 to 27.4 MiB. - Confusion-group resolution and post-processing use
bytes.translateprefilters instead of per-byte Python scans.
Improvements
prefer_superset=Trueis now documented as the recommended mode and will become the default in chardet 8.0. Callers that depend on subset names should start passingprefer_superset=Falseexplicitly.chardet.equivalencesis now a deprecation shim: evaluation predicates moved tochardet.evaluation, output-name remapping tochardet.output_names. Removal in 8.0.- Internal pipeline reorganization: language fill, markup-superset promotion, and post-processing moved into their own modules. No behavior change.
Full Changelog: 7.4.3...7.5.0
7.4.3
Patch release: fixes a crash when input contains null bytes inside a <meta charset> declaration.
Bug Fixes
- Fixed
ValueError: embedded null charactercrash when input contained a<meta charset>declaration with a null byte in the encoding name (e.g.b'<meta charset="\x00utf-8">').codecs.lookup()raisesValueErroron embedded nulls, andlookup_encoding()was only catchingLookupError. Also added defensiveValueErrorcatches in_validate_bytes()and_to_utf8()for completeness. (#369, thanks @DRMacIver for the report)
Full Changelog: 7.4.2...7.4.3
7.4.2
Patch release: fixes a crash on short inputs and closes a bunch of WHATWG/IANA alias gaps.
Bug Fixes
- Fixed
RuntimeError: pipeline must always return at least one resulton ~2% of all possible two-byte inputs (e.g.b"\xf9\x92"). Multi-byte encodings like CP932 and Johab could score above the structural confidence threshold on very short inputs, but then statistical scoring would return nothing, leaving an empty result list instead of falling through to the fallback. (#367, #368, thanks @jasonwbarnett)
Improvements
- Added ~90 encoding aliases from the WHATWG Encoding Standard and IANA Character Sets registry so that
<meta charset>labels likex-cp1252,x-sjis,dos-874,csUTF8, and thecswindows*family all resolve correctly through the markup detection stage. Every alias was driven by a failing spec-compliance test, not speculative. (#366) - Added a spec-compliance test suite covering Python decode round-trips for all 86 registry encodings, WHATWG label resolution, IANA preferred MIME names, and Unicode/RFC conformance (BOM sniffing, UTF-8 boundary cases, UTF-16 surrogate pairs). This is the test suite that would have caught the 7.4.1 BOM bug before release. (#366)
Full Changelog: 7.4.1...7.4.2
7.4.1
Bug Fixes
- BOM-prefixed UTF-16/32 input now returns
utf-16/utf-32instead ofutf-16-le/utf-16-be/utf-32-le/utf-32-be. The endian-specific codecs don't strip the BOM on decode, so callers were getting a stray U+FEFF at the start of their text. BOM-less detection is unchanged. (#364, #365)
Full Changelog: 7.4.0...7.4.1
chardet 7.4.0
chardet 7.4.0 brings accuracy up to 99.3% (from 98.6% in 7.3.0) and significantly faster cold start thanks to a new dense model format.
What's New
Performance:
- New dense zlib-compressed model format (v2) drops cold start (import + first detect) from ~75ms to ~13ms with mypyc
Accuracy (98.6% → 99.3%):
- Eliminated train/test data overlap via content fingerprinting
- Added MADLAD-400 and Wikipedia as supplemental training sources
- Improved non-ASCII bigram scoring: high-byte bigrams are now preserved during training and weighted by per-bigram IDF
- Encoding-aware substitution filtering (substitutions only apply for characters the target encoding can't represent)
- Increased training samples from 15K to 25K per language/encoding pair
Bug fixes:
- Added dedicated structural analyzers for CP932, CP949, and Big5-HKSCS (these were previously sharing their base encoding's byte-range analyzer, missing extended ranges)
Metrics
| chardet 7.4.0 (mypyc) | chardet 6.0.0 | charset-normalizer 3.4.6 | |
|---|---|---|---|
| Accuracy (2,517 files) | 99.3% | 88.2% | 85.4% |
| Speed | 551 files/s | 12 files/s | 376 files/s |
| Language detection | 95.7% | 40.0% | 59.2% |
Full changelog: https://chardet.readthedocs.io/en/latest/changelog.html
7.3.0
License
- 0BSD license — the project license has been changed from MIT to 0BSD, a maximally permissive license with no attribution requirement. All prior 7.x releases should also be considered 0BSD licensed as of this release.
Features
- Added
mime_typefield to detection results — identifies file types for both binary (via magic number matching) and text content. Returned in alldetect(),detect_all(), andUniversalDetectorresults. (#350) - New
pipeline/magic.pymodule detects 40+ binary file formats including images, audio/video, archives, documents, executables, and fonts. ZIP-based formats (XLSX, DOCX, JAR, APK, EPUB, wheel, OpenDocument) are distinguished by entry filenames. (#350)
Bug Fixes
- Fixed incorrect equivalence between UTF-16-LE and UTF-16-BE in accuracy testing — these are distinct encodings with different byte order, not interchangeable
Performance
- Added 4 new modules to mypyc compilation (orchestrator, confusion, magic, ascii), bringing the total to 11 compiled modules
- Capped statistical scoring at 16 KB — bigram models converge quickly, so large files no longer score the full 200 KB. Worst-case detection time dropped from 62ms to 26ms with no accuracy loss.
- Replaced
dataclasses.replace()with directDetectionResultconstruction on hot paths, eliminating ~354k function calls per full test suite run
Build
chardet 7.2.0
Features
- Added
include_encodingsandexclude_encodingsparameters todetect(),detect_all(), andUniversalDetector— restrict or exclude specific encodings from the candidate set, with corresponding-i/--include-encodingsand-x/--exclude-encodingsCLI flags (#343) - Added
no_match_encoding(default"cp1252") andempty_input_encoding(default"utf-8") parameters — control which encoding is returned when no candidate survives the pipeline or the input is empty, with corresponding CLI flags (#343) - Added
-l/--languageflag tochardetectCLI — shows the detected language (ISO 639-1 code and English name) alongside the encoding (#342)
Fixes
Full changelog: https://chardet.readthedocs.io/en/latest/changelog.html