Skip to content

Releases: huggingface/transformers

Release list

Patch release: v5.14.1

Patch release: v5.14.1 Latest
Latest

Choose a tag to compare

View all tags
@Cyrilvallez Cyrilvallez released this 16 Jul 09:41
v5.14.1
a08ace4
This commit was signed with the committer’s verified signature.
Cyrilvallez Cyril Vallez
SSH Key Fingerprint: O743lyKbz6cYCDq5uTRdzyLHAo4JjdtgAF9Q3yVhAQA
Verified
Learn about vigilant mode.

Patch release v5.14.1

This patch solves a few issues which appeared when integrating Inkling model, most notably an issue affecting models using EncoderDecoderCache during assisted generation. It also fixes an issue that could appear during prefill with StaticCache and sdpa without padding for Inkling which uses a position_bias.
It contains the following commits:

Contributors

IlyasMoutawwakil, Cyrilvallez, and vasqu
Assets 2
Loading

Release v5.14.0

Release v5.14.0

Choose a tag to compare

View all tags
@ArthurZucker ArthurZucker released this 15 Jul 19:02
v5.14.0
2fa8c19

Release v5.14.0

New Model additions

Inkling (fresh from Thinking Machines): 975B total, 41B active

image

Inkling is a general-purpose multimodal model that accepts text, image and audio inputs and
generates text outputs. It is intended for use in English and other languages, and across
multiple coding languages. The model is designed to be used by developers building AI-
powered applications, including agentic and tool-use systems, coding assistants, chatbots, and
retrieval-augmented generation systems, and is suitable for general-purpose conversational
use, instruction-following, and other natural language and multimodal tasks. It is released with
open weights to support research, fine-tuning and integration into third-party products by
downstream developers.

TIPSv2

image

Links: Documentation

TIPSv2 DPT

image

Links: Documentation

🚨 Breaking changes

GPTNeoX now remaps embed_out to lm_head and GPTBigCode has _supports_attention_backend = True enabled for vLLM compatibility; users relying on the previous weight naming or attention backend behavior for these models should update their code accordingly.

  • 🚨 Fix GPTBigCode and GPTNeoX for the Transformers modelling backend for vLLM (#47198) by @hmellor

Kernels

Several kernel-related fixes and improvements were made, including pinning the kernels dependency to a compatible version in the benchmark workflow, removing a deprecated package_name argument from LocalLayerRepository, and making the DeepGEMM Triton fallback more robust when CUDA_HOME is unset or misconfigured. Additionally, SDPA prefill was updated to leverage the FlashAttention kernel with StaticCache, yielding significant performance gains (up to 260% faster for large input sizes).

Generation

Generation improvements include adding Multi-Token Prediction (MTP) decoding support, static ensemble verification for speculative decoding to improve draft token acceptance rates, and a fix for crashes in greedy assisted generation with different tokenizers. A misleading double-negative warning message for synced_gpus in continuous batching mode was also corrected.

Performance

Fixed a Flash Attention performance regression affecting models like Qwen3-VL and resolved a MoE decode optimization bug where the grouped-to-batched matrix multiplication switch was not applied to experts residing in submodels (e.g., VLMs with a nested text config).

Cache

Cache dispatch logic was simplified by introducing explicit layer-type mappings for sliding and static layers, reducing complexity in cache routing. Additionally, fixes were made for read-only cache failures in CPU CI environments and for MPS graph cache growth during variable-length batch training on Apple Silicon.

Bugfixes and improvements

Read more

Contributors

tarekziade, andreasgoulas, and 43 other contributors
Loading

Patch release v5.13.1

Patch release v5.13.1

Choose a tag to compare

View all tags
@ArthurZucker ArthurZucker released this 11 Jul 09:15
v5.13.1
4626421

Patch release v5.13.1

This patch is focused on enabling transformers for the latest release of vllm!

  • Be more defensive with remap_legacy_layer_types for custom models (#47245) from @hmellor
  • Fix custom code which doesn't know about the new linear layer type names (#47174) from @hmellor
  • Fix case where _LazyAutoMapping.register is passed a str key (#47148) from @hmellor

Contributors

hmellor
Loading

Release v5.13.0

Release v5.13.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 03 Jul 16:06
v5.13.0
6af945f

Release v5.13.0

New Model additions

KimiK 2.5, 2.6, and 2.7

image

This release includes the architecture for Kimi 2.5 which is used by 2.5-2.7:

Kimi K2.5 is an open-source, native multimodal agentic model that advances practical capabilities in long-horizon coding, coding-driven design, proactive autonomous execution, and swarm-based task orchestration. The model was proposed in Kimi K2.5: Visual Agentic Intelligence and further improved in [Kimi K2.6: Advancing Open-Source Coding](Kimi K2.5: Visual Agentic Intelligence).

Kimi K2.5 achieves significant improvements on complex, end-to-end coding tasks, generalizing robustly across programming languages (Rust, Go, Python) and domains spanning front-end, DevOps, and performance optimization. The model is capable of transforming simple prompts and visual inputs into production-ready interfaces and lightweight full-stack workflows, generating structured layouts, interactive elements, and rich animations with deliberate aesthetic precision.

Links: Documentation

MiMo-V2-Flash

image

MiMo-V2-Flash is a Mixture-of-Experts (MoE) language model developed by the Xiaomi MiMo team. Designed to establish a new balance between long-context modeling capabilities and inference efficiency, the model is built for strong performance in complex reasoning and agentic tasks. Trained on 27T tokens with native 32k sequence lengths, MiMo-V2-Flash seamlessly supports an extended 256K context window while significantly reducing KV-cache storage compared to standard global attention models.

Links: Documentation

Nemotron 3.5 ASR

image

Nemotron 3.5 ASR is a 600M-parameter multilingual speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizes—80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cache-aware FastConformer-RNNT architecture is central to this capability: unlike traditional buffered streaming, which repeatedly reprocesses overlapping audio windows, the model processes only each new incoming chunk while reusing cached encoder context from prior chunks. This eliminates redundant computation, significantly improves efficiency, and minimizes end-to-end delay without sacrificing accuracy, making it well suited to real-time transcription workloads.

Links: Documentation

NemotronAsrStreaming

Nemotron ASR Streaming is a 600M-parameter English speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizes—80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cache-aware FastConformer-RNNT architecture is central to this capability: unlike traditional buffered streaming, which repeatedly reprocesses overlapping audio windows, the model processes only each new incoming chunk while reusing cached encoder context from prior chunks. This eliminates redundant computation, significantly improves efficiency, and minimizes end-to-end delay without sacrificing accuracy, making it well suited to real-time transcription workloads.

Links: Documentation

Qwen3 ASR

image

Qwen3 ASR is an automatic speech recognition model from Alibaba's Qwen team that combines a Whisper-style audio encoder with a Qwen3 language model decoder for speech-to-text transcription. The model supports automatic language detection and multilingual transcription.

A forced aligner model is also included. It can be used to timestamp a provided transcript and its audio. It uses the same audio encoder model with a classification head that predicts a word's length. This model can be used with the transcript from any ASR model (see the example below with Parakeet CTC).

Links: Documentation

ZAYA

image

ZAYA1 is a 760M active / 8.4B total parameter MoE language model trained by Zyphra. It combines Compressed
Convolutional Attention (CCA), a nonlinear ZAYA1 router, and residual scaling.

Links: Documentation

VideoPrism

The VideoPrism model was proposed in the paper VideoPrism: A Foundational Visual Encoder for Video Understanding by Google DeepMind (blog post).

VideoPrism is a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. The model is pretrained on a large-scale heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding through global-local distillation of semantic video embeddings and a token shuffling scheme, enabling the model to focus primarily on the video modality while leveraging text associated with videos. VideoPrism achieves state-of-the-art performance on 31 out of 33 video understanding benchmarks across four broad task groups, from web video question answering to computer vision for science.

Links: Documentation

RADIO

RADIO (Reduce All Domains Into One) is a family of vision foundation models from NVIDIA trained by multi-teacher distillation (e.g. CLIP, DINOv2, SAM) into a single ViT backbone. It produces both an image-level summary embedding and dense spatial features, and supports variable input resolutions through a Cropped Position Embedding (CPE) patch generator.

Links: Documentation

MiniCPM3

MiniCPM3 is the third-generation MiniCPM dense language model from OpenBMB. The 4B variant
(openbmb/MiniCPM3-4B) outperforms many 7B–9B open
models on standard benchmarks while remaining lightweight enough for on-device usage.

MiniCPM3 combines several architectural ideas:

  • Multi-head Latent Attention (MLA) from DeepSeek-V2, which compresses the key/value cache
    into a low-rank latent representation while still using rotary embeddings on a portion of the
    query/key heads.
  • A standard SwiGLU MLP (no MoE).
  • Three scalar scaling factors that govern signal flow:
    • scale_emb — scales input embeddings.
    • scale_depth / sqrt(num_hidden_layers) — scales residual connections.
    • hidden_size / dim_model_base — scales hidden states before the language model head.

Links: Documentation

Breaking changes

A broad set of modeling changes have been made to standardize layer declarations, mask/cache construction, and hybrid-attention handling, making many models cleanly exportable (ONNX, torch.export, ExecuTorch) and fullgraph-compilable — users relying on internal modeling APIs may need to update their code accordingly.

Attention masking for image tokens in Gemma 3/4 models has been fixed to correctly respect sliding window boundaries in local layers, which changes model behavior and may affect reproducibility of previous results.

  • 🚨 [gemma 3/4] Fix bidirectional attention masking crossing sliding window boundaries (#46850) by @douglas-reid

The Expert Parallelism (E...

Read more

Contributors

kurtmckee, tarekziade, and 68 other contributors
Loading

Patch release v5.12.1

Patch release v5.12.1

Choose a tag to compare

View all tags
@vasqu vasqu released this 15 Jun 17:29
v5.12.1
ddb849a

Patch release v5.12.1

Updated the lower bound for PEFT and a fix for auto tokenizer to properly resolve the mistral tokenizer (when mistral-common is installed). This is similar to v.5.10.3 minus the fixes that were already included in the main release - vLLM will first target 5.10.3 🤗

Full Changelog: v5.12.0...v5.12.1

Contributors

hmellor and itazap
Loading

Patch release v5.10.4

Patch release v5.10.4

Choose a tag to compare

View all tags
@vasqu vasqu released this 15 Jun 17:29
v5.10.3
89eb876

Patch release v5.10.4

Update: Note that on pypi 5.10.3 doesn't exist and this this saved under 5.10.4 (so essentially a minor version skipped). Sorry about that, that's on me. Just wanted to clarify to make this less confusing!

A few fixes needed for vLLM to sync with transformers 🤗

Full Changelog: v5.10.2...v5.10.3

Contributors

hmellor, itazap, and 2 other contributors
Loading

Release v5.12.0

Release v5.12.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 12 Jun 14:39
v5.12.0
e0e7504

Release v5.12.0

New Model additions

MiniMax-M3-VL

image

MiniMax-M3-VL is the vision-language member of the MiniMax-M3 family that pairs a CLIP-style vision tower with 3D rotary position embeddings with the MiniMax-M3 text backbone. It uses a mixed dense/sparse Mixture-of-Experts decoder with SwiGLU-OAI gated experts and a lightning indexer for block-sparse attention. The model processes images through a Conv3d patch embedding system and includes specialized components for efficient multimodal understanding and generation.

Links: Documentation

PP-OCRv6: update documentation and slow tests (#46576)

image

The official weights for PP-OCRv6 are out: PP-OCRv6 is a lightweight OCR system that combines architectural innovation with data-centric optimization. It redesigns the backbone, detection neck, and recognition neck around a unified MetaFormer-style building block with structural reparameterization. Three model tiers (medium, small, tiny) share the same block primitives, covering deployment scenarios from server to edge.

  • PP-OCRv6: update documentation and slow tests (#46576) by @ zhang-prog

Add Parakeet-RNNT (#46331)

ParakeetForRNNT: a Fast Conformer Encoder + an RNN-T (RNN Transducer) decoder

  • RNN-T Decoder: Standard neural transducer:
    • LSTM prediction network maintains language context across token predictions.
      • Joint network combines encoder and decoder outputs.
      • Greedy transducer decoding for inference: a blank emission advances the encoder frame by one, a non-blank emission stays on the same frame.

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

Contributors

kashif, tarekziade, and 14 other contributors
Loading

Release v5.11.0

Release v5.11.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 10 Jun 16:32
v5.11.0
e7b5b96

Release v5.11.0

New Model additions

DiffusionGemma

image

DiffusionGemma is engineered to reduce the sequential bottlenecks of standard causal language models by employing an encoder-decoder architecture specifically optimized for inference speed. During inference, DiffusionGemma leverages multi-canvas sampling, where rather than generating one token at a time, the model iteratively denoises a full block of tokens using a diffusion sampler. This block-autoregressive approach facilitates text generation at higher speeds compared to traditional sequential generation methods.

Links: Documentation

DeepSeek-V3.2

image

DeepSeek-V3.2-Exp is an experimental model from DeepSeek-AI that introduces DeepSeek Sparse Attention (DSA), a trainable, fine-grained sparse attention mechanism designed to improve training and inference efficiency in long-context scenarios. Built on top of DeepSeek-V3.1-Terminus with a 685B-parameter Mixture-of-Experts backbone, it reduces the quadratic cost of attention over long sequences by attending only to a selected subset of past tokens while maintaining virtually identical benchmark performance. The work was extended in DeepSeek-V3.2 which pairs DSA with scalable reinforcement learning and achieves gold-medal level results on competition math and competitive programming benchmarks.

Links: Documentation | Paper

Kernels

The KernelConfig API was extended to support n-to-1 module fusion and parameter transformation, simplifying how custom kernels are integrated with Transformers modules. Additional fixes include resolving a dtype mismatch in the Mamba2 CUDA kernel path for NemotronH/Zamba2, adding fine-grained fp8/fp4 Triton kernel support, and correcting the FalconMamba fast-path warning to recommend pip install kernels instead of mamba-ssm.

Parallelization

Fixed model parallel beam search bugs in the Qwen2-VL, Qwen2.5-VL, and Qwen3-VL MoE model families, and added documentation for tensor parallelism support with continuous batching.

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

Contributors

NicolasHug, csantosbh, and 33 other contributors
Loading

Patch release v5.10.2

Patch release v5.10.2

Choose a tag to compare

View all tags
@vasqu vasqu released this 04 Jun 18:43
v5.10.2
0dad7b8

Patch release v5.10.2

There was a big bug in the model conversion of models related to clip, this affected models like sam3 and others. Please make sure to update 🙏

Full Changelog: v5.10.1...v5.10.2

Contributors

zucchini-nlp
Loading

Release v5.10.1

Release v5.10.1

Choose a tag to compare

View all tags
@ArthurZucker ArthurZucker released this 03 Jun 15:37
v5.10.1
90c3ae5

Release v5.10.1

v5.10.0 was yanked as we publish on a corrupted branch. Sorry everyone, this happens when we rush a release!!!

New Model additions

Gemma4 unified+ Gemma4 MTP

image

Gemma 4 12B Unified is an encoder-free multimodal model with pretrained and instruction-tuned variants. Unlike standard Gemma 4, which uses dedicated encoder towers, Gemma 4 12B Unified projects raw inputs directly into the language model's embedding space through lightweight linear pipelines. This results in a simpler architecture while maintaining strong multimodal performance.

Key differences from standard Gemma 4:

  • No Vision Tower: Raw pixel patches are projected directly into LM space via a Dense + LayerNorm pipeline with factorized 2D positional embeddings, replacing the vision encoder.
  • No Audio Tower: Raw 16 kHz waveform samples are chunked into fixed-length frames and projected through a simple RMSNorm → Linear pipeline, replacing the mel spectrogram + Conformer encoder.
  • Shared Multimodal Pipeline: Both vision and audio use the same Gemma4UnifiedMultimodalEmbedder (RMSNorm → Linear) for the final projection to text hidden space.

You can find the original Gemma 4 12B Unified checkpoints under the Gemma 4 release.

Sapiens2

Sapiens2 is a family of high-resolution vision transformers pretrained on ~1 billion curated human images, designed for human-centric computer vision tasks including pose estimation, body-part segmentation, surface normal estimation, and pointmap estimation. The models scale from 0.4B to 5B parameters and train at native 1K resolution, with hierarchical 4K variants for extended spatial reasoning. Sapiens2 achieves substantial improvements over its predecessor with +4 mAP in pose estimation, +24.3 mIoU in body-part segmentation, and 45.6% error reduction in normal estimation.

Links: Documentation | Paper

DeepSeek-OCR-2

DeepSeek-OCR-2 is an OCR-specialized vision-language model built on a distinctive architecture that combines a SAM ViT-B vision encoder with a Qwen2 hybrid attention encoder, connected through an MLP projector to a DeepSeek-V2 Mixture-of-Experts (MoE) language model. The model features a hybrid attention mechanism that applies bidirectional attention over image tokens and causal attention over query tokens, enabling efficient and accurate document understanding. It supports both plain OCR tasks and grounding capabilities with coordinate-aware output for document conversion to markdown format.

Links: Documentation

Mellum

Mellum is a code-focused Mixture-of-Experts language model developed by JetBrains. It is derived from the Qwen3-MoE architecture with per-layer-type RoPE and interleaved sliding window attention. The model has 12B total parameters with 2.5B active parameters per token, using 64 routed experts with 8 activated per token across 28 layers.

Links: Documentation

Breaking changes

The Gemma4 vision pooler now casts inputs to float32 before scaling to prevent float16 overflow (inf saturation) with large checkpoints, which may cause minor numerical differences in outputs for users running Gemma-4 vision models in float16.

Audio Language Models (ALMs) now have a dedicated base model class without a language modeling head, aligning them with the design of Vision Language Models (VLMs); users relying on the previous model class structure should update their code to use the new base model class where appropriate.

Parallelization

This release includes numerous bug fixes for model parallelism across multiple models (Gemma4, AltCLIP, ChineseClip, Blip-2, Whisper, Ovis2, Moshi) and parallel execution strategies, including fixes for tensor parallelism (TP), expert parallelism (EP), beam search under model parallel settings, and loss over-counting under TP/EP configurations. The continuous batching manager was also reworked for clearer control flow and improved TP race condition handling, and FSDP initialization via from_pretrained was introduced.

Cache

Fixed a regression in encoder-decoder cache initialization where the decoder config was incorrectly applied to the cross-attention cache, and resolved a RuntimeError caused by buffer size limits when warming up the cache on MPS devices. Additional test infrastructure improvements were made to support read-only cache environments used in CI.

Quantization

Added support for DeepGEMM BF16, mixed FP8/FP4, and MegaMoE quantization via a grouped linear refactor, while fixing two bugs: an FP8 MoE reverse substring issue affecting DSv4 initialization, and a BitsAndBytes 4-bit/8-bit quantization bug that silently dropped chunked tensors from one-to-many weight converters.

Bugfixes and improvements

  • Fix wrong changes produced by style/repo. check bot (#46371) by @ydshieh in [#46371]
  • Fix path traversal when saving Bark voice preset embeddings (#46237) by @LinZiyuu in [#46237]
  • Pass library_name/version to Hub calls via a shared HfApi (#46318) by @Wauplin in [#46318]
  • docs: update ACL Anthology URL in CITATION.cff (#46352) by @irfaan101 in [#46352]
  • [docs] contributing (#45465) by @stevhliu in [#45465]
  • [docs] Romanian translation of contributing.md, modular_transformers.md, multimodal_processing.md, add_vision_processing_components.md, add_audio_processing_components.md, modeling_rules.md, model_output_tracing.md, auto_docstring.md, testing.md, pr_checks.md and add_new_model.md . (#46345) by @filipinescu in [#46345]
  • [docs] xpu continuous batching (#46334) by @stevhliu in [#46334]
  • Fix incorrect attribute mapping relationships in GLM MoE DSA Config (#46338) by @Dovis01 in [#46338]
  • Fix grammar typos in Whisper documentation (#46336) by @calliec-1223 in [#46336]
  • [docs] update num_items_in_batch for causal LMs (#46335) by @stevhliu in [#46335]
  • Update compressed tensors minimum version (#46342) by @SunMarc in [#46342]
  • Fix _is_package_available reporting available without a version (#46125) by @blipbyte in [#46125]
  • remove sec (#46346) by @ydshieh in [#46346]
  • fix: include transitive relative imports when loading from local directory (#46022) by @trducng in [#46022]
  • perf(feature_extraction_sequence): skip re-splitting already-batched numpy arrays in pad() (#46329) by @Anai-Guo in [#46329]
  • [Zamba] Support attn_implementation dispatch (#46317) by @YangKai0616 in [#46317]
  • Fix TestAppRoutes test failures caused by deprecated asyncio.get_event_loop() on Python 3.10+ (#46340) by @ydshieh in [#46340]
  • [Qwen3VL] Fix video token placeholder: use self.video_token instead of hardcoded "<|placeholder|>" (#46296) by @kpal002 in [#46296]
  • chore(linter): fixes for rule 16 (#46023) by @tarekziade in [#46023]
  • [docs] Romanian translation of weightconverter.md, models.md, custom_models.md, monkey_patching.md, fusion_mapping.md, how_to_hack_models.md, model_sharing.md and serialization.md. (#46309) by @filipinescu in [#46309]
  • Normalize CUDA OOM errors when comparing commit failures in check_bad_commit (#46322) by @ydshieh in [#46322]
  • Fix unhandled exception noise from background safetensors conversion thread (#45752) by @dhruv7477 in [#45752]
  • Add Expectations for pipeline token classification tests (#46151) by @ka...
Read more

Contributors

shadeMe, tarekziade, and 53 other contributors
Loading