Instructions to use Sandeep4235/qwen2.5-7b-adapter with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Sandeep4235/qwen2.5-7b-adapter with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Sandeep4235/qwen2.5-7b-adapter") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Sandeep4235/qwen2.5-7b-adapter", device_map="auto") - PEFT
How to use Sandeep4235/qwen2.5-7b-adapter with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Sandeep4235/qwen2.5-7b-adapter with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Sandeep4235/qwen2.5-7b-adapter" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Sandeep4235/qwen2.5-7b-adapter", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Sandeep4235/qwen2.5-7b-adapter
- SGLang
How to use Sandeep4235/qwen2.5-7b-adapter with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Sandeep4235/qwen2.5-7b-adapter" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Sandeep4235/qwen2.5-7b-adapter", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Sandeep4235/qwen2.5-7b-adapter" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Sandeep4235/qwen2.5-7b-adapter", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Unsloth Desktop
- Docker Model Runner
How to use Sandeep4235/qwen2.5-7b-adapter with Docker Model Runner:
docker model run hf.co/Sandeep4235/qwen2.5-7b-adapter
Qwen2.5-7B-Instruct LoRA Adapter
Qwen2.5 7B · LoRA Fine-tuned · Alpaca Instruction Following · Unsloth QLoRA
🚀 Model Summary
Qwen2.5-7B-Instruct LoRA Adapter is a 7.61-billion parameter causal language model fine-tuned (via Unsloth QLoRA) from unsloth/Qwen2.5-7B-Instruct-bnb-4bit to enhance instruction-following capabilities on the Alpaca dataset. The adapter targets all linear projection layers (attention + MLP) with rank-16 LoRA updates, producing sharper, more compliant responses to structured instructions while preserving the base model's broad knowledge and reasoning ability.
Mission. BluePatterns AI built this adapter to demonstrate that efficient, single-GPU fine-tuning (NVIDIA T4, 16 GB VRAM) can produce meaningful instruction-following gains over a strong base model — lowering the barrier for developers who want to customize LLMs without enterprise compute budgets.
💡 Model Architecture & Specifications
This adapter inherits the dense Transformer architecture of Qwen2.5-7B-Instruct and applies a LoRA adapter trained via Unsloth's 4-bit QLoRA pipeline.
| Specification | Detail |
|---|---|
| Developed by | Sandeep Hipparagi (BluePatterns AI) |
| Model type | Causal Language Model (Causal-LM) |
| Base model | unsloth/Qwen2.5-7B-Instruct-bnb-4bit (Qwen2.5-7B-Instruct, 4-bit NF4) |
| Total parameters (base) | 7.61B (6.53B non-embedding) |
| Architecture | Dense Transformer, decoder-only (auto-regressive) |
| Attention | Grouped-Query Attention (GQA) — 28 Q heads, 4 KV heads |
| Layers | 28 |
| Hidden size | 3,584 |
| Intermediate (FFN) size | 18,944 |
| Head dimension | 128 |
| Positional encoding | RoPE (base frequency 1,000,000) |
| Normalization | RMSNorm (pre-normalization) |
| Activation | SwiGLU |
| QKV bias | Yes (attention QKV bias retained from Qwen2) |
| Vocabulary size | 151,646 |
| Embedding tying | No |
| Native context length | 32,768 tokens (up to 131,072 with YaRN) |
| Max generation length | 8,192 tokens |
| Fine-tuning method | Unsloth QLoRA (4-bit NF4 quantization) |
| LoRA rank (r) | 16 |
| LoRA alpha | 16 |
| LoRA dropout | 0 |
| Bias | none |
| Adapter target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Training dataset | yahma/alpaca-cleaned |
| Optimizer | AdamW 8-bit |
| Learning rate | 2e-4 |
| Batch size (per device) | 2 |
| Gradient accumulation steps | 4 (effective batch size = 8) |
| Max sequence length (training) | 2,048 tokens |
| Training precision | FP16 (T4 does not natively support bf16) |
| Gradient checkpointing | Enabled (Unsloth optimized) |
| Hardware | NVIDIA T4 GPU (16 GB VRAM) |
| Language(s) | English |
| License | Apache 2.0 |
🛠 Quickstart & Usage
Option A — Load and infer with transformers + peft (recommended)
This approach loads the 4-bit quantized base model and applies the LoRA adapter from this repository.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model_id = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit"
adapter_id = "Sandeep4235/qwen2.5-7b-adapter"
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
# Load base model in 4-bit with automatic device placement
model = AutoModelForCausalLM.from_pretrained(
base_model_id,
device_map="auto",
torch_dtype=torch.float16,
)
# Apply the LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()
# Build the prompt using Qwen2.5's ChatML chat template
messages = [
{"role": "system", "content": "You are a helpful, expert AI assistant."},
{"role": "user", "content": "Explain the concept of quantum entanglement in simple terms."},
]
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
# Generate
with torch.no_grad():
output_ids = model.generate(
input_ids,
max_new_tokens=512,
do_sample=True,
temperature=0.7,
top_p=0.8,
pad_token_id=tokenizer.eos_token_id,
)
generated = output_ids[0][input_ids.shape[-1]:]
response = tokenizer.decode(generated, skip_special_tokens=True)
print(response)
Option B — Load and infer with Unsloth (fastest inference)
import torch
from unsloth import FastLanguageModel
from transformers import TextStreamer
base_model_id = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit"
adapter_id = "Sandeep4235/qwen2.5-7b-adapter"
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=base_model_id,
max_seq_length=2048,
dtype=None, # auto-detects FP16 on T4
load_in_4bit=True,
)
# Load the fine-tuned adapter weights
model.load_adapter(adapter_id)
FastLanguageModel.for_inference(model) # enable Unsloth's faster inference
messages = [
{"role": "system", "content": "You are a helpful, expert AI assistant."},
{"role": "user", "content": "Write a Python function to reverse a linked list."},
]
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
).to("cuda")
text_streamer = TextStreamer(tokenizer, skip_prompt=True)
_ = model.generate(
input_ids=input_ids,
streamer=text_streamer,
max_new_tokens=512,
use_cache=True,
temperature=0.7,
min_p=0.1,
)
Option C — Merge adapter into base model for standalone deployment
If you want a single standalone model (no adapter loading at runtime), merge the LoRA weights into the base model and save:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model_id = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit"
adapter_id = "Sandeep4235/qwen2.5-7b-adapter"
save_path = "./qwen2.5-7b-merged"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
base_model_id,
device_map="auto",
torch_dtype=torch.float16,
)
# Apply adapter, then merge into base weights
model = PeftModel.from_pretrained(model, adapter_id)
model = model.merge_and_unload()
# Save the merged model
model.save_pretrained(save_path)
tokenizer.save_pretrained(save_path)
print(f"Merged model saved to {save_path}")
📊 Evaluation and Benchmarks
Formal quantitative benchmarks for this adapter (e.g., instruction-following accuracy, MT-Bench, or AlpacaEval scores against the unadapted base) have not yet been published. Performance was assessed through qualitative evaluation during fine-tuning, focusing on response quality, instruction adherence, and format compliance on held-out Alpaca samples.
General Academic Benchmarks (inherited from base)
The following scores reflect the Qwen2.5-7B-Instruct base model as published by the Qwen team and are provided for reference only — they are not measurements of this fine-tuned adapter.
| Benchmark | Qwen2.5-7B-Instruct (base) | Metric |
|---|---|---|
| MMLU-Pro | 56.3 | Pass@1 accuracy |
| MMLU-redux | 75.4 | Pass@1 accuracy |
| GSM8K | 91.6 | Pass@1 accuracy |
| HumanEval | 84.8 | Pass@1 |
| MBPP | 79.2 | Pass@1 |
| IFeval (strict-prompt) | 71.2 | Accuracy |
These figures are the base model's published results and do not represent this adapter's fine-tuned performance. Independent benchmarking is encouraged.
⚠️ Intended Uses & Limitations
Primary Use Cases
- Instruction following — executing structured prompts in the Alpaca format (
### Instruction:/### Response:) with improved compliance over the base model. - Task-oriented text generation — answering questions, writing code, summarizing, and explaining concepts in response to natural-language instructions.
- Rapid prototyping — serving as a starting point for further domain-specific fine-tuning on top of a strong, efficient base.
- Single-GPU customization — demonstrating viable QLoRA fine-tuning on a single T4 GPU for developers with limited compute access.
Out-of-Scope Uses
- Safety-critical decision-making. Do not use for medical diagnosis, legal advice, financial trading, or any automated system where outputs directly affect individual rights or safety.
- High-stakes factual lookup. The model may hallucinate facts, citations, or code. Always verify critical information against authoritative sources.
- Multilingual tasks beyond English. Training data (Alpaca-cleaned) is English-only; performance in other languages is untested.
- Long-context reasoning beyond 2,048 tokens (training limit). While the base model supports up to 128K context, the adapter was trained at 2,048 tokens. Quality may degrade on inputs significantly longer than the training sequence length.
Limitations & Biases
- Alpaca dataset biases. The Alpaca dataset reflects the biases of its generation pipeline (originally distilled from
text-davinci-003). Responses may inherit stylistic or topical biases from this distribution. - Hallucination risk. Like all LLMs, the model may generate plausible but incorrect information. Always verify factual claims.
- Limited training scope. The adapter was trained for a modest number of steps on a general-purpose instruction dataset. It is not a domain expert and should not be expected to match specialized models on specific tasks.
- FP16 precision. Training was done in FP16 (T4 hardware limitation) rather than bf16, which may introduce minor numerical differences compared to bf16-trained adapters.
- Quantization loss. The 4-bit NF4 base quantization introduces minor quality degradation compared to full-precision weights. Use the merged full-precision model for maximum quality.
🔒 Responsible AI & Safety Alignment
Alignment Techniques
- Supervised Fine-Tuning (SFT). The adapter was trained on the Alpaca-cleaned dataset using
trl.SFTTrainer. No RLHF or DPO was applied in this release. - Inherited safety alignment. The base model (
Qwen2.5-7B-Instruct) already undergoes extensive safety alignment during its post-training stage. This adapter preserves that alignment while adjusting instruction-following behavior. - ChatML template compliance. The adapter is designed to work with Qwen2.5's native ChatML chat template, ensuring consistent formatting and predictable behavior.
Deployment Recommendations
- Input validation. Sanitize all user inputs. Reject prompt-injection attempts and malicious instruction patterns.
- Output audit. Review generated content for accuracy and appropriateness before use in any public-facing context.
- Content filtering. Pair the model with an input/output content filter (e.g., Llama Guard, NeMo Guardrails) for production deployments.
- Rate limiting. Deploy behind an API gateway with authentication and rate limiting.
- Context length management. While the base model supports 128K context, keep inference inputs within the 2,048-token training window for best quality, or fine-tune with longer sequences for extended-context use cases.
🏋️ Training Details
Training Data
The adapter was fine-tuned on yahma/alpaca-cleaned, a cleaned and deduplicated version of the Stanford Alpaca dataset. The dataset contains ~52,000 instruction-response pairs covering a wide range of tasks including question answering, code generation, creative writing, and summarization.
Training Environment
| Parameter | Value |
|---|---|
| Hardware | NVIDIA T4 GPU (16 GB VRAM) |
| Cloud provider | Google Colab |
| Fine-tuning framework | Unsloth + trl.SFTTrainer |
| Deep learning framework | PyTorch (FP16) |
| Gradient checkpointing | Enabled (Unsloth optimized) |
🤝 Citation & Acknowledgements
If you use this adapter in your research or product, please cite it as follows:
@misc{hipparagi2025qwen25adapter,
title = {Qwen2.5-7B-Instruct LoRA Adapter: Efficient Instruction-Following Fine-tuning via QLoRA on a Single T4 GPU},
author = {Sandeep Hipparagi},
organization = {BluePatterns AI},
year = {2025},
url = {https://huggingface.co/Sandeep4235/qwen2.5-7b-adapter},
note = {Fine-tuned from unsloth/Qwen2.5-7B-Instruct-bnb-4bit on the Alpaca-cleaned dataset under Apache 2.0}
}
Acknowledgements
- Qwen Team — for open-sourcing the Qwen2.5-7B-Instruct model under Apache 2.0.
- Unsloth — for the efficient QLoRA fine-tuning framework that made single-GPU training viable.
- Stanford Alpaca / yahma — for the cleaned Alpaca instruction dataset.
- Hugging Face — for model hosting and the open ML ecosystem.
📬 Contact
Sandeep Hipparagi — AI Developer, Co-Founder of BluePatterns AI
- Hugging Face: Sandeep4235
- Organization: BluePatterns AI
BluePatterns AI is focused on bridging the gap between frontier and open-source models — democratizing access to reliable, accessible intelligence for diverse communities.
Built by Sandeep Hipparagi · BluePatterns AI · Apache 2.0
Model tree for Sandeep4235/qwen2.5-7b-adapter
Base model
Qwen/Qwen2.5-7B