Hi,
I'm currently fine-tuning llama3-instruct-8b on a custom dataset using unsloth's FastLanguageModel. I'm using Hugging Face's SFTTrainer to train the model. Surprisingly, the gradient norm and evaluation loss become NaN after a few steps. I've seen a blog from unsloth mentioning that NaNs may appear due to a bug, but they also mentioned that the bug was fixed by Hugging Face and unsloth now (here, under the llama3-Quirks section). So, I not only updated unsloth and Hugging Face but also added the "pad_token" mentioned in the blog. Despite these attempts, the NaN problem still persists. Is there something else that I'm missing? Can someone help me out with this?
Here's the code snippet of how I'm loading the model:
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = model_name,
max_seq_length = args.max_seq_length,
dtype = compute_dtype,
load_in_4bit = args.use_4bit,
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)
tokenizer.add_special_tokens({"pad_token": "<|reserved_special_token_0|>"})
model.config.pad_token_id = tokenizer.pad_token_id # updating model config
tokenizer.padding_side = 'right
model = FastLanguageModel.get_peft_model(
model,
r = args.lora_r, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
target_modules = lora_modules,
lora_alpha = args.lora_alpha,
lora_dropout = args.lora_dropout, # Supports any, but = 0 is optimized
bias = args.lora_bias, # Supports any, but = "none" is optimized
# [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
random_state = 3407,
use_rslora = False, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
)Following is the training code:
training_arguments = TrainingArguments(
output_dir=output_dir,
num_train_epochs=args.epochs,
per_device_train_batch_size=args.per_device_train_batch_size,
per_device_eval_batch_size=args.per_device_eval_batch_size,
gradient_accumulation_steps=args.gradient_accumulation_steps,
optim=args.optimizer,
save_steps=args.save_steps,
logging_steps=args.logging_steps,
learning_rate=args.learning_rate,
weight_decay=args.weight_decay,
fp16=fp16,
bf16=bf16,
max_grad_norm=args.max_grad_norm,
max_steps=args.max_steps,
warmup_ratio=args.warmup_ratio,
# group_by_length=args.group_by_length,
lr_scheduler_type=args.lr_scheduler_type,
logging_strategy="steps",
report_to="tensorboard",
evaluation_strategy="steps",
# ddp_find_unused_parameters=False,
)
trainer = SFTTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
dataset_text_field="chats",
max_seq_length=args.max_seq_length,
tokenizer=tokenizer,
args=training_arguments,
packing=packing
)
Hi,
I'm currently fine-tuning llama3-instruct-8b on a custom dataset using unsloth's FastLanguageModel. I'm using Hugging Face's SFTTrainer to train the model. Surprisingly, the gradient norm and evaluation loss become NaN after a few steps. I've seen a blog from unsloth mentioning that NaNs may appear due to a bug, but they also mentioned that the bug was fixed by Hugging Face and unsloth now (here, under the llama3-Quirks section). So, I not only updated unsloth and Hugging Face but also added the "pad_token" mentioned in the blog. Despite these attempts, the NaN problem still persists. Is there something else that I'm missing? Can someone help me out with this?
Here's the code snippet of how I'm loading the model:
Following is the training code: