❓ Questions & Help
I've fined tune distilgpt2 with run_language_modeling.py under my local output_dir and want to fine tune with the model in output_dir again, but it raised torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'config'.
Details
I've fined tune distilgpt2 with run_language_modeling.py as follows:
python3 run_language_modeling.py --output_dir=/home/xxx/gpt_model/transformers/examples/language-modeling/output_dir --model_type=gpt2 --model_name_or_path=distilgpt2 --per_device_train_batch_size=10 --do_train --train_data_file=/home/xxx/gpt_model/data_info/data.txt --block_size=64 --save_steps=100 --overwrite_output_dir
It works fine and i can get the fine-tuned model after training with my data data.txt. Now i want to fine-tune in the output_dir so i run run_language_modeling.py as follows on my new_data.txt:
python3 run_language_modeling.py --output_dir=/home/xxx/gpt_model/transformers/examples/language-modeling/output_new_data --model_type=gpt2 --model_name_or_path=/home/xxx/gpt_model/transformers/examples/language-modeling/output_dir/ --per_device_train_batch_size=20 --do_train --train_data_file=/home/xxx/gpt_model/data_info/new_data.txt --block_size=64 --save_steps=1000 --overwrite_output_dir
But it raise exception and exit. The stderr is as follows
/home/xxx/gpt_model/pytorch/pytorch/torch/optim/lr_scheduler.py:235: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
Traceback (most recent call last):
File "run_language_modeling.py", line 320, in <module>
main()
File "run_language_modeling.py", line 284, in main
trainer.train(model_path=model_path)
File "/home/xxx/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/trainer.py", line 683, in train
self.total_flos = getattr(model.config, "total_flos", 0)
File "/home/xxx/gpt_model/pytorch/pytorch/torch/nn/modules/module.py", line 779, in __getattr__
type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'config'(1) I'm training on multiple GPUs, but i did'n specify a specific gpu to run. So at first i think this may caused by multiple GPUs, and add the following code under
|
model = torch.nn.DataParallel(model) |
if isinstance(model,torch.nn.DataParallel):
model = model.module
It has no error but doesn't work. The output is as follows . After that, the process is killed and return code echo $? is 0.
Epoch: 0it [00:00, ?it/s]
/home/lenajin/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/trainer.py:1087: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)(2) Following with #1991 , I try to train with following script to train with gpu whose id is 1.
CUDA_VISIBLE_DEVICES=1 python3 run_language_modeling.py --output_dir=/home/xxx/gpt_model/transformers/examples/language-modeling/output_dir --model_type=gpt2 --model_name_or_path=distilgpt2 --per_device_train_batch_size=20 --do_train --train_data_file=/home/xxx/gpt_model/data_info/data.txt --block_size=64 --save_steps=100 --overwrite_output_dir
it return with return code echo $? =0. At first i don't know what's the problem, but now i changed --per_device_train_batch_size=20 from 20 to 10. It seems all is well, and now my GPU-Util is about 98%. Maybe it return without training duing to my limit gpu CC？It's wired but at least it works now. Maybe there can give some error message about the return reason?
❓ Questions & Help
I've fined tune distilgpt2 with
run_language_modeling.pyunder my localoutput_dirand want to fine tune with the model inoutput_diragain, but it raisedtorch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'config'.Details
I've fined tune distilgpt2 with
run_language_modeling.pyas follows:It works fine and i can get the fine-tuned model after training with my data
data.txt. Now i want to fine-tune in theoutput_dirso i runrun_language_modeling.pyas follows on mynew_data.txt:But it raise exception and exit. The stderr is as follows
(1) I'm training on multiple GPUs, but i did'n specify a specific gpu to run. So at first i think this may caused by multiple GPUs, and add the following code under
model = torch.nn.DataParallel(model)
transformers/src/transformers/trainer.py
Line 641 in 52d250f
if isinstance(model,torch.nn.DataParallel): model = model.moduleIt has no error but doesn't work. The output is as follows . After that, the process is killed and return code
echo $?is0.(2) Following with #1991 , I try to train with following script to train with gpu whose id is
1.it return with return code
echo $?=0. At first i don't know what's the problem, but now i changed--per_device_train_batch_size=20from 20 to 10. It seems all is well, and now myGPU-Utilis about98%. Maybe it return without training duing to my limit gpu CC？It's wired but at least it works now. Maybe there can give some error message about the return reason?