train from cache without dataset needs to be updated #128
Description
Describe the bug
Training from cache without dataset (by setting cache.use_cache_without_dataset=true) does no longer work. The reason for this is that the structure of the cache path was updated by putting each scenario-type in a separate folder. When loading scenarios from cache this is not considered so far.
The bug can be fixed by changing this line from
candidate_scenario_dirs = [path for log_dir in cache_dir.iterdir() for path in log_dir.iterdir()]
to
candidate_scenario_dirs = [path for log_dir in cache_dir.iterdir() for type_dir in log_dir.iterdir() for path in type_dir.iterdir()]
Setup
- devkit-0.6
Steps To Reproduce
Steps to reproduce the behavior:
- Train with
cache.use_cache_without_dataset=true
Stack Trace
Error executing job with overrides: ['+training=training_vector_model', 'py_func=train', 'cache.cache_path=/path/to/cache', 'scenario_filter.limit_total_scenarios=0.1', 'data_loader.params.batch_size=16', 'data_loader.params.num_workers=8', 'scenario_builder=nuplan_mini', 'scenario_builder.data_root=/path/to/data', 'lightning.trainer.params.max_epochs=25', 'optimizer.lr=1e-4', 'callbacks=train_callbacks_no_visualization', 'experiment_name=baseline_multi_lgcn', 'group=/path/to/exp/', 'cache.use_cache_without_dataset=true']
Traceback (most recent call last):
File "/home/aah1si/nuplan/nuplan-devkit/nuplan/planning/script/run_training.py", line 57, in main
engine = build_training_engine(cfg, worker)
File "/home/aah1si/nuplan/nuplan-devkit/nuplan/planning/training/experiments/training.py", line 56, in build_training_engine
datamodule = build_lightning_datamodule(cfg, worker, torch_module_wrapper)
File "/home/aah1si/nuplan/nuplan-devkit/nuplan/planning/script/builders/training_builder.py", line 56, in build_lightning_datamodule
scenarios = build_scenarios(cfg, worker, model)
File "/home/aah1si/nuplan/nuplan-devkit/nuplan/planning/script/builders/scenario_builder.py", line 171, in build_scenarios
assert len(scenarios) > 0, 'No scenarios were retrieved for training, check the scenario_filter parameters!'
AssertionError: No scenarios were retrieved for training, check the scenario_filter parameters!```