[CPU]detectron2_fcos_r_50_fpn multiple thread float32 static shape default wrapper eager_two_runs_differ accuracy failure in 2025-03-24 nightly release #150094
Description
🐛 Describe the bug
detectron2_fcos_r_50_fpn multiple thread float32 static shape default wrapper accuracy failure
the bad commit: 842d515
/workspace/pytorch# bash inductor_single_run.sh multiple inference accuracy torchbench detectron2_fcos_r_50_fpn float32
Testing with inductor.
multi-threads testing....
loading model: 0it [00:03, ?it/s]
cpu eval detectron2_fcos_r_50_fpn
WARNING:common:fp64 golden ref were not generated for detectron2_fcos_r_50_fpn. Setting accuracy check to cosine
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
eager_two_runs_differ
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,accuracy,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips,compilation_latency
cpu,detectron2_fcos_r_50_fpn,4,eager_two_runs_differ,0,0,0,0,0,0,0,0
the last good commit: 85f6d61
/workspace/pytorch# bash inductor_single_run.sh multiple inference accuracy torchbench detectron2_fcos_r_50_fpn float32
Testing with inductor.
multi-threads testing....
loading model: 0it [00:03, ?it/s]
cpu eval detectron2_fcos_r_50_fpn
WARNING:common:fp64 golden ref were not generated for detectron2_fcos_r_50_fpn. Setting accuracy check to cosine
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
pass
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,accuracy,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips,compilation_latency
cpu,detectron2_fcos_r_50_fpn,4,pass,944,29,22,4,0,0,22,145.010350
Versions
SW info
| name | target_branch | target_commit | refer_branch | refer_commit |
|---|---|---|---|---|
| torchbench | main | 373ffb19 | main | 373ffb19 |
| torch | main | 621c801 | main | f80bee4 |
| torchvision | main | 0.19.0a0+d23a6e1 | main | 0.19.0a0+d23a6e1 |
| torchtext | main | 0.16.0a0+b0ebddc | main | 0.16.0a0+b0ebddc |
| torchaudio | main | 2.6.0a0+318bace | main | 2.6.0a0+c670ad8 |
| torchdata | main | 0.7.0a0+11bb5b8 | main | 0.7.0a0+11bb5b8 |
| dynamo_benchmarks | main | nightly | main | nightly |
Repro:
inductor_single_run.sh
bash inductor_single_run.sh multiple inference accuracy torchbench detectron2_fcos_r_50_fpn float32
Suspected guilty commit: 842d515
torchbench-detectron2_fcos_r_50_fpn-inference-float32-static-default-multiple-accuracy-crash_guilty_commit.log
cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @chuanqi129