Skip to content

Second forward call of a compiled model (exact same input shapes, strides) is extremely slow due to cuda graphs #120309

Closed
Closed
Second forward call of a compiled model (exact same input shapes, strides) is extremely slow due to cuda graphs#120309
Labels
module: cuda graphsAbility to capture and then replay streams of CUDA kernelsmodule: dynamic shapesoncall: pt2

Description

@fxmarty
fxmarty
opened on Feb 21, 2024

🐛 Describe the bug

When using torch.compile(..., mode="reduce-overhead") on CUDA device, the second forward call with exact same input shapes, strides, device, dtype is extremely slow, with 0% GPU usage and 100% CPU usage.

When using ctrl+C (not an error, just forcing exit), we see that PyTorch is spending time in cudagraph self._record(wrapped_function.model, recording_inputs):

  File "/home/felix/transformers/src/transformers/models/llama/modeling_llama.py", line 992, in forward
    causal_mask = self._update_causal_mask(attention_mask, inputs_embeds)
  File "/home/felix/transformers/src/transformers/models/llama/modeling_llama.py", line 992, in torch_dynamo_resume_in_forward_at_992
    causal_mask = self._update_causal_mask(attention_mask, inputs_embeds)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 455, in _fn
    return fn(*args, **kwargs)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_dynamo/external_utils.py", line 25, in inner
    return fn(*args, **kwargs)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 893, in forward
    return compiled_fn(full_args)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/utils.py", line 79, in g
    return f(*args)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 101, in runtime_wrapper
    all_outs = call_func_at_runtime_with_args(
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/utils.py", line 103, in call_func_at_runtime_with_args
    out = normalize_as_list(f(args))
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 118, in rng_functionalization_wrapper
    return compiled_fw(args)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 825, in __call__
    return self.get_current_callable()(inputs)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 818, in run
    return compiled_fn(new_inputs)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/cudagraph_trees.py", line 370, in deferred_cudagraphify
    return fn(inputs)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 764, in run
    return model(new_inputs)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/cudagraph_trees.py", line 1757, in run
    out = self._run(new_inputs, function_id)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/cudagraph_trees.py", line 1831, in _run
    return self.record_function(new_inputs, function_id)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/cudagraph_trees.py", line 1862, in record_function
    node = CUDAGraphNode(
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/cudagraph_trees.py", line 902, in __init__
    ] = self._record(wrapped_function.model, recording_inputs)
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/cudagraph_trees.py", line 1094, in _record
    with preserve_rng_state(), torch.cuda.device(
  File "/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/cuda/graphs.py", line 172, in __enter__
    gc.collect()

This is surprising that this happens at the second forward call.

pip3 uninstall torch torchvision torchaudio triton pytorch-triton transformers accelerate
pip3 install --pre torch==2.3.0.dev20240219+cu121 pytorch-triton==3.0.0+901819d2b6 --index-url https://download.pytorch.org/whl/nightly/cu121

git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout --track origin/repro-bug-pytorch-compile-cudagraph
pip install -e .

and then

from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
import torch
from transformers.cache_utils import StaticCache
import time

tokenizer = AutoTokenizer.from_pretrained(
    "NousResearch/Llama-2-7b-chat-hf", padding_side="left", pad_token="<s>"
)

with torch.device("cuda"):
    model = AutoModelForCausalLM.from_pretrained(
        "NousResearch/Llama-2-7b-chat-hf",
        torch_dtype=torch.float16,
        attn_implementation="sdpa",
    )

inputs = tokenizer(
    ["I would", "Today I am in Paris and"], padding=True, return_tensors="pt"
).to(model.device)

print("call generate")

n_runs = 3
new_tokens = 10
gen_config = GenerationConfig(
    max_new_tokens=new_tokens,
    min_new_tokens=new_tokens,
    use_cache=True,
    pad_token_id=tokenizer.pad_token_id,
    num_beams=1,
    do_sample=False,
    eos_token_id=None,  # This is required for min_new_tokens to actually have an effect.
)
model.generation_config.eos_token_id = None  # greedy_search falls back on this eos_token_id that we need to set to None as well for min_new_tokens to have an effect.

gen_out = model.generate(**inputs, generation_config=gen_config)

decoded = tokenizer.batch_decode(gen_out, skip_special_tokens=True)

print("decoded original", decoded)

print("-------------- STATIC CACHE")

import logging
#torch._logging.set_logs(dynamo=logging.INFO, aot=logging.INFO, inductor=logging.INFO, graph_breaks=True, guards=True, recompiles=True, output_code=True, graph_code=True, graph=True)

print("compiling...")

torch.cuda.synchronize()

start = time.time_ns()
model.forward = torch.compile(model.forward, mode="reduce-overhead")

torch.cuda.synchronize()
end = time.time_ns()

latency_compile_ms = (end - start) * 1e-6
print(f"torch.compile call: {latency_compile_ms:.3f} ms")

for i in range(n_runs):
    torch.cuda.synchronize()
    start = time.time_ns()
    gen_out = model.generate(**inputs, generation_config=gen_config, cache_implementation="static")
    torch.cuda.synchronize()
    end = time.time_ns()

    latency_ms = (end - start) * 1e-6
    print(f"\n- {i}-th `generate` call latency per token (new_tokens={gen_config.min_new_tokens}): {latency_ms / gen_config.min_new_tokens:.3f} ms")

decoded = tokenizer.batch_decode(gen_out, skip_special_tokens=True)

print("decoded static", decoded)

Giving:

--------- WITHOUT TORCH.COMPILE
----- in forward 0
name=input_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([7]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
forward call latency: 173.107 ms
----- in forward 1
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 8]), stride=(8, 1), dtype=torch.int64, device=cuda:0
forward call latency: 33.231 ms
----- in forward 2
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 9]), stride=(9, 1), dtype=torch.int64, device=cuda:0
forward call latency: 30.570 ms
----- in forward 3
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 10]), stride=(10, 1), dtype=torch.int64, device=cuda:0
forward call latency: 30.615 ms
----- in forward 4
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 11]), stride=(11, 1), dtype=torch.int64, device=cuda:0
forward call latency: 30.987 ms
----- in forward 5
...

--------- WITH TORCH.COMPILE
----- in forward 0
name=input_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([7]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
/home/felix/miniconda3/envs/fx/lib/python3.9/site-packages/torch/_inductor/compile_fx.py:148: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
forward call latency: 30207.690 ms
----- in forward 1
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 8]), stride=(8, 1), dtype=torch.int64, device=cuda:0
forward call latency: 27293.830 ms
----- in forward 2
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 9]), stride=(9, 1), dtype=torch.int64, device=cuda:0
forward call latency: 1484.173 ms
----- in forward 3
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 10]), stride=(10, 1), dtype=torch.int64, device=cuda:0
forward call latency: 20.806 ms
----- in forward 4
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 11]), stride=(11, 1), dtype=torch.int64, device=cuda:0
forward call latency: 20.412 ms
----- in forward 5
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 12]), stride=(12, 1), dtype=torch.int64, device=cuda:0
forward call latency: 20.866 ms
----- in forward 6
...

- 0-th `generate` call latency per token (new_tokens=10): 5913.885 ms


----- in forward 0
name=input_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([7]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
forward call latency: 1784.737 ms      <---------------------------- EXTREMELY SLOW.
----- in forward 1
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 8]), stride=(8, 1), dtype=torch.int64, device=cuda:0
forward call latency: 1851.579 ms      <---------------------------- EXTREMELY SLOW.
----- in forward 2
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 9]), stride=(9, 1), dtype=torch.int64, device=cuda:0
forward call latency: 1421.504 ms      <---------------------------- EXTREMELY SLOW.
----- in forward 3
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 10]), stride=(10, 1), dtype=torch.int64, device=cuda:0
forward call latency: 1740.283 ms      <---------------------------- EXTREMELY SLOW.
----- in forward 4
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 11]), stride=(11, 1), dtype=torch.int64, device=cuda:0
forward call latency: 1948.687 ms      <---------------------------- EXTREMELY SLOW.
----- in forward 5
...

- 1-th `generate` call latency per token (new_tokens=10): 1727.494 ms


----- in forward 0
name=input_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([7]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 7]), stride=(7, 1), dtype=torch.int64, device=cuda:0
forward call latency: 11.576 ms
----- in forward 1
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 8]), stride=(8, 1), dtype=torch.int64, device=cuda:0
forward call latency: 10.835 ms
----- in forward 2
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 9]), stride=(9, 1), dtype=torch.int64, device=cuda:0
forward call latency: 10.708 ms
----- in forward 3
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 10]), stride=(10, 1), dtype=torch.int64, device=cuda:0
forward call latency: 10.669 ms
----- in forward 4
name=input_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=position_ids, shape=torch.Size([2, 1]), stride=(1, 1), dtype=torch.int64, device=cuda:0
name=cache_position, shape=torch.Size([1]), stride=(1,), dtype=torch.int64, device=cuda:0
name=past_key_values, value=None
name=use_cache, value=True
name=attention_mask, shape=torch.Size([2, 11]), stride=(11, 1), dtype=torch.int64, device=cuda:0
forward call latency: 10.681 ms
----- in forward 5
...

The log comes from https://github.com/huggingface/transformers/compare/a8c4e1036ac6f0f78e512235cf42c17c7d3cc762...repro-bug-pytorch-compile-cudagraph?expand=1

A potential solution is to use @torch.compiler.disable on the _update_causal_mask method. This removes logs as

I0220 11:30:57.881897 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 13
I0220 11:30:57.902060 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 14
I0220 11:30:57.922157 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 15
I0220 11:30:57.942382 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 16
I0220 11:30:57.962995 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 17
I0220 11:30:57.983414 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 18
I0220 11:30:58.004108 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 19
I0220 11:30:58.024602 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 20
I0220 11:30:58.045034 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 21
I0220 11:30:58.065743 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 22
I0220 11:30:58.086160 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 23
I0220 11:30:58.106957 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 24
I0220 11:30:58.127666 140023118176640 torch/_inductor/cudagraph_trees.py:375] recording cudagraph tree for symint key 25

in inductor logs. But it is not a perfect solution either as then fullgraph=True can not be used.

It is quite surprising to me that the second forward call is slow. To me only the first should be.

Versions

Collecting environment information...
PyTorch version: 2.3.0.dev20240219+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.24.3
Libc version: glibc-2.31

Python version: 3.9.13 (main, Oct 13 2022, 21:15:33)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.3.52
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA DGX Display
GPU 4: NVIDIA A100-SXM4-80GB

Nvidia driver version: 535.129.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      43 bits physical, 48 bits virtual
CPU(s):                             128
On-line CPU(s) list:                0-127
Thread(s) per core:                 2
Core(s) per socket:                 64
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          AuthenticAMD
CPU family:                         23
Model:                              49
Model name:                         AMD EPYC 7742 64-Core Processor
Stepping:                           0
Frequency boost:                    enabled
CPU MHz:                            2925.932
CPU max MHz:                        2250,0000
CPU min MHz:                        1500,0000
BogoMIPS:                           4491.21
Virtualization:                     AMD-V
L1d cache:                          2 MiB
L1i cache:                          2 MiB
L2 cache:                           32 MiB
L3 cache:                           256 MiB
NUMA node0 CPU(s):                  0-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Vulnerable
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca sme sev sev_es

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.4
[pip3] onnx==1.15.0
[pip3] onnxruntime==1.16.1
[pip3] pytorch-triton==3.0.0+901819d2b6
[pip3] torch==2.3.0.dev20240219+cu121
[pip3] torch-tb-profiler==0.4.0
[pip3] triton==2.2.0
[conda] numpy                     1.23.4                   pypi_0    pypi
[conda] pytorch-triton            3.0.0+901819d2b6          pypi_0    pypi
[conda] torch                     2.3.0.dev20240219+cu121          pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] triton                    2.2.0                    pypi_0    pypi

cc @mcarilli @ezyang @msaroufim @bdhirsh @anijain2305 @zou3519

Metadata

Metadata

Assignees

No one assigned

    Labels

    module: cuda graphsAbility to capture and then replay streams of CUDA kernelsmodule: dynamic shapesoncall: pt2

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions