recompilation happens and only Aten-level backends are called without real inputs). As we use dummy data to materialize tensors, this solution might still not work but improves the coverage.

@tohtana, what are the known failure cases of this solution?

@tohtana, what are the known failure cases of this solution?

@sfc-gh-truwase I can think of these two cases:

• Operators that take indices (e.g. embedding, scatter): They will throw an error if indices that exceed the target tensor size. In this PR, we fill the dummy value with 1 (like ones). Perhaps embedding should be okay for most of cases. If we encounter this issue, it would be good to add an option to control more detail behaviors (e.g. saving only int tensors expecting large activation tensors are mostly float)
• torch.where, torch.nonzero: Output shapes of these operators can change depending on the input. We will need a different approach to address this (e.g. gather inputs from all ranks and run through them until we get a stable graph)

@sfc-gh-truwase I can think of these two cases:

Thanks for the explanation. I have two thoughts:

1. My understanding is that these limitations are due to runtime properties of execution as opposed to static properties of the code. If so, then they are normal limitations of compiler approach. In that case, we might want to skip compilation for these case as opposed to silent failures.
2. If my understanding is incorrect, then can we document these limitations in the code until we are able to handle them?

@sfc-gh-truwase, Thank you for your comment!
I currently don't think the dummy values affect correctness as they are used only for profiling.

For two cases I mentioned, the followings can happen:

• Operators that take indices (e.g. embedding, scatter): If an invalid value is given, they throw an error.
• Operator that produces dynamic shapes depending on inputs (torch.where, torch.nonzero): The output might cause an error with following operators during profiling, or inaccurate profiling results in non-optimal graph modification.

An alternative approach is to offload and keep real inputs. As there is a tradeoff between CPU memory consumption and stability/accuracy of profiling, we could give the user the choice.

@sfc-gh-truwase Thanks for the feedback! I've extended the InputStorage to cover wider variety of scenarios.

Here is the summary:

• Enhanced InputStorage to keep real values for integer tensors by default:
  • Added keep_int_input_tensors: bool = True config option (enabled by default)
  • Integer tensors (indices, masks, etc.) now preserve their actual values instead of using dummy ones
  • This addresses correctness issues with operators like embedding and scatter that rely on valid indices
• Added option to keep all input tensors:
  • Added keep_all_input_tensors: bool = False config option for comprehensive tensor preservation
  • Useful for debugging or cases where dummy values cause issues with any tensor type

For both options, we offload the real tensor to the host memory.