Feature request

I am requesting static cache support for the XLA compiler for transformer models. This will prevent recompilation at each step caused by slicing

Motivation

To enhance the efficiency of autoregressive generation in transformer models, static cache support for the XLA can be implemented to mitigate the need for recompilation at each generation step.

Your contribution

Use the Index_copy method to update static cache inplace

• Additionally, While out-of-place operations could offer further improvements, they introduce new tensors with different addresses, complicating calculations. Addressing this issue is crucial for fully leveraging the potential of XLA.