Transformers learn in-context by gradient descent

Notebooks for easy replication of the results in the paper Transformers learn in-context by gradient descent.

As the naming suggests, the three notebooks can be used to reproduce the results for the

1. specific token construction where we concatenate in- and outputs.
2. usual token construction where we provide in- and outputs in neighbouring tokens.
3. experiments on non-linear regression tasks.

You can also use the following links to run the notebooks in Google colab.

1. https://colab.research.google.com/github/transformerGD/transformers-learn-in-context-by-gradient-descent/blob/main/normal_token_construct.ipynb
2. https://colab.research.google.com/github/transformerGD/transformers-learn-in-context-by-gradient-descent/blob/main/constructed_token_setup.ipynb
3. https://colab.research.google.com/github/transformerGD/transformers-learn-in-context-by-gradient-descent/blob/main/non_linear_regression.ipynb