My implementation of the Transformer model (Attention is All You Need, Vaswani et al.)
I've created this implementation of the Transformer model to gain a better understanding of modern machine learning models, especially encoder-decoder and autoregressive models, and how they actually work.
Here are some links that I've found useful over in understanding various concepts within the transformer model:
- https://arxiv.org/pdf/1706.03762.pdf The original paper.
- https://jalammar.github.io/illustrated-transformer/ Illustrates a lot of the main concepts of the attention mechanism of transformers.
- https://e2eml.school/transformers.html Covers a lot about each of the sublayers/modules within the transformer, and why we use them within the transformer.
- https://www.pinecone.io/learn/batch-layer-normalization/ Helped a lot with understanding Batch vs. Layer normalization.