Attention

In this post we will present an introduction of how Spatial Graph Neural Networks (GNNs) or Graph Convolutional Neural Networks (GCNs) work. First, we are going to define graph data structures. Then, we are going to explain the mechanism on GNNs. And finally, we will explain how to incorporate an attention mechanism in the network. Notation of GNNs During the whole text, we will use the notation of GNN as Spatial Graph Neural Network, although GCN or Graph Convolutional Neural Network is another notation to say it. There are other types of GNNs like Spectral Graph Neural Networks, but in this post we will focus on the first mentioned ones. ...

Transformers have demonstrated excellent capabilities and they overcome challenges such NLP, Text-To-Image Generation or Image Completion with large datasets, great model size and enough compute. Talking about transformers nowadays is as casual as talking about CNNs, MLPs or Linear Regressions. Why not take a glance through this state-of-the-art architecture? In this post, we’ll introduce the Sequence-to-Sequence (Seq2Seq) paradigm, explore the attention mechanism, and provide a detailed, step-by-step explanation of the components that make up transformer architectures. ...