{"@attributes":{"version":"2.0"},"channel":{"title":"Attention on Oriol Al\u00e0s Cerc\u00f3s","link":"https:\/\/oriolac.github.io\/tags\/attention\/","description":"Recent content in Attention on Oriol Al\u00e0s Cerc\u00f3s","generator":"Hugo -- 0.150.0","language":"en-us","copyright":"Oriol Al\u00e0s Cerc\u00f3s","lastBuildDate":"Wed, 24 Jun 2026 17:10:23 +0100","item":[{"title":"Under the Hood of Graph Neural Networks: Message Passing, Over-Smoothing and Attention","link":"https:\/\/oriolac.github.io\/posts\/20260624-gnns\/","pubDate":"Wed, 24 Jun 2026 17:10:23 +0100","guid":"https:\/\/oriolac.github.io\/posts\/20260624-gnns\/","description":"<p>In this post we will present an introduction of how <strong>Spatial Graph Neural Networks (GNNs)<\/strong> or <strong>Graph Convolutional\nNeural\nNetworks (GCNs)<\/strong> work. First, we are going to define graph data structures. Then, we are going to explain the mechanism\non GNNs. And finally, we will explain how to incorporate an attention mechanism in the network.<\/p>\n<div class=\"callout callout-info\" role=\"note\">\n<div class=\"callout-body\">\n<p class=\"callout-title\">\n<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"18\" height=\"18\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"><circle cx=\"12\" cy=\"12\" r=\"10\"\/><path d=\"M12 16v-4\"\/><path d=\"M12 8h.01\"\/><\/svg>\nNotation of GNNs<\/p>\n<div class=\"callout-content\"><blockquote>\n<p>During the whole text, we will use the notation of GNN as Spatial Graph Neural Network, although GCN or Graph\nConvolutional Neural Network is another notation to say it. There are other types of GNNs like Spectral Graph Neural\nNetworks, but in this post we will focus on the first mentioned ones.<\/p>"},{"title":"Introduction to Attention Mechanism and Transformers","link":"https:\/\/oriolac.github.io\/posts\/20241029-attention\/","pubDate":"Mon, 17 Feb 2025 12:31:23 +0100","guid":"https:\/\/oriolac.github.io\/posts\/20241029-attention\/","description":"<p>Transformers have demonstrated excellent capabilities and they overcome challenges such <em>NLP<\/em>, <em>Text-To-Image Generation<\/em> or <em>Image Completion<\/em>\nwith large datasets, great model size and enough compute.\nTalking about transformers nowadays is as casual as talking about <em>CNNs<\/em>, <em>MLPs<\/em> or <em>Linear Regressions<\/em>. Why not take a glance through this state-of-the-art architecture?<\/p>\n<p>In this post, we\u2019ll introduce the Sequence-to-Sequence (Seq2Seq) paradigm, explore the attention mechanism, and provide a detailed,\nstep-by-step explanation of the components that make up transformer architectures.<\/p>"}]}}