[{"content":"In this post we will present an introduction of how Spatial Graph Neural Networks (GNNs) or Graph Convolutional Neural Networks (GCNs) work. First, we are going to define graph data structures. Then, we are going to explain the mechanism on GNNs. And finally, we will explain how to incorporate an attention mechanism in the network.\nNotation of GNNs\nDuring the whole text, we will use the notation of GNN as Spatial Graph Neural Network, although GCN or Graph Convolutional Neural Network is another notation to say it. There are other types of GNNs like Spectral Graph Neural Networks, but in this post we will focus on the first mentioned ones.\nWhat are graphs? Graphs are data structures with two main elements: the nodes and their relationships between elements, named edges. The core of graphs is that we can define very easily the interaction of entities. Here, the links are key to the power of relational-data. Graphs are used in several applications, for instance, in social networks, distribution networks, and state-machines.\nFirst steps We have already seen three clear concepts:\nGraph. A data type consisting of nodes and edges. Nodes\/Vertices. Endpoint in a graph, they are also called vertices or points. Edges. The link or relationship between nodes. When trying to describe how is a specific graph, we could always draw the elements and the link between them if necessary. Although it is great to see some properties in a straight-forward way, we could not calculate anything from it. One of the most common mathematical formalisms for describing relations in graphs are adjacency matrices.\nFig. 1. Adjacency matrix of a graph.\nGraphs can be classified as directed or unidrected. In undirected graphs, the adjacency matrix is a symmetric matrix across the diagonals. Each column and row correspond to a node and their values are their corresponding links to the nodes of the graph. For example, in column \\(k\\) we can see the links of node \\(v_k\\) and in row \\(k\\) we can see the same links too, but transposed.\nGiven two nodes \\(v_j\\) and \\(v_i\\), if the value of \\(e_{i, j}\\) or \\(e_{j, i}\\) is zero, that means that no link exists between these nodes. Otherwise, there is a link between these nodes.\nIf the adjacency matrix is not symmetric, we have a directed graph. The links of directed graphs have directions, so that means that a node \\(v_j\\) can interact to \\(v_i\\), but it does not have to be the other way around. Directed edges are usually represented by an arrow, denoting a one-way relationship.\n\\[ G = \\begin{pmatrix} 1 & 0 & 0 & 1\\\\ 0 & 0 & \\textcolor{blue}{1} & \\textcolor{blue}{1}\\\\ 0 & \\textcolor{red}{0} & 0 & 1\\\\ 1 & \\textcolor{red}{0} & 1 & 0 \\end{pmatrix} \\qquad \\text{This is a directed graph.} \\]If not all the non-zero values of the adjacency matrix are one, we are talking about weighted graphs. Weighted graphs can have weighted links.\nOther key terms Other key terms about edges and nodes are:\nSelf-loop. An edge that connects a node to itself. Parallel edges. Multiple edges that connect the same two nodes. Joint nodes or neighbours are those nodes that are directly connected via an edge. A node \\(v_i\\) is adjacent to another node \\(v_j\\) if there is a edge. The set of all the neighbours from a node is called its neighbourhood. \\[ \\mathcal{N}(v_i) = \\{ v_j \\in V \\mid (v_i, v_j) \\in E \\} \\] Fig. 2. A graph that has a self-loop, parallel edges. The diagram shows the neighbourhood of node.\nOnce defined the different kind of graphs depending on the nodes and the edges, we can see other properties checking the structure of a specific node.\nSize is the number of edges. Order is the number of nodes. The degree of a node is the number of edges associated to a node. It is the count of its adjacent nodes. The degree distribution is the distribution of all the degrees of all nodes in a graph. In directed graphs, there are two types of degrees: the in-degrees for edges directed to the node and the out-degrees, for edges directed outward from the node. The diameter of a graph is the maximum number of the longest shortest path. Fig. 3. Degrees and diameter of the graph. The diameter is 4 given the walk (0,1,2,5,6).\nA connected graph is a graph where all its nodes are connected. Otherwise, it is a disconnected graph. For a disconnected graph, each disconnected piece is called a component. For a directed graph, a strongly connected graph is when it is always possible to reach any node from any other node. In directed graphs, we can have strongly connected components and weakly connected components. Strongly connected components are the subgraphs that are connected to each other and any node in the subgraph can reach any other node in the subgraph. Weakly connected components are subgraphs where any node can be reached but not all the nodes can reach each other. Graph Traversals\nWhen inspecting a graph we might ask ourselves a lot of questions. For example, in a social network how many connections do I have to hop to meet Meryl Streep? In a distribution system, which is the smallest number of hops from one node to another? Which is the largest number of hops without repeating in a graph?\nThe trip that we have to do to travel from a given node to a second node is called traversal or walk. A walk is open when the ending node is different from the starting node. If we start and end with the same node, we call it closed walk.\nA path is a walk when no node is repeated. When the path is closed, we call it cycle. The diameter of a graph is the maximum number of steps of a path. A diameter is also called the longest shortest path.\nA trail is when the walk do not repeat edges. A circuit is when we have a closed trail.\nHow Graph Neural Networks work? A Graph Neural Network (GNN) is a deep learning model that allows you to represent and learn from graphs (Scarselli et al., 2009). The reason why they were invented is due to the lack of inductive biases in the nature of traditional machine learning and deep learning models such as MLP. In machine learning involving tabular data, images, or text, our data is organized in an expected way, with implicit and more explicit rules. When dealing with tabular data, rows are treated as observations while columns as features. But in graph data, relations between “rows” or instances are meaningful. IN GNNs, we have the information of each node, called embeddings and the links that connects each node to its neighbourhood.\nMain idea of GNNs\nThe essential idea of graph neural networks is to iteratively update the node representations by combining the representations of their neighbours and their own representations.\nThe main objective of a Graph Neural Network is to encode the data structure to then predict. But, what? There are several tasks that we could come up with, but they can be classified as:\nNode-level tasks. Given a graph, classify the nodes, use regression, or detect anomalies. Edge-level tasks. Similar to the node-level tasks, but for edges. Graph-level tasks. Graph classification or regression. Graph AutoEncoders (GAE) could be also in this category. In section 1 we have already understood the importance of the links when defining a graph. GNNs came up to use this nature in the network architecture: use a mechanism to encode and exchange information across the graph structure during the inference: the graph convolutional layers (GCL). The GCN layer \\(l\\) is mathematically defined as\n\\[ \\mathbf{x}_i^{(l)} = \\sum_{j \\in \\mathcal{N}(i) \\cup \\{ i \\}} \\frac{1}{\\sqrt{\\deg(i)} \\cdot \\sqrt{\\deg(j)}} \\cdot \\left(\\mathbf{W}^{\\top} \\cdot \\mathbf{x}_j^{(l-1)} \\right) + \\mathbf{b}, \\]where neighboring node features are first transformed by a learnable weight matrix \\(\\mathbf{W}^{\\top}\\), normalized by their degree, and finally summed up. Lastly, we apply the bias vector to the aggregated output.\nHowever, when we talk unformally about GCL, we talk about a bundle of sequential layers:\nMessage Passing layer: where information is aggregated from neighbours and updated for each node. Activation Layer: where the information is passed to the next layer. Dropout layer: switching off some neurons to improve generalization and performance. Normalization Layer: normally Batch Normalization, that means the activated outputs to zero with a variance of 1. The message-passing method Message passing is designed specifically for asking about the graph data structure. For each node in the graph, each message passing step represents a communication that spans nodes one hop away. If we want our node representations to take account node from 5 hops from each node, we should need 5 message passing layers.\nMessage passing can be understood as a form of convolution but applied to the neighbourhood of the nodes instead of a neighbourhood of pixels. Through convolution operators, we are encoding neighbour states to the current node and gathering more global information of the graph.\nInductive biases of GNNs\nHence, Graph Neural Networks can be considered an abstraction of Convolutional Neural Networks.\nA popular way to introduce GCLs is to break down the filter into two operations:\nAGGREGATE-NODES. Given a node \\(v_i\\) and its linked nodes \\(\\mathcal{N}(v_i)\\), we aggregate the information of each neighbour node: \\[ m_i^{(l)} = \\operatorname{AGGREGATE} \\left( \\left\\{ h_j^{(l)} \\mid v_j \\in \\mathcal{N}(v_i) \\right\\} \\right) \\]The aggregation function \\(AGGREGATE\\) is commonly a sum operation, although it can be the mean, minimum, maximum, or multiplication.\nFig. 4. Aggregate function of node \\(6\\). This step aggregates the embeddings of the neighbours of the node by multiplying or summing. The resulting value is the message \\(m_6^{(l)}\\)\nUPDATE-EMBEDDING. Then, we update the embedding of the node \\(v_i\\) with the aggregated information applying linear transformations: \\[ h_i^{(l+1)} = \\sigma \\left(W^{(l)} \\cdot \\operatorname{CONCAT} \\left(h_i^{(l)}, m_i^{(l)} \\right) + b^{(l)} \\right) \\] Fig. 5. We concat the current embedding of the node \\(6\\) with the aggregate message of its neighbours. The resulting value is multiplied by the kernel matrix \\(W^{(l)}\\) and added to the bias vector \\(b^{(l)}\\).\nSince we will be working on convolutions, a key item must be introduced: the kernel \\(W^{(l)}\\). The kernel is a matrix that we will be using to transform the input data (from the neighbours) and highlight specific features from them. The kernel is the learnable weight matrix to be optimized by our loss function, along with the bias \\(b^{(l)}\\).\nInductive biases of GNNs\nGraph-based learning techniques focuses on approaches that are permutation invariant or equivariant. This means that the model is not influenced by the ordering of the graph representation. Therefore, if we shuffle the rows and the columns of the adjacency matrix, our results should not change.\nAlthough this example helps to understand how GNNs work at first glance, it is not the most used algorithm. The generic message-passing equation is\n\\[ \\mathbf{x}_i^{(l)} = \\gamma^{(l)} \\left( \\mathbf{x}_i^{(l-1)}, \\bigoplus_{j \\in \\mathcal{N}(i)} \\phi^{(l)} \\left( \\mathbf{x}_i^{(l-1)}, \\mathbf{x}_j^{(l-1)}, \\mathbf{e}_{j,i} \\right) \\right) \\]where \\(\\bigoplus\\) is the aggregator function, \\(\\phi^{(l)}\\) is the message function at layer \\(l\\) and \\(\\gamma^{(l)}\\) is the update function. At first, it might look a bit messy since the order is not defined as it is in the example above. So let’s break it by parts again.\nThe message function \\(\\phi^{(l)}\\) defines what information each neighbor sends to node \\(i\\). In the first example, the message function is based solely on sending the embedding information \\(h_j^{(l)}\\) to node \\(i\\). Nevertheless, differentiable functions such as MLPs (Multi Layer Perceptrons) can be applied. In addition, the message function can also have as an input the node embedding \\(h_i^{(l)}\\) as well as the link between the neighbours (useful for weighted graphs). In our example, \\[ \\phi^{(l)} \\left(\\mathbf{x}_i^{(l-1)}, \\mathbf{x}_j^{(l-1)}, \\mathbf{e}_{j,i} \\right) = x_j^{(l-1)}\\] The aggregator function \\(\\bigoplus\\) is the permutation invariant function: sum, max, mean… The update function \\(\\gamma^{(l)}\\) is the function that updates the node embedding using the old embedding plus the aggregated message. As you can see, the generic message-passing joins the update with the concat function, which is not needed to be a neither concatenation. Normally, the update functions are also differentiable functions such as MLPs. The following equation is the update function for the first example: \\[ \\gamma^{(l)} \\left(\\mathbf{x}_i^{(l-1)}, \\bigoplus_{j \\in \\mathcal{N}(i)} x_j^{(l-1)} \\right) = \\sigma \\left(W^{(l)} \\cdot \\operatorname{CONCAT} \\left(x_i^{(l-1)}, \\bigoplus_{j \\in \\mathcal{N}(i)} x_j^{(l-1)} \\right) + b^{(l)} \\right)\\] How to apply GNNs to downstreaming tasks? Since now, we have introduced how to process the data of the nodes with the information of their neighbours. However, we did not apply the information of the nodes to the downstream tasks. In a lot of cases, the GNNs are used only as encoders, while decoders or head networks, such as MLPs, are used to apply the learned graph representation to downstreaming tasks such as node classification, edge prediction, or graph classification.\nThe GNNs will output the node embeddings, we must think another way around to apply downstream tasks. For example, we can have an MLP to classify the node given the output embeddings given a node (treating then nodes as instances or rows). We could also apply a function to the embeddings of two nodes to get the distance between them and, then, predict if they are connected or not. Regarding graph classification, we could apply global pooling to the embeddings of the nodes to get the final representation of the graph and, then, use an MLP to classify it.\nFig. 6. Given the output of the GNN, we can apply downstream tasks such as node, edge, or graph prediction (regression or classification).\nAttention mechanism in GNNs A GNN with a lot of layers makes the nodes to be more similar to each other, since they will have more global information than first layers. Although it can be great for some tasks, the embedding nodes tend to be more similar to each other, vanishing local information. This problem is called over-smoothing. A way to check if our GNN over-smooths the embeddings of the nodes is to calculate the similarity between each embedding and the average embedding of the nodes. If the similarity of the node and the average is close to 1, it means that the GNN is clearly over-smoothing.\nTo over-smooth or not to over-smooth?\nDepending on the case, we do not care if some nodes are “over-smoothed”, especially those that are central or has closeness centrality. Furthermore, in some downstream tasks, these nodes might be important. There are different techniques to avoid over-smoothing such as applying skip connections, similar to U-Net (Ronneberger et al., 2015) or ResNets (He et al., 2016). As you could notice, nodes that have more links tend to be more similar to the average embedding. Specifically, one of the techniques that helps to prevent over-smoothing in these cases is to include attention in the message passing. Therefore, the attention mechanism allows to learn which nodes the network has to put emphasis on.\nSee also Introduction to Attention Mechanism and Transformers \u2192 There are two different attention mechanisms that became popular in the literature: GAT (Veli\u010dkovi\u0107 et al., 2018) and GATv2 (Brody et al., 2022). The difference between them is where they apply the attention mechanism.\nGraph Attention Network (GAT) Computes the attention weights once per training loop by using individual node and neighborhood features, being static across all layers.\nIn standard GAT, the unnormalized attention score is usually written as:\n\\[ e_{i, j} = \\text{LeakyReLU} (a^T [W \\cdot h_i || W \\cdot h_j]) \\]where \\(a\\) is the learnable attention vector, and \\(W\\) another learnable parameter. We can have also two different learnable parameters \\(W_s\\) (for the source \\(h_i\\)) and \\(W_t\\) (for the target \\(h_j\\)).\nFig. 7. Unnormalised Attention Score mechanism in GAT.\nThen, softmax is applied to normalize the attention scores.\n\\[ \\alpha_{i, j} = \\text{softmax}_{j \\in \\mathcal{N}(i)} (e_{i, j}) \\]We can express GAT and GATv2 as specific choices of: \\(\\phi^{(l)}\\). For example:\n\\[ \\phi^{(l)}_{\\text{GAT}} (x_i^{(l-1)}, x_j^{(l-1)}, e_{i,j}) = \\alpha_{i, j} \\cdot x_j^{(l-1)} \\]We can also add a learnable parameter to process the attention vector alongside with the neighbour node embedding.\n\\[ \\phi^{(l)}_{\\text{GAT}} (x_i^{(l-1)}, x_j^{(l-1)}, e_{i,j}) = \\alpha_{i, j} \\cdot W^{(l)} \\cdot x_j^{(l-1)} \\]The problem found in GAT is subtle: the attention vector can be decomposed as:\n\\[ a^T [W h_i || W h_j] = a_s^T \\; W \\; h_i + a_t^T \\; W \\; h_j \\]For a fixed source node \\(i\\), the term \\(a_s^T \\; W \\; h_i \\) is constant to all its neighbours \\(j\\). Therefore, the ranking is mostly determined by \\(a_t^T \\; W \\; h_j \\). This kind of attention mechanism is called static, since the ranking of the neighbours does not depend on the source node but only the target node. That means that the attention could be calculated before knowing the link between the nodes. In other words, the attention can be calculated and cached for the whole graph.\nGATv2 GATv2 changes the order of operations in the attention mechanism.\n\\[ e_{i, j} = a^T \\text{LeakyReLU} ( W [ h_i || h_j]) = a^T \\text{LeakyReLU} ( W_s h_i + W_t h_j) \\]As in GAT, we can have two learnable parameters \\(W_s\\) and \\(W_t\\) for the attention scores, one for the source node and one for the target node plus the attention vector \\(a\\). Then, we will apply softmax to normalize the attention scores.\n\\[ \\alpha_{i,j} = \\text{softmax}_j \\left( e_{i, j} \\right) \\]The main difference is that the non-linearity is applied before the final attention projection. This allows the source node \\(i\\) to influence the ranking of the neighbours \\(j\\). This small change makes GATv2 dynamic attention, showing more expressive attention than GAT while keeping the same parametric cost. Since this mechanism implies calculating the attention for each link, the attention cannot be calculated for the whole graph but for each node. Hence, the number of links from the nodes (the order) increases the cost.\nUsing the generic message-passing equation, we can rewrite the GATv2 attention function as it was with GAT:\n\\[ \\phi^{(l)}_{\\text{GATv2}} (x_i^{(l-1)}, x_j^{(l-1)}, e_{i,j}) = \\alpha_{i,j} \\; \\text{LeakyReLU} ( W_s h_i + W_t h_j), \\]but the difference is in the attention score already explained. We can keep \\(\\gamma^{(l)} \\) and \\(\\bigoplus\\) the same.\nIt is important to note that the attention mechanism adds new learnable parameters to the model, which can be significantly increased in the number of parameters. However, the attention mechanism can help not only to prevent over-smoothing but detect specific important links in the graph.\nConclusion Graph Neural Networks extend deep learning to data where the relationships between entities are as important as the entities themselves. Instead of processing nodes independently, GNNs exploit the graph structure through message passing, allowing each node to update its representation by combining its own features with information coming from its neighbourhood.\nAlthough GNNs adds a new dimension to deep learning and graph understanding, stacking many message-passing layers can lead to over-smoothing. Attention mechanisms help address this issue by allowing the model to learn which neighbours should contribute more strongly to each node update.\nReferences Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009). The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 61\u201380. https:\/\/doi.org\/10.1109\/TNN.2008.2005605\nHe, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770\u2013778). IEEE. https:\/\/doi.org\/10.1109\/CVPR.2016.90\nRonneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In N. Navab, J. Hornegger, W. M. Wells, & A. F. Frangi (Eds.), Medical Image Computing and Computer-Assisted Intervention \u2013 MICCAI 2015 (pp. 234\u2013241). Springer. https:\/\/doi.org\/10.1007\/978-3-319-24574-4_28\nVeli\u010dkovi\u0107, P., Cucurull, G., Casanova, A., Romero, A., Li\u00f2, P., & Bengio, Y. (2018). Graph attention networks. International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rJXMpikCZ\nBrody, S., Alon, U., & Yahav, E. (2022). How attentive are graph attention networks? International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=F72ximsx7C1\nCitation @article{alas2026, title = "Under the Hood of Graph Neural Networks: Message Passing, Over-Smoothing and Attention", author = "Al\u00e0s Cerc\u00f3s, Oriol", journal = "oriolac.github.io", year = "2026", month = "June", url = "https:\/\/oriolac.github.io\/posts\/20260624-gnns\/" } ","permalink":"https:\/\/oriolac.github.io\/posts\/20260624-gnns\/","summary":"

In this post we will present an introduction of how Spatial Graph Neural Networks (GNNs)<\/strong> or Graph Convolutional\nNeural\nNetworks (GCNs)<\/strong> work. First, we are going to define graph data structures. Then, we are going to explain the mechanism\non GNNs. And finally, we will explain how to incorporate an attention mechanism in the network.<\/p>\n

\n
\n

\n <\/svg>\n\n Notation of GNNs<\/p>\n

\n

During the whole text, we will use the notation of GNN as Spatial Graph Neural Network, although GCN or Graph\nConvolutional Neural Network is another notation to say it. There are other types of GNNs like Spectral Graph Neural\nNetworks, but in this post we will focus on the first mentioned ones.<\/p>","title":"Under the Hood of Graph Neural Networks: Message Passing, Over-Smoothing and Attention"},{"content":"Object detection is one of the most popular tasks in computer vision, since it can be applied to a wide range of applications: robotics, autonomous driving or fault detection. In this post, we will try to give a brief overview of the YOLO algorithm and the components that make it work.\nTo do that, I have classified the main components of the algorithm into three categories:\nCharacteristics based on the model architecture, how YOLO-based models improved the performance by using a new architecture and which are the improvements made. Strategies based on the model training, such as the function loss or data augmentation. Methods for post-processing the output of the model, such as the non-maximum suppression (NMS) and the confidence threshold. Two-stage vs One-stage Detectors Before YOLO, SoTA detectors were based on a two-stage detector: the first stage is used to detect the bounding boxes, and the second stage is used to classify the bounding boxes. This kind of model is called region-based detectors, because they need the region to then run the classification.\nFig. 1. RCNN architecture. (Bhalla, 2022)\nIn contrast, YOLO is a one-stage detector, YOLO models skip the first stage and runs directly over a dense sampling of possible locations and gives the bounding boxes and the classification all at once. The first idea behind YOLO was to reduce the computational cost of the region-based models (increasing the FPS) although mantaining or decreasing a little bit the performance. This idea was inspired by Single Shot MultiBox Detector (SSD), introduced in 2016.\nFrames per second (FPS)\nFPS measures how many frames a model can process per second \u2014 the key metric for real-time inference. A detector running at 7 FPS cannot drive a car or track objects in live video; one running at 45 FPS can. The architecture: Backbone, neck and head A YOLO model usually has three main parts:\nThe backbone network, which extracts features from the input image. The backbone progressively reduces spatial resolution while increasing semantic richness. The neck, which combines the feature maps from the backbone with a convolutional layer. So the neck helps the detector handle objects of different sizes. The head, which produces the final prediction. Fig. 2. YOLO-based network architecture. Check image source\nThese parts are strongly connected to the idea of multi-scale detection. YOLO does not predict bounding boxes from only one feature map. Instead, it preodicts object at different scale resolutions. For example, if the input image has size 640 \u00d7 640, a YOLO model may produce three detection scales:\nP3 -> 80 x 80 -> small objects P4 -> 40 x 40 -> medium objects P5 -> 20 x 20 -> large objects At the beginning of the backbone, the feature maps have high spatial resolution and contain low-level information, such as edges, corners, textures, and small visual patterns. As the image passes through deeper layers, the spatial resolution decreases, but the semantic meaning of the features increases. The backbone network gives intermediate feature maps to the neck according to the idea of multi-scale detection. Usually the backbone is already pre-trained on a large dataset.\nSince the backbone is meant to extract features from the input image in different scales, it is important to create a network that has neither stride operations nor pooling layers, as they can reduce the spatial resolution and semantic information.\nThe neck takes the feature maps from the backbone and mixes these features so that each detection scale benefits from both.\nThe detection heads are the final prediction layers. Usually, YOLO has one detection head per scale. Each head predicts bounding boxes (4 values), confidence score (1 value), and class probabilities (C values) for its corresponding feature map. The final prediction output of each head has the shape \\(S \\times S \\times (5B + C) \\). If there are 3 different scales, then the output will have three tensors with the previous shape.\nFig. 3. YOLOv4 Architecture. (Terven & Cordova-Esparza, 2023)\nB is the number of bounding boxes per grid cell. It can only be 1, 2 or more. If B is 1, then the model predicts only one bounding box per grid cell.\nThe stride The stride (S) tells you how much the input of the image has been downsampled. Therefore, each cell in the feature map corresponds to a region of the original image. The \\(5B + C\\) part of the output is the bounding box (x, y, height and width) and class probabilities.\nThis is important because YOLO predicts object centers relative to grid cells. Small objects need high-resolution feature maps, so they are usually predicted at lower stride, such as stride 8.\nFig. 4. Example of strides using different scales, with the centroid of the bounding box to determine which is the stride cell of the image to predict.\nThe confidence score A YOLO head usually predicts something like: tx, ty, tw, th, objectness, class probabilities While tx and ty are the predicted center offset of the bounding box and tw and th are the predicted width and height of the bounding box, the objectness value is the probability that an object exists in this prediction. The confidence score is commonly calculated by multiplying the objectness value with the class probabilities.\n\\[\\text{Confidence score} = \\text{Objectness} \\times \\text{Class probability}\\]Confidence score is used to remove weak predictions from the output of the model and reduces the number of low-quality detections\nThe anchor boxes YOLO models can be divided into two families:\nAnchor-free models, such as (Ge et al., 2021). Anchor-based models, such as (Redmon & Farhadi, 2018). Anchors are predefined box shapes. YOLO models use anchors to predict offsets relative to these boxes. In anchor-based model of three scales, we can have the following anchors:\np3 = [(62, 66), (45, 213), (105, 104)] p4 = [(196, 76), (153, 143), (96, 316)] p5 = [(266, 266), (350, 465), (420, 500)] Anchors are usually defined by classifying the objects of the training set using an unsupervised clustering algorithm, such as k-means.\nWhen using anchors, the output the model is a tensor of shape \\(S \\times S \\times A \\times (5B + C) \\). Suppose we are at grid cell (i, j) on a feature map with stride s. The model predicts raw values: tx, ty, tw, th. Hence, a classical YOLO-style decoding example with anchor (60, 40) is:\nbx = (sigmoid(tx) + j) * stride # e.g. (0.55 + 30) \u00d7 8 = 244.4 by = (sigmoid(ty) + i) * stride # e.g. (0.40 + 20) \u00d7 8 = 163.2 bw = anchor_w * exp(tw) # e.g. 40 \u00d7 1.10 = 44.0 bh = anchor_h * exp(th) # e.g. 60 \u00d7 0.82 = 49.2 In modern anchor-free YOLO variants such as YOLOX, anchors may not be used explicitly. Instead, the model directly predicts box distances or center-based boxes.\nDecoupled head YOLOX utilizes a decoupled head, a significant departure from the single-head design in the previous YOLO models.\nIn traditional YOLO models, the head predicts object classes and bounding box coordinates using the same set of features. This approach simplified the architecture back in 2015, but it had a drawback. It can lead to suboptimal performance, since classification and localization of the object was performed using the same set of extracted features, and thus leads to conflict. Therefore, YOLOX introduced a decoupled head.\nFig. 5. YOLOX decoupled head architecture. (Ge et al., 2021)\nThe decoupled head consists of two separate branches:\nClassification Branch. Focuses on predicting the class probabilities for each object in the image. Regression Branch. Concentrates on predicting the bounding box coordinates and dimensions for the detected objects. See also Loss Functions and Activation Functions \u2192 Model training Unlike image classification, where the model predicts one label for the whole image, object detection requires solving several problems at the same time: deciding whether an object exists in a given location, estimating the coordinates of its bounding box, and assigning the correct class. For this reason, YOLO training is usually based on a multi-part loss function that combines localization, objectness, and classification terms.\nIntersection over Union (IoU) Intersection over Union (IoU) is a measure of the similarity between two bounding boxes. It is the division between the area of the intersection and the area of the union.\n\\[IoU = \\frac{\\text{Area of Intersection}}{\\text{Area of union}}\\]It is used in several steps of the training process:\nTraining loss. Anchor assignment. Post-processing techniques such as non-maximum suppression (NMS). Evaluation metrics such as mAP. Fig. 6. Intersection over Union (IoU) formula.\nmAP \u2014 Mean Average Precision\nmAP is the standard metric for comparing object detectors across classes and IoU thresholds. For each class, predictions are sorted by confidence score and a precision-recall curve is computed. The area under that curve is the Average Precision (AP) for that class. mAP averages AP over all \\(C\\) classes:\n\\[mAP = \\frac{1}{C}\\sum_{c=1}^{C}AP_c, \\qquad AP_c = \\sum_{n} (R_n - R_{n-1})\\, P_n\\]where \\(R_n\\) is the recall and \\(P_n\\) the best precision at each confidence threshold.\nLoss function The loss function tells the model how wrong its predictions are during training. The YOLO loss trains the model to solve three tasks at the same time:\nThe box loss: localize the object correctly. The objectness loss: predict whether an object exists. The class loss: classify the object correctly. A simplified YOLO loss can be written as:\n\\[ L = \\lambda_{\\text{box}}L_{\\text{box}} + \\lambda_{\\text{obj}}L_{\\text{obj}} + \\lambda_{\\text{cls}}L_{\\text{cls}} \\]where \\(\\lambda_{\\text{box}}\\), \\(\\lambda_{\\text{obj}}\\), and \\(\\lambda_{\\text{cls}}\\) are weighting factors used to balance the contribution of each term.\nThe box loss The box loss measures how well the predicted bounding box matches the ground-truth box. Older YOLO versions used mean squared error (MSE) over the box coordinates, but modern YOLO models usually use IoU-based losses because they are more directly aligned with the object detection objective. A simple IoU loss can be defined as:\n\\[ L_{\\text{box}} = 1 - IoU(b, \\hat{b}) \\]where \\(b\\) is the ground-truth box and \\(\\hat{b}\\) is the predicted box.\nHowever, modern detectors often use more advanced variants such as GIoU, DIoU, or CIoU. For example, CIoU includes not only the overlap between boxes, but also the distance between their centers and their aspect-ratio consistency:\n\\[ L_{\\text{CIoU}} = 1 - IoU + \\frac{\\rho^2(b, \\hat{b})}{c^2} + \\alpha v \\]where \\(\\rho^2(b, \\hat{b})\\) is the squared distance between the centers of the predicted and ground-truth boxes, \\(c^2\\) is the squared diagonal length of the smallest enclosing box, and \\(\\alpha v\\) penalizes differences in aspect ratio.\nThe objectness loss The objectness loss teaches the model whether a prediction contains an object. YOLO makes thousands of predictions per image. Therefore, it is important to balance the false positives. How? By adding weights when the model predicts a positive object. For example, a training implementation may use two different weights:\nneg_obj_weight_with_pos: the weight applied to negative predictions in a scale where at least one positive object exists. neg_obj_weight_no_pos: the weight applied to negative predictions in a scale where no positive object exists. This distinction is useful in multi-scale YOLO training. Suppose that an image contains a small object assigned to the P3 scale, but no objects are assigned to P4 or P5. In that case, P3 contains both positive and negative samples, while P4 and P5 contain only negative samples. If the loss gives too much weight to all negative predictions, the model may learn to predict background everywhere and become too conservative. These weights help balance the objectness loss so that negative examples are useful but do not dominate the training signal.\nThe class loss The class loss teaches the model which class is present in a positive prediction. There are two common ways to compute it. If each object belongs to exactly one class, the model can use a softmax activation followed by categorical cross-entropy:\n\\[ L_{\\text{cls}} = - \\sum_{c=1}^{C} y_c \\log(\\hat{p}_c) \\]where \\(y_c\\) is the ground-truth class indicator and \\(\\hat{p}_c\\) is the predicted probability for class \\(c\\).\nHowever, many YOLO implementations use binary cross-entropy independently for each class:\n\\[ L_{\\text{cls}} = - \\sum_{c=1}^{C} \\left[ y_c \\log(\\hat{p}_c) + (1-y_c)\\log(1-\\hat{p}_c) \\right] \\]This formulation treats class prediction as \\(C\\) independent binary classification problems. It is especially useful when multi-label classification is possible, although it is also commonly used in single-label YOLO detectors.\nCheck that each stride \\((S, S)\\) has only one positive prediction. Therefore, the model cannot learn to predict two objects in the same location.\nData Augmentation Data augmentation is another important part of YOLO training. Its goal is to expose the model to more visual variation without manually collecting more data. Common augmentations include random scaling, cropping, horizontal flipping, color jittering, mosaic augmentation, and MixUp.\nMixUp combines two images and their labels into a single training example. The resulting image is a weighted combination of both images:\n\\[ \\tilde{x} = \\lambda x_1 + (1-\\lambda)x_2 \\]where \\(x_1\\) and \\(x_2\\) are two training images, and \\(\\lambda\\) controls how much each image contributes to the final mixed image.\nFig. 5. MixUp Data Augmentation. Check source\nMixUp is used for example in YOLOX models and it has been found to be more effective in larger models.\nPost-processing After the model produces its raw predictions, these outputs still need to be converted into final detections. A YOLO model usually predicts many candidate boxes for the same object, many low-confidence boxes, and sometimes overlapping detections from different scales. Post-processing transforms these dense predictions into a clean final set of bounding boxes by applying confidence filtering, decoding the predicted coordinates, and removing duplicated detections with non-maximum suppression.\nNon-maximum suppression (NMS) Non-Maximum Suppression (NMS) is used to remove duplicate detections in bounding box prediction. YOLO often predicts many boxes around the same object. NMS keeps the strongest one and removes highly overlapping boxes.\nFig. 6. Non-Maximum Suppression. (Terven & Cordova-Esparza, 2023)\nThe process is as follows:\nSort the predictions by their confidence score. For each class: Keep the box with the highest confidence score. Check other boxes with high IoU overlap with the kept box. Repeat until all boxes have been processed. Class-agnostic vs class-aware NMS\nClass-agnostic NMS suppresses overlapping boxes regardless of predicted class \u2014 simpler and faster. Class-aware NMS applies suppression separately per class, which matters when two objects of different classes genuinely overlap in the same region (e.g., a person holding a tennis racket). Bhalla, D. (2022, June). Region Proposal Network (RPN): A complete guide. ListenData. https:\/\/www.listendata.com\/2022\/06\/region-proposal-network.html\nGe, Z., Liu, S., Wang, F., Li, Z., & Sun, J. (2021). YOLOX: Exceeding YOLO series in 2021. arXiv. https:\/\/doi.org\/10.48550\/arXiv.2107.08430\nRedmon, J., & Farhadi, A. (2018). YOLOv3: An incremental improvement. arXiv. https:\/\/doi.org\/10.48550\/arXiv.1804.02767\nTerven, J., & Cordova-Esparza, D. (2023). A comprehensive review of YOLO architectures in computer vision: From YOLOv1 to YOLOv8 and YOLO-NAS. Machine Learning and Knowledge Extraction, 5, 1680\u20131716. https:\/\/doi.org\/10.3390\/make5040083\n@article{alas2026, title = "Reviewing YOLO: You Only Look Once", author = "Al\u00e0s Cerc\u00f3s, Oriol", journal = "oriolac.github.io", year = "2026", month = "April", url = "https:\/\/oriolac.github.io\/posts\/20260501-yolo\/" } ","permalink":"https:\/\/oriolac.github.io\/posts\/20260501-yolo\/","summary":"

Object detection is one of the most popular tasks in computer vision, since it can be applied to a wide range of\napplications: robotics, autonomous driving or fault detection. In this post, we will try to give a brief overview of\nthe YOLO algorithm and the components that make it work.<\/p>\n

To do that, I have classified the main components of the algorithm into three categories:<\/p>\n