Scribd
Upload
320 views9 pages

The Transformer Architecture

The document summarizes the overall architecture of the Transformer model. It consists of an encoder and decoder, each with stacked blocks containing multi-head attention and feedforward layers. The encoder processes the input sequence and outputs hidden states, while the decoder predicts the output sequence using those hidden states through cross-attention and self-attention layers. Key components include positional embeddings, layer normalization, and a predicting head that produces the output sequence.

Uploaded by

alexandre albalustro
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
320 views9 pages

The Transformer Architecture

The document summarizes the overall architecture of the Transformer model. It consists of an encoder and decoder, each with stacked blocks containing multi-head attention and feedforward layers. The encoder processes the input sequence and outputs hidden states, while the decoder predicts the output sequence using those hidden states through cross-attention and self-attention layers. Key components include positional embeddings, layer normalization, and a predicting head that produces the output sequence.

Uploaded by

alexandre albalustro
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

The Transformer Architecture TheAiEdge.

io

Decoder
'you'
Encoder
Encoder output
Predicting head

Encoder Decoder
block block
Encoder Decoder
block block
Encoder Decoder
block block
Token Token
embedding embedding

Position Position
embedding embedding
'how' 'are' 'you' 'doing' '?' [SOS] 'I' 'am' 'good' 'and'

Input Output
sequence sequence

The Overall Architecture


The Transformer Architecture TheAiEdge.io

even i
odd i

The Position Embedding


The Transformer Architecture TheAiEdge.io

Encoder block

Multihead Layer Feed Layer


Attention Normalization Forward Normalization
Layer network

The Encoder Block


The Transformer Architecture TheAiEdge.io

Keys
Self-attentions

Wk
Hidden Queries Softmax

states
Wq

Wv

Values

Hidden
states

The Self-Attention Layer


The Transformer Architecture TheAiEdge.io

Hidden
state

Layer
Normalization

The Layer Normalization


The Transformer Architecture TheAiEdge.io

dmodel
dff
dff

Linear layer dmodel

Linear layer

The Position-wise Feed-forward Network


The Transformer Architecture TheAiEdge.io

Encoder
output

Decoder block

Hidden
states

Cross Feed Layer


Attention Forward
Multihead Layer Normalization
Layer Layer
Attention Normalization network
Normalization
Layer

The Decoder Block


The Transformer Architecture TheAiEdge.io

Keys
Cross-attentions
Encoder
Wk
output
Queries Softmax

Wq

Wv

Values

Hidden
Decoder states
hidden
states

The Cross-Attention Layer


The Transformer Architecture TheAiEdge.io

‘How’

‘are’

‘you’ Encoder
‘doing’

‘?’ dmodel Vocabulary


Decoder size
Vocabulary size
hidden
[SOS]
states Sequence
‘I’ size

‘am’ Decoder
‘good’

‘and’ ArgMax
predictions
Linear layer

‘you’

The Predicting Head

You might also like