Neural Machine Translation (Italian to English)

A Transformer-based neural machine translation system for Italian-to-English, with the Transformer implemented from scratch in PyTorch.

Overview

This project implements the encoder-decoder Transformer architecture (Vaswani et al., "Attention Is All You Need") directly in PyTorch, without relying on a pre-built Transformer library. It uses SentencePiece for subword tokenization and supports both greedy and beam-search decoding.

A trained checkpoint (saved_model/best_ckpt.tar) and the SentencePiece models (sp/) are included, so translation can be run without retraining. It is a proof of concept: the model is comparatively small and trained for a limited number of epochs, so output quality is uneven (see Limitations).

Model

Defined in src/transformer.py and src/layers.py, configured in src/constants.py:

• Encoder-decoder Transformer, 6 layers each.
• Model dimension d_model = 512, 8 attention heads, feed-forward size 2048.
• Multi-head scaled dot-product self-attention and cross-attention, implemented from scratch.
• Sinusoidal positional encoding, layer normalization, and residual connections; dropout rate 0.1.
• LogSoftmax output over the target vocabulary, trained with NLLLoss.

Tokenization

• SentencePiece (unigram model), vocabulary size 16000.
• Reserved token IDs: pad (0), sos (1), eos (2), unk (3).
• Trained source and target models are committed under sp/ (src_sp.model / trg_sp.model and their .vocab files).

Training data

The included checkpoint was trained on the Europarl v7 Italian-English parallel corpus (https://www.statmt.org/europarl/), a corpus of European Parliament proceedings. Because the corpus is formal and domain-specific, the model performs best on well-structured, formal Italian.

Training defaults (src/constants.py): Adam optimizer, learning rate 1e-4, batch size 80, maximum sequence length 200, 10 epochs. The checkpoint with the lowest validation loss is saved to saved_model/best_ckpt.tar.

Repository structure

.
├── requirements.txt
├── saved_model/                # Trained checkpoint: best_ckpt.tar
├── sp/                         # SentencePiece models and vocabularies
│   ├── src_sp.model / .vocab
│   └── trg_sp.model / .vocab
└── src/
    ├── run.py                  # Inference entry point + training loop (Manager)
    ├── constants.py            # Paths, hyperparameters, special tokens
    ├── custom_data.py          # Tokenization, padding, Dataset / DataLoader
    ├── data_structure.py       # BeamNode and priority queue for beam search
    ├── transformer.py          # Transformer model (encoder / decoder)
    ├── layers.py               # Attention, feed-forward, positional encoding
    └── sentencepiece_train.py  # Train SentencePiece models and split data

Setup

pip install -r requirements.txt

Dependencies: PyTorch, SentencePiece, NumPy, tqdm, scikit-learn, torchtext.

Inference

Run from the src/ directory (paths in constants.py are relative to it). Beam search:

python run.py --model best_ckpt.tar --input "come stai?" --decode beam

Greedy decoding:

python run.py --model best_ckpt.tar --input "come stai?" --decode greedy

Arguments:

• --model — checkpoint file name inside saved_model/.
• --input — Italian sentence to translate.
• --decode — beam or greedy (beam size is 8, set in constants.py).

Training

To train from scratch you need to supply your own parallel corpus:

1. Place raw parallel text at data/raw_data.src (Italian) and data/raw_data.trg (English), one sentence per line.
2. Run python src/sentencepiece_train.py to train the SentencePiece models and split the data into data/src/{train,valid}.txt and data/trg/{train,valid}.txt.
3. The training loop lives in the Manager class in src/run.py (Manager(is_train=True).train()); the committed run.py entry point runs inference. Multi-GPU training is supported via nn.DataParallel when more than one GPU is available.

How it works

Inference tokenizes the input with SentencePiece, runs it through the encoder, then decodes with either greedy search (argmax at each step) or beam search (beam size 8, length-normalized scores). The decoded token IDs are converted back to text with the target SentencePiece model.

Limitations

This is a proof of concept. The model is small and trained for a limited number of epochs, so translations can contain repeated tokens, weak punctuation, or hallucinated words, and quality is best on formal Italian. Example outputs:

Input	Beam output (truncated)
come stai?	How are you?
sto cercando lavoro	I am looking for work ...
come sta Signor Rossi?	How is Mr Rossi? ...

Possible improvements

• Train on a larger, more general corpus (for example the Tatoeba Challenge).
• Add repetition or end-of-sequence penalties to reduce hallucination.
• Compress the model for faster inference.
• Add a simple web interface for interactive translation.

License

Released under the MIT License. See LICENSE.