Skip to main content
History

The original Transformer was built to translate sentences

In 2017, researchers used attention to connect words across a sentence and reported stronger translation results with a more parallel training process.

8 Nov 2026

Translate a sentence and a word near the end may change how you read one near the beginning. The Transformer, described in a 2017 paper, used a mechanism called attention to let different positions in a sentence weigh information from one another. The authors tested the design by translating between languages.

The original Transformer puts an encoder on the left and a decoder on the right. Both stack attention and feed-forward layers, and the decoder also attends to the encoder output.
The original Transformer architecture for translation. The encoder reads the input sequence and the decoder uses that representation to predict the next output token. · Ashish Vaswani and coauthors, Attention Is All You Need, figure 1. Reproduced under Google’s permission for journalistic use.

Image source · Reuse terms

Earlier systems often processed the input one step at a time, carrying information forward as they went. The Transformer did away with that recurrent structure. Its design made it easier to do more of the work in parallel during training, while adding information about the order in which words appeared.

Translation gave the design an early test

The paper reports results on English-to-German and English-to-French benchmarks. It showed the design could translate well under those test conditions. Its original form took a source sentence in one language and produced a sentence in another, a different job from the open-ended conversations associated with later Transformer models.

The historical connection is in the machinery. A design for deciding which parts of a sequence matter to one another proved useful well beyond the task that first demonstrated it.

Read the source.