Transformer

Tap it to watch a prepared next-word example. Try the Encoder–decoder diagram in the Splash tab.

A transformer is a neural network first described in 2017, for translating text. It splits text into tokens, and in each layer every token uses attention: it looks at all the others and weighs how much each one matters to it. Language models built this way predict the next token.

Tap it: arcs jump between THE CAT SAT ON, thicker where attention is stronger, one color for each of two heads; the tiles rise through the layers, and MAT drops in. The Encoder–decoder diagram in the Splash tab is the classic design. Its key: an eye is attention; a lidded eye, masked attention (it only looks back); +, add and norm; », feed forward; a vector, embedding; a wave, positional encoding; a slash, linear; bars, softmax; N×, repeated N times. HELLO WORLD goes in, and given START HOLA, the prepared example ends with MUNDO. The words and attention strengths are prepared; this view does not calculate attention or predict text.

The classic encoder-decoder diagram names the main parts of the 2017 transformer correctly. The words and attention arcs are prepared examples; no attention matrix, translation, or next-word inference is calculated.

1. Correct

   The classic diagram follows the 2017 encoder-decoder architecture.¹

   The box labels include embedding, position, masked self-attention, encoder attention, add and norm, feed-forward, linear, and softmax.

2. Simplified

   The splash presents prepared next-word and translation examples.¹

   MAT and the MUNDO tile label are literal outputs, and arc strengths are stored. The How-to and About now say that these are prepared examples.

3. Simplified

   The first attention picture is schematic rather than a causal decoder calculation.¹

   Some arcs lead to later token positions; there is no QK product, mask, normalization, or learned attention value.

1. Attention Is All You Need, Vaswani and coauthors, 2017

Simplified

MAT and MUNDO are stored outputs. Arc strengths are fixed, not attention probabilities. The first picture permits links to later tokens, so it should not be read as causal decoder attention.

Made in Splashery. Its code is MIT.

This link opens it in the Playground. To put it on your own page, add the element and one script tag. Every option.

https://splashery.app/play/?toy=transformer

<script type="module" src="https://splashery.app/play/src/element.js"></script>
<splashery-toy toy="transformer"></splashery-toy>