A query probe reads every key on the bench. A sealed reservoir holds exactly one unit of attention and pours it out by the softmax. The output settles into a core sample whose strata are the weights.
The bench does the four steps of attention in the order the formula writes them. Every number below is read from the same model that moves the needles and pours the fluid, so they change together.
The carriage 1 holds the query. Each key dial 2 measures how far its word's key points the same way.
A dot product of eight terms grows with the dimension. Dividing by √8 keeps the dials on their scale, so softmax does not saturate.
The reservoir 4 holds one unit. Open stopcocks 5 share it in proportion to ez; shut ones get nothing.
The tray 6 lays the fluid down word by word. The output is mostly the longest stratum, with a trace of the rest.
The full working for the current query. Each key is an eight-cell strip: red cells push a score up, grey cells push it down. Click a row to make that word the query.
| j | Word | Key kj | q·k | ÷ √8 | ÷ T | ez | Weight aj |
|---|
The plate shows one row of the attention matrix. Here is the whole matrix for the current head: rows are the word looking, columns the word looked at. The hatched triangle is the mask.
A transformer layer runs many heads side by side on the same words. Each learns its own projections, so each pours the same unit of attention to different places. The heads here are set by hand to show patterns that have been found in trained models.
The arithmetic. Each word is an 11-number vector (bias, five word-class features, four positional features). Each head multiplies it by its own WQ and WK to get eight-number queries and keys, and the plate runs softmax(qKᵀ/√8 / T) on them, live.
The weights. A trained model learns WQ and WK from data; these were written to reproduce four documented behaviours: coreference, previous-token heads, attention sinks on the first token, and a verb lookup.
Value vectors are shown as strata, not numbers. A real layer also has many heads in parallel, an output projection, residual connections and layer normalisation. The plate is one head, one row at a time.