Attention
Data
Data
Sentence
Model
Model
Dimension
4
8
12
16
Heads
1
2
4
Query/key size
2
4
Causal
Only earlier words
Sharpness
× 1.00
Value weights
Random
Identity
Seed
View
View
Queries and keys
Score matrices
Weight matrices (A)
Within-word connections
Attention connections (across words)
Curved connections
Bundle the 2-D connections
Color neurons by value
2-D
3-D
Front
Within words
Across words
One attention layer, drawn as a dense layer · press 3-D to unfold it