Next-word prediction
Train
Generate
Data
Data
Corpus
TinyStories
Tokens
Words
Byte-pair · 1,000 tokens
Byte-pair · 500 tokens
Byte-pair · 200 tokens
Characters
Min. count
Source
TinyStories (Eldan & Li 2023), first million words of the validation split
Model
Model
Window
Dimension
2
3
4
8
16
32
Training
Training
Adam rate
Batch
words
Seed
View
View
Sentence arcs
Network connections
Live
Edit
Fast
Initializing WebGPU…
Run
Speed
1 words/s
2 words/s
5 words/s
Batches, as fast as possible
Epoch
—
Words
—
Batches
0
Perplexity
—
Loss
—