Word2vec
Skip-gram
CBOW
Network
Data
Data
Corpus
TinyStories
TinyCode (Python)
Min. count
Source
TinyStories (Eldan & Li 2023), first million words of the validation split
Model
Model
Dimension
2
3
4
8
16
32
50
64
Window
Negatives
Subsampling
Off
10⁻²
10⁻³
10⁻⁴
Training
Training
Learning rate
Epochs
Seed
View
View
Sentence arcs
Probability
σ(score) · as trained
Softmax over vocabulary
Run
Speed
1 words/s
2 words/s
5 words/s
20 words/s
100 words/s
1,000 words/s
10,000 words/s
100,000 words/s
As fast as possible
Epoch
—
Words
—
Rate
—
Loss
—