Q-learning cart

Game

Episode 1

← Left

Coast

Right →

Learning rate 0.0010 Discount γ 0.980 Exploration ε 1.00 Reward Speed 1.0× Game size 100% Fast-forward (skip animation) Show where the ball will land

0 episodes

Caught 0

Missed 0

Success (last 50) –

Avg reward (last 50) –

caught missed average of last 50

positive weight negative weight

s = (bx, by, bvx, bvy, cx, cv) · actions: left, coast, right

act: random with prob. ε, else argmax_a Q(s, a)

target y = r + γ · Q_target(s′, argmax_a′ Q(s′, a′)) · y = r at the end

update: minimise Huber(Q(s, a) − y) with Adam

r = 0 until the ball arrives · replay memory 20 000 steps, batch 32 · target net copied every 300 updates · ε × 0.99 per episode (floor 0.05)