Gradient Shaker — An optimizer you can feel

Click or drag to place the puck

Iteration0

Loss0.000

Local slope0.000

Best seen0.000

Tune the optimizer

Space run / pause · Esc reset

Learning rate η0.18

Typical toy range 0.10–0.30. Real values depend on the model and optimizer.

Momentum coefficient β0.20

0.80–0.95 is a familiar working range. Too much can oscillate.

Gradient noise σ0.00

Usually start at zero. Mini-batches already introduce structured noise.

The honest bridge

From this toy to real training.

The curve is a map of one part of training—not a tiny replica of the whole thing. These are the exact ideas it stands in for, plus the pieces we have intentionally left outside the frame.

Puck position → parameters

The model’s learnable weights, compressed here to one coordinate on the horizontal axis.

Curve height → loss

A score for how wrong the current model is. Lower points on the curve mean lower loss.

Arrow → gradient

Backpropagation computes which weight changes would raise loss fastest; we go opposite.

Revealed path → updates

The optimizer repeatedly changes weights. One breadcrumb is one training step.

Knobs → hyperparameters

Choices made by the trainer—not learned weights. They shape how learning behaves.

One real training step

The curve visualizes steps 3–5.

Real training must first produce a mistake to learn from. Then the same loop runs again on another batch of examples.

Load a batchExamples and, usually, target answers.

PredictA forward pass through the current model.

Score errorThe loss function turns mistakes into a number.

Trace blameBackprop finds gradients for every weight.

UpdateThe optimizer uses gradients and its settings.

The road ahead→ mini-batches and epochs → train vs. validation curves → overfitting and regularization → schedules, Adam, and real tuning experiments

Field guide · Axolotl

Where the toy becomes a training config.

The visualization uses normalized units. Axolotl uses literal optimizer values, batches, schedules, and regularization. Treat this as a vocabulary map and starting experiment—not a universal recipe.

The formulas underneath

Momentum update vₜ = βvₜ₋₁ − η∇L(θₜ) + σεₜ

Parameter update θₜ₊₁ = θₜ + vₜ

Effective batch B = micro_batch × accumulation × GPUs

η · learning rateBecomes learning_rate; search it logarithmically.

β · momentumAdamW splits this memory into adam_beta1 and adam_beta2.

σ · gradient noiseNo direct Axolotl knob; mini-batch sampling supplies most of it.

Illustrative QLoRA start

Axolotl YAML

reference, not prescription

# optimizer and run shape
adapter: qlora
optimizer: paged_adamw_32bit
learning_rate: 2.0e-4
lr_scheduler: cosine
warmup_ratio: 0.10
weight_decay: 0.0

# effective batch = 2 × 4 × GPU count
micro_batch_size: 2
gradient_accumulation_steps: 4
num_epochs: 4
max_grad_norm: 1.0

Full fine-tune usually starts with a smaller learning rate. LoRA / QLoRA often tolerates a larger adapter learning rate. Validation loss decides whether the configuration generalizes.

Gradient descent through a hidden loss landscape

Tune the optimizer

From this toy to real training.

The curve visualizes steps 3–5.

Where the toy becomes a training config.

Axolotl YAML