| Pitfall | Solution | |---------|----------| | Loss not decreasing | Check that causal mask is applied correctly. Verify learning rate (start with 3e-4 for AdamW). | | Exploding gradients | Add gradient clipping ( torch.nn.utils.clip_grad_norm_ (model.parameters(), 1.0) ). | | Model only repeats common phrases | Increase embedding size or add dropout (0.1). | | Out-of-memory on GPU | Use gradient accumulation (simulate larger batch size) or reduce sequence length from 512 to 256. |
A token is an integer. An embedding converts that integer into a dense vector of size d_model (e.g., 512). Since attention mechanisms are permutation-invariant, we must inject position information. build a large language model %28from scratch%29 pdf