Transformer¶

Start tokens, vectors, attention, positions and residuals, then one small implementation. A research paper is a later reference rather than the full prerequisite course.

Selected resources¶

Core mechanistic intuition · Soft routing & tensor flow¶

• Intuition: Self-attention is fundamentally content-based dynamic weighted information retrieval (soft routing). Each token projects a Query (what information am I seeking?) and a Key (what features do I contain?), computing attention affinities via inner products to aggregate Values (what information payload do I carry?).
• Minimal tensor pipeline:
  • Input embedding: \(X \in \mathbb{R}^{B \times T \times D}\)
  • Linear projections: \(Q = X W_Q,\; K = X W_K,\; V = X W_V \in \mathbb{R}^{B \times T \times D}\)
  • Attention map: \(A = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) \in \mathbb{R}^{B \times T \times T}\) (explicit all-to-all dependency graph)
  • Value aggregation & output projection: \(O = A V W_O \in \mathbb{R}^{B \times T \times D}\)
• Trade-offs: Capturing unrestricted global dependencies requires \(O(T^2)\) pairwise interactions. In autoregressive inference, the memory bottleneck shifts from compute to KV Cache memory footprint and memory bandwidth limits.

Check yourself · Critical thinking¶

• What is the mathematical purpose of the scaling factor \(\sqrt{d_k}\)? What happens to the Softmax gradient when \(d_k\) is large without scaling?
• How does a causal mask operate at the matrix arithmetic level, and why is an architectural mask necessary rather than relying solely on the loss function?
• Why does Pre-LN (normalizing before the sub-layer) offer significantly smoother gradient flow in deep networks compared to original Post-LN?

Revisit papers and historical bookmarks after implementing the small model.

Related topics¶