Learning Fundamentals, Optimization & Classical ML¶
This is shared vocabulary. Use small experiments to diagnose probability, optimization and evaluation gaps while reusing your mathematics/engineering background. P0 is shared foundation, P1 orientation, P2 specialization.
Map and priority¶
| Depth | Topic | Understand |
|---|---|---|
| P0 | Mathematics and statistics | Vectors, matrices, derivatives, conditional probability, likelihood and moments; association vs. causation |
| P0 | Learning and data | Supervision, loss, train/validation/test, generalization, leakage and imbalance |
| P0 | Optimization and networks | Backprop, SGD/Adam, schedules, normalization, regularization and initialization |
| P1 | Classical methods | Linear/logistic models, SVM, trees/ensembles/boosting, clustering and PCA; baseline first |
| P2 | Probabilistic and causal modeling | Bayesian inference, graphical models, interventions and identifiability as needed |
Core mechanistic intuition · From classical ML to neural nets¶
- Feature space & inductive bias: Linear models assume hyperplane separation; decision trees perform axis-aligned orthogonal partitions (invariant to monotonic feature scaling); kernel SVMs map to implicit reproducing Hilbert spaces; neural networks learn continuous coordinate warpings via layered function compositions.
- Optimization & backprop: Backpropagation is dynamic programming caching intermediate jacobians of the chain rule. Gradient descent stability depends on the curvature (Hessian conditioning) of the loss surface—the primary reason learning rate warmup, normalization layers, and adaptive moment estimation (Adam) are required.
Selected resources¶
-
Backbone · Karpathy Zero to Hero
Code-first backbone to fill gaps in loss, gradient backprop, training loops and validation.
-
Official experiment · scikit-learn classifier comparison
Inspect decision boundaries and variance across linear, kernel and tree models on shared 2D datasets.
Guided experiment · AI-1 · Baselines and overfitting¶
- Run the classifier comparison; verify that preprocessing fits training data only.
- Keep one linear and one tree model; vary only noise or model complexity.
- Tune on validation and report test once; use grouped/temporal splits when required.
- Show learning curves, errors and at least two appropriate metrics.
Acceptance: CPU is sufficient. Record three settings and explain bias, variance, leakage and metric choices.
Check yourself · Critical thinking¶
- Why are tree-based models largely invariant to feature scaling, whereas gradient-based neural networks and distance metrics (KNN/SVM) require strict normalization?
- In highly imbalanced tasks (e.g., 99:1), why is accuracy misleading, and how do precision-recall AUC and cost-sensitive losses resolve this?
- When splitting data with temporal correlation or sensor-group clustering, why does naive random shuffling cause optimistic performance leakage?
See trees and ensembles; use evaluation throughout.