Double descent
Summary
This partial ingest is based on a clipped encyclopedia page. It summarizes double descent as the learning-curve phenomenon where test error first decreases, then peaks near an interpolation threshold, and then decreases again as model capacity continues to increase.
Key Claims
- Double descent challenges the classical expectation that adding too many parameters simply worsens overfitting.
- The error peak often appears near the interpolation threshold, where the training procedure can nearly fit the training data exactly.
- The phenomenon has been observed in linear regression, neural networks, and broader overparameterized learning systems.
- Some explanations appeal to effective dimension: the number of functionally relevant degrees of freedom can be smaller than the raw parameter count.
Methods / Formalism
- Informal model-complexity curve: test error as a function of capacity can follow a descent-then-ascent-then-descent pattern.
- Interpolation threshold: the capacity/data regime where training error reaches approximately zero.
- The clipped reference points to linear-regression, replica-trick, Hessian/effective-dimension, and PAC-Bayes compression analyses as theoretical routes.
Evidence / Experiments
- The source is a reference page, not an original experiment.
- It cites empirical and theoretical work including Belkin et al., Nakkiran et al., Rocks et al., and later demystification/explanation papers.
Connections
- Core reference source for Double Descent.
- Sits in Deep Learning Fundamentals as a modern generalization puzzle for overparameterized models.
- Connects to Probability and Statistics through generalization error, bias-variance reasoning, random covariates, and statistical learning theory.
- Useful next to S-Curves and Saturating Growth as a reminder that not every learning or scaling curve is monotone logistic or monotone exponential.
Open Questions
- Which mechanisms explain double descent in a given setting: interpolation, label noise, optimization bias, effective dimension, or data/model mismatch?
- How should double descent affect model-size and data-size choices in practical training pipelines?
- When do modern scaling laws smooth over the interpolation peak rather than showing visible double descent?
Citation
Wikipedia contributors. (2026). Double descent. Wikipedia.