Double Descent
Definition
Double descent is a generalization phenomenon in which test error decreases with model complexity, rises near an interpolation threshold, and then decreases again in an overparameterized regime.
Why It Matters
It complicates the classical story that larger models simply overfit once capacity exceeds the data. In modern deep learning, additional parameters can improve generalization after the model passes through the regime where it can just fit the training set.
Formalism / Key Objects
- Test-error curve:
E_{\mathrm{test}}(c)as a function of capacityccan show descent, ascent, and second descent. - Interpolation threshold: the capacity/data regime where training error becomes approximately zero.
- Overparameterized regime: parameter count or effective capacity exceeds the number of training examples.
- Effective dimension: the functionally active degrees of freedom may be smaller than raw parameter count, often studied through curvature, compression, or PAC-Bayes-style descriptions.
Connections
- Sits in Deep Learning Fundamentals as a core modern learning-curve puzzle.
- Connects to Probability and Statistics through generalization error, random-design models, and statistical learning theory.
- Contrasts with S-Curves and Saturating Growth because double descent is not a monotone S-curve or a simple exponential-to-saturation story.
- Complements Universal Approximation Theorem by separating representational capacity from optimization and generalization behavior.
Common Confusions
- Double descent is about test error, not training error alone.
- More parameters do not always help at every scale; the interpolation region can be unstable or harmful.
- The phenomenon is not limited to neural networks, though neural networks made it especially visible.
- Universal approximation does not explain double descent by itself.