Sigmoid Functions
Definition
Sigmoid functions are bounded S-shaped real functions, often monotone, with limiting behavior at both extremes. The canonical logistic sigmoid is
which maps real inputs to (0,1).
Why It Matters
Sigmoids are a shared mathematical shape across neural activations, probability response curves, cumulative distribution functions, and bounded growth models. They are useful whenever a system transitions smoothly between regimes rather than growing or changing without limit.
Formalism / Key Objects
- Logistic function:
\sigma(x)=1/(1+e^{-x}). - Derivative:
\sigma'(x)=\sigma(x)(1-\sigma(x)), a bell-shaped curve. - Focused annex: Logistic Sigmoid and Saturating Growth stores the derivative and growth-model equations.
- Inverse logit:
\operatorname{logit}(p)=\log(p/(1-p))forp \in (0,1). - Common variants: hyperbolic tangent, error-function sigmoid, arctangent sigmoid, Gompertz-style curves, and CDFs of common probability distributions.
Connections
- S-Curves and Saturating Growth uses sigmoid-like curves when growth has finite limiting resources.
- Logistic Sigmoid and Saturating Growth records the reusable logistic equations.
- Universal Approximation Theorem historically used sigmoidal activations to prove density of feed-forward networks in spaces of continuous functions.
- In Deep Learning Fundamentals, sigmoids are historically important activations even though modern networks often use ReLU-family functions.
- In Probability and Statistics, sigmoid-shaped CDFs and response curves map latent real-valued quantities into bounded probabilities or proportions.
Common Confusions
- “Sigmoid” can mean a broad S-shaped function class or specifically the logistic function, depending on context.
- A sigmoid curve is not the same as an exponential curve; it may look exponential locally while remaining bounded globally.
- Using a sigmoid activation does not by itself guarantee good training or generalization.