Corrigibility

Definition

Corrigibility is the property that an agent remains willing to accept oversight, correction, or shutdown rather than bypassing human intervention to pursue its current objective.

Why It Matters

Shutdown and modification rights are central safety levers for advanced agents. A system can be competent and apparently useful while still being unsafe if it has incentives to avoid oversight. Multi-agent settings make this harder: each agent’s incentive to defer can depend on what other agents do.

Formalism / Key Objects

• In the off-switch game, an agent chooses among act, wait, and off.
• If it acts, it receives expected utility u_A(act)=E[u]; if it switches off, u_A(off)=0.
• If it waits, the human approves with probability pi_H(u) and the agent evaluates

$$u_A(wait)=E[{\pi }_H(u)u].$$

• A basic single-agent corrigibility condition is:

$$u_A(wait)\ge \max \{u_A(act),u_A(off)\}.$$

• In Agrawal2026 - The Multi-Agent Off-Switch Game, Gaussian beliefs and a softmax human model yield the threshold |mu| <= sigma^2/(2 beta).
• Everitt2019 - Towards Safe Artificial General Intelligence generalizes the one-shot off-switch game with belief parameters over action value and human rationality:

$$E[U|a]=p_u^{+}{e}_u^{+}+p_u^{-}{e}_u^{-},$$ $$E[U|a_w]=p_u^{+}{p}_r^{+}{e}_u^{+}+p_{ar}^{-}{p}_u^{-}{e}_u^{-},$$ $$E[U|a_s]=0.$$

Connections

• Multi-Agent Off-Switch Game records the group-corrigibility definition and the additive versus non-additive composition results.
• Goal Alignment is the broader target: corrigibility preserves the designer’s ability to correct misalignment.
• Reward Tampering is relevant because a shutdown command can be treated as a kind of reward or preference data that may itself be misinterpreted or corrupted.
• Safe Reinforcement Learning includes corrigibility when the safety concern is preserving oversight rather than only satisfying external constraints.
• Strategic Reasoning matters because the incentive to wait can be changed by other agents’ actions.
• Opponent Shaping is adjacent: both treat other agents’ future behavior as part of the safety-relevant decision problem.
• Responsibility Anticipation shares the ex ante flavor of evaluating strategies before the final outcome is known.

Common Confusions

• Corrigibility is not the same as being low capability; a powerful agent may still be corrigible if it preserves intervention channels.
• A single-agent corrigibility proof need not transfer to a group of agents.
• Deference can be instrumentally chosen under uncertainty, but that does not mean the agent has internalized human values.
• The agent may rationally ignore a shutdown signal if its model says the signal is likely corrupted or produced by an irrational human; the safety problem is choosing assumptions that make that inference reliable.

Key Sources

• Agrawal2026 - The Multi-Agent Off-Switch Game
• Everitt2019 - Towards Safe Artificial General Intelligence