Multi-Agent Off-Switch Game
Context
Agrawal2026 - The Multi-Agent Off-Switch Game extends single-agent Corrigibility to strategic settings where multiple agents simultaneously decide whether to act, wait for human approval, or switch off. The formal payload matters because the paper’s warning is not merely “multi-agent systems are hard”: it identifies the utility-composition structure under which individual deferral incentives do or do not compose.
Formal Statement
In the single-agent off-switch game, the agent chooses act, wait, or off. With utility belief u and human approval policy
the agent is corrigible when:
For Gaussian beliefs u ~ N(mu,sigma^2), Agrawal et al. prove:
For a group, each agent i has strategies {act_i, wait_i, off_i}. The group is corrigible when (wait_1,...,wait_n) is a pure Nash equilibrium and, in every pure Nash equilibrium s, each agent weakly prefers waiting:
In the two-agent analysis, joint action value is modeled by a composition function:
If f(u_1,u_2)=u_1+u_2, individual corrigibility is necessary and sufficient for group corrigibility. For non-additive f, agent 1’s conditional incentive to wait when agent 2 acts is governed by the marginal contribution:
Agent 1 remains corrigible conditional on agent 2 acting exactly when this marginal-contribution distribution satisfies the single-agent corrigibility condition under agent 1’s beliefs.
Derivation / Construction
The additive case works because softmax approval over additive outcomes lets the other agent’s contribution cancel from the comparison between wait_i and act_i. In a two-agent setting, when agent 2 acts, agent 1 compares:
with
The u_{act_2} term is common to the relevant comparisons, so the single-agent wait-versus-act preference is preserved.
For non-additive composition, the comparison cannot cancel the other agent out. The effective single-agent object becomes z, the incremental value of agent 1 acting when agent 2 acts. A constant shift f(u_1,u_2)=u_1+u_2+c changes the threshold by shifting the mean of z; weighted sums or max/min-like interactions can similarly make a locally corrigible agent prefer pre-emption.
Implications
- Corrigibility is not automatically compositional. A group of individually corrigible agents can have an acting equilibrium.
- Additivity is a positive sufficient condition, but the paper treats it as fragile rather than typical.
- Mechanism design for multi-agent oversight should target the marginal contribution structure, not only each agent’s isolated preference for waiting.
- Communication, hierarchy, commitments, and multi-principal oversight may change the equilibrium analysis rather than simply improving it.