The Multi-Agent Off-Switch Game

Summary

This partial ingest is based on the extracted full PDF text. Agrawal, Ebadian, and Hammond extend the single-agent off-switch game to simultaneous multi-agent settings and ask whether Corrigibility composes. Their core result is negative: agents that each prefer to wait for human approval in their induced single-agent game can have a collectively incorrigible Nash equilibrium once their actions interact through a non-additive joint utility function.

Key Claims

• Single-agent corrigibility can be represented as a preference for wait over both act and off; under Gaussian utility beliefs and a softmax human model, corrigibility holds exactly inside a threshold region controlled by the belief variance and the human irrationality parameter.
• Group corrigibility is defined through pure Nash equilibria: all-wait should be a Nash equilibrium, and in every Nash equilibrium every agent should weakly prefer waiting to acting or switching off.
• Additive joint utilities are a special case where individual corrigibility composes: if f(u_1,u_2)=u_1+u_2, each agent’s preference for waiting is preserved regardless of the other agent’s strategy.
• The additive result is knife-edge. With non-additive utilities, the relevant object becomes an agent’s marginal contribution conditional on another agent acting, so individually corrigible agents can become collectively incorrigible.
• The paper frames multi-agent corrigibility as a mechanism-design problem: architectures, rewards, or coordination mechanisms may need to preserve approximate additivity or otherwise prevent strategic pre-emption of oversight.

Methods / Formalism

• The single-agent off-switch game gives the agent three choices: act, wait, or off. If it waits, the human approves according to a softmax policy

$${\pi }_H(u)=\frac{e^{u/\beta }}{e^{u/\beta }+1},$$

and the agent evaluates wait as E[pi_H(u) u].

• Single-agent corrigibility is:

$$u_A(wait)\ge \max \{u_A(act),u_A(off)\}.$$

• For Gaussian beliefs u~N(mu,sigma^2), the paper proves the threshold |mu| <= sigma^2/(2 beta).
• In the two-agent model, joint utility is determined by a composition function f(u_act1,u_act2). Additive composition preserves the single-agent comparison by cancellation.
• For non-additive f, the paper derives the marginal-contribution test: when agent 2 acts, agent 1’s conditional corrigibility depends on whether z=f(u_act1,u_act2)-u_act2 satisfies the single-agent corrigibility condition under agent 1’s beliefs.
• The formal definitions and theorem schema are pulled into Multi-Agent Off-Switch Game.

Evidence / Experiments

• The paper is primarily formal and illustrative rather than empirical.
• It includes two-agent examples and visualizations showing where Nash equilibria remain all-wait under additive utilities and where acting equilibria emerge under shifted or weighted non-additive utilities.
• The examples are used to show robustness of emergent incorrigibility across several non-additive composition functions, while additive composition is the fragile case.

Connections

• Extends Corrigibility from a single-agent oversight model to Strategic Reasoning about equilibria among multiple agents.
• Updates Safe Multi-Agent Reinforcement Learning by showing that safety properties phrased as individual deferral incentives may fail under collective dynamics.
• Complements Opponent Shaping: both sources show that multi-agent learning or choice cannot be reduced to isolated agent properties.
• Connects to Responsibility Anticipation because both papers evaluate agents before outcomes are fixed, using strategic structure rather than only realized behavior.
• Contrasts with Constrained Markov Potential Games, where potential alignment provides a positive route to equilibrium learning under constraints; here, non-additive interaction can destroy deferral incentives.

Open Questions

• Which concrete coordination mechanisms make all-wait equilibria robust under realistic non-additive utility interactions?
• Does approximate additivity suffice for practical corrigibility guarantees, or are stronger mechanism-design constraints needed?
• How do belief updates, communication, hierarchy, or multiple human principals change the equilibrium structure?
• Can empirical MARL environments be built where shutdown or oversight incentives fail for the same marginal-contribution reason?

Citation

Agrawal, Akash, Soroush Ebadian, and Lewis Hammond. 2026. “The Multi-Agent Off-Switch Game.” In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026). DOI: 10.65109/HQQZ1937.