Multi-Agent PPO

Definition

Multi-agent PPO adapts proximal policy optimization to multi-agent settings, commonly through independent local PPO learners or decentralized actors trained with a centralized critic.

Why It Matters

MAPPO and IPPO are strong practical baselines for cooperative MARL. They also form the optimization substrate for several constrained or safety-aware MARL methods, so safe-MARL claims should account for how implementation-sensitive these baselines are.

Formalism / Key Objects

• In a shared-reward DEC-POMDP, MAPPO usually trains local actors pi_i(a_i | o_i) and a centralized value function V(s, o_1,\ldots,o_n) or other global critic input.
• IPPO trains PPO-style local actor/value functions without centralized value inputs.
• The PPO clipped surrogate has the single-agent schematic form

$$L^{clip}(\theta )=E_t[\min (r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ)A_t)].$$

• Yu et al. identify five high-impact implementation choices for cooperative MARL: value normalization, value-function input representation, limited data reuse, smaller clipping ranges than default PPO, and tuned large batch sizes.
• Cooperative MARL Baseline Formalisms records the shared-reward DEC-POMDP setup and contrasts MAPPO/IPPO with additive value decomposition.

Connections

• Cooperative Multi-Agent Reinforcement Learning is the main setting for MAPPO/IPPO benchmark studies.
• Value Decomposition Networks are an off-policy value-factorization comparison point.
• Multi-Agent Non-Stationarity explains why repeated PPO epochs and minibatch reuse can degrade performance in hard cooperative tasks.
• MACPO Sequential Trust Region and Gu2022 - Multi-Agent Constrained Policy Optimisation extend MAPPO-style baselines toward constrained safe MARL.
• AI Evaluation and Benchmarking matters because MAPPO’s reputation depends heavily on fair benchmark implementation.

Common Confusions

• MAPPO is not simply PPO run independently per agent; it typically uses decentralized actors with centralized value information during training.
• IPPO can be strong even without centralized critic inputs, especially in homogeneous shared-reward benchmarks.
• PPO’s on-policy nature does not automatically make it weak in cooperative MARL; implementation details can dominate the comparison.

Key Sources

• Yu2022 - The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games
• Gu2022 - Multi-Agent Constrained Policy Optimisation