wiki

Multi-Agent Constrained Policy Optimisation

3 min read

Multi-Agent Constrained Policy Optimisation

Summary

This partial ingest is based on the extracted full PDF text. Gu et al. formulate safe MARL as a constrained Markov game and extend Constrained Policy Optimization and multi-agent trust-region learning into two model-free safe MARL algorithms: MACPO and MAPPO-Lagrangian. The paper also introduces Safe Multi-Agent MuJoCo and Safe Multi-Agent Robosuite benchmark suites for continuous-control safe MARL.

Key Claims

  • Safe MARL requires each agent to respect its own cost constraints while coordinating so the joint behavior remains safe.
  • Sequential multi-agent trust-region updates can decompose a joint advantage into per-agent updates without assuming value-function factorization.
  • The theoretical safe policy-iteration procedure guarantees monotonic reward improvement and satisfaction of each agent’s safety constraints at every iteration.
  • MACPO gives a hard constrained trust-region approximation, while MAPPO-Lagrangian gives a softer PPO-style alternative.
  • The benchmark results show MACPO and MAPPO-Lagrangian rapidly lowering costs and keeping exploration near or inside feasible regions, while unconstrained MARL baselines violate constraints.

Methods / Formalism

  • The constrained Markov game is N,S,A,p,rho_0,gamma,R,C,c, with joint reward R, per-agent cost functions C_j^i:S x A_i -> R, and thresholds c_j^i.
  • The paper uses multi-agent advantage decomposition to express a joint surrogate advantage as a sum of sequentially unfolded per-agent surrogate advantages.
  • For agent i and cost j, a cost surrogate L_j^{i,pi}(bar pi^i) bounds how the expected cost changes when policies stay close in KL distance.
  • Theorem 4.4 states that the safe multi-agent policy-iteration sequence has J(pi_{k+1}) >= J(pi_k) and satisfies J_j^i(pi_k) <= c_j^i for all agents and constraints.
  • MACPO relaxes max-KL constraints into expected KL constraints and solves a per-agent constrained quadratic approximation. MACPO Sequential Trust Region records the core update and theorem.

Evidence / Experiments

  • The paper evaluates on Safe Multi-Agent MuJoCo and Safe Multi-Agent Robosuite, including ManyAgent Ant, Ant, HalfCheetah, and two-arm peg-in-hole variants.
  • Baselines include IPPO, MAPPO, and HAPPO; safe methods include MACPO and MAPPO-Lagrangian.
  • Reported curves show MACPO and MAPPO-Lagrangian achieving near-zero or consistently low costs across tasks, whereas unconstrained baselines often achieve reward by violating constraints.
  • MACPO’s hard constrained update decreases cost more stably early in training; MAPPO-Lagrangian can lag early because a fixed number of gradient-ascent steps may not solve the constrained problem exactly.
  • The authors release benchmark environments and code for reproducibility.

Connections

Open Questions

  • How robust are the per-iteration safety claims under neural approximation, finite sampling, partial observability, and decentralized execution?
  • When should safe MARL use constrained optimization inside the learner rather than an external Shielding layer?
  • Are the SMAMuJoCo and SMARobosuite constraints representative of deployment safety, or mainly useful as reproducible algorithmic stress tests?

Citation

Gu, Shangding, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. 2022. “Multi-Agent Constrained Policy Optimisation.” In Proceedings of the International Conference on Machine Learning.

Graph View

Backlinks