wiki

An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks

4 min read

An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks

Summary

Papadopoulos et al. argue that cooperative MARL evaluation has over-relied on benchmarks such as SMAC and Google Research Football, which include team-game dynamics but do not adequately test fully cooperative real-world requirements such as multi-robot coordination, warehouse/resource management, search and rescue, and human-AI cooperation. They benchmark eleven MARL algorithms across a broader set of complex fully cooperative tasks, including high-dimensional image observations, and release PyMARLzoo+ as an extension of (E)PyMARL with broader benchmark support.

This is a partial ingest based on the full extracted text, with emphasis on benchmark design, empirical findings, and baseline implications.

Key Claims

  • Cooperative MARL benchmarks should include tasks with excessive coordination, joint exploration, sparse reward, collision dynamics, multi-step resource dependencies, and high-dimensional observations.
  • Algorithms that are strong on SMAC or GRF can underperform standard baselines on fully cooperative benchmarks.
  • Training time and computational footprint should be reported alongside reward, because practical algorithm choice depends heavily on wall-clock and resource cost.
  • Image-based MARL observations deserve explicit evaluation; performance on low-dimensional state vectors does not transfer automatically.
  • PyMARLzoo+ lowers the cost of testing algorithms across PettingZoo, Overcooked, PressurePlate, Capture Target, Box Pushing, and existing (E)PyMARL-compatible environments.

Methods / Formalism

  • Fully cooperative MARL is formulated as a Dec-POMDP

where A=A_1 x ... x A_N is the joint action space, P(s' | s,a) is the transition function, r(s,a) is the shared reward, F_i(s) gives each agent’s observation, and each agent conditions its policy pi_i(a_i^t | h_i^t) on an action-observation history.

  • The benchmark suite covers LBF, RWARE, MPE/Spread, PettingZoo tasks such as Entombed Cooperative, Pistonball, and Cooperative Pong, plus Overcooked, PressurePlate, Capture Target, and Box Pushing.
  • Algorithms include QMIX, MAA2C, COMA, MAPPO, QPLEX, HAPPO, MAT-DEC, EOI, CDS, MASER, and EMC.
  • For image observations, the study compares frozen pre-trained encoders such as ResNet18 with trainable CNN-based encoders.
  • Cooperative MARL Baseline Formalisms records the shared Dec-POMDP baseline and benchmark-result matrix that this source helps motivate.

Evidence / Experiments

  • Standard MARL algorithms QPLEX, MAPPO, MAA2C, and CDS show the most consistent performance across the fully cooperative tasks.
  • Exploration-focused algorithms with strong SMAC/GRF results, including EOI, EMC, and MASER, often underperform on the broader fully cooperative benchmark set; CDS is the main exception among that family.
  • Value-decomposition methods can degrade as the number of agents or coordination complexity rises. QPLEX improves over QMIX in several settings but still fails in difficult LBF and Overcooked tasks.
  • Actor-critic methods such as MAPPO and MAA2C are strong in several Spread, LBF, Overcooked, and PressurePlate cases, but no single algorithm dominates all settings.
  • In Pistonball, a frozen ResNet18 encoder gives higher and more stable performance with much lower resource cost than a trainable CNN in the reported comparison.
  • The paper reports wall-clock training times and highlights that off-policy methods can take longer due to replay-buffer costs, while image encoders introduce their own GPU/compute tradeoffs.

Connections

  • Strengthens Cooperative MARL Benchmarking with a concrete benchmark-diversity agenda and an implementation artifact.
  • Updates Cooperative Multi-Agent Reinforcement Learning by showing that strong baselines depend on the task family; “SoTA on SMAC” is not enough evidence for general cooperative competence.
  • Adds measurement discipline to AI Evaluation and Benchmarking: benchmark breadth, observation modality, training time, and implementation parity all affect capability claims.
  • Relates to Benchmark Gaming in a softer sense: over-specializing to a narrow benchmark family can inflate perceived algorithmic progress even without deliberate evaluator exploitation.
  • Provides baseline context for Safe Multi-Agent Reinforcement Learning, where safety interventions should be compared against strong, diverse cooperative baselines rather than a narrow SMAC/GRF slice.

Open Questions

  • Which fully cooperative tasks best predict performance in robotics, warehouse, or human-AI coordination deployments?
  • How should benchmark suites balance task diversity against reproducibility and computational budget?
  • Are image encoders measuring visual representation quality, cooperative learning quality, or both?
  • What minimum benchmark matrix should a new safe-MARL or cooperative-MARL method report?

Citation

Papadopoulos, George, Andreas Kontogiannis, Foteini Papadopoulou, Chaido Poulianou, Ioannis Koumentis, and George Vouros. 2025. “An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks.” AAMAS 2025 / arXiv:2502.04773.

Graph View

Backlinks