Disentangling Interaction using Maximum Entropy Reinforcement Learning in Multi-Agent Systems

Summary

Rother, Weisswange, and Peters introduce Action Aligned Interaction Learning (AAIL), a framework for “coexistence” environments where agents have their own tasks but can help or hinder nearby agents with independent goals. The core move is to learn task policies and interaction-impact policies separately with maximum-entropy RL, then recombine their Q-values at deployment using entropy and Jensen-Shannon-distance weights.

This is a partial ingest based on the full extracted text, with emphasis on the formal construction, experiments, and wiki connections.

Key Claims

• Many HRI and multi-agent settings are neither fully cooperative nor zero-sum: agents may coexist while pursuing independent tasks, so a robot should preserve its own task performance while avoiding unnecessary negative impact on others.
• Training a joint policy for every task-agent combination scales combinatorially, while separate task and impact policies can scale roughly linearly in the number of tasks and agents.
• Maximum-entropy RL makes policy blending tractable because soft Q-functions can be combined as energy-style components.
• Entropy can indicate when the task policy is decisive, and Jensen-Shannon distance can weight impact policies by how compatible their action distributions are with the current task policy.
• AAIL avoids a failure mode of naive impact blending: if impact policies are over-weighted, the agent may help others at the expense of completing its own task.

Methods / Formalism

• The paper uses stochastic-game notation with agents N, state space S, agent action spaces A_i, transition kernel P, per-agent rewards r_i, and discount factor gamma.
• Individual success is the accumulated per-agent return R_i = sum_t r_t^i; coexistence success is measured by the sum of agents’ returns in an episode.
• A “task policy” solves the ego task. An “impact policy” is trained to improve the value of another agent’s task.
• Maximum-entropy RL optimizes

$$\sum _{t=0}^T{E}_{(s_t,a_t)\sim {\rho }_{\pi }}\left[r(s_t,a_t)+\alpha H(\pi (\cdot \mid s_t))\right].$$

• The impact of agent i’s action on agent j is represented as a value difference:

$$I_i^j(s_t,a_t^i)=V^j(s_{t+1})-V^j(s_t).$$

• AAIL learns impact-aware Q-functions with impact as the reward signal, then composes a compound Q-function:

$$Q_C=w_t{Q}_t+(1-w_t)\sum _i\frac{w_i}{{\sum }_j{w}_j}{Q}_i.$$

• The task weight is high when the task policy is low-entropy; impact weights use the Jensen-Shannon distance between the task distribution and impact-policy distributions. See Action-Aligned Interaction Learning for the focused construction.
• Scope assumptions include discrete actions, known or estimated other-agent tasks, and enough internal task models to estimate another agent’s value function.

Evidence / Experiments

• In a particle coexistence environment, the authors train 35 task policies and 35 corresponding impact policies. They report that a joint learner would need 52,521,875 networks to cover the same variety of task combinations.
• The method is tested against a single-task learner and a non-aligned interaction learner (NAIL). AAIL keeps more task discipline than NAIL when task and impact objectives conflict.
• In aligned-goal scenarios, AAIL and NAIL can improve team reward over single-task learning; in non-aligned or opposing-goal scenarios, AAIL avoids NAIL’s tendency to sacrifice the ego task.
• The paper also introduces a cooking-zoo environment inspired by Overcooked/cooking-gym. In a two-agent cooking scenario with PPO-controlled partners, AAIL slightly reduces the controlled agent’s own dish completion rate but substantially improves the partner’s progress, supporting the coexistence framing.

Connections

• Extends Multi-Agent Coordination beyond shared-reward coordination: agents need to coordinate spatially and behaviorally without assuming a common objective.
• Adjacent to Ad Hoc Teamwork because both settings lack prior coordination, but coexistence does not assume a shared team reward.
• Adds a policy-composition mechanism to Cooperative Multi-Agent Reinforcement Learning that should be compared with shared-reward baselines rather than treated as a direct replacement.
• Relates to Multi-Agent Non-Stationarity because other agents’ goals and policy responses determine the value of impact-aware actions.
• Complements Wang2020 - Too Many Cooks: Coordinating Multi-agent Collaboration Through Inverse Planning: Wang et al. infer subtask allocation, while Rother et al. learn impact policies and recombine them with task policies.

Open Questions

• How robust is the entropy/JSD blending rule when other agents’ goals are inferred incorrectly rather than known?
• Can the approach be extended cleanly to continuous action spaces without losing the product-of-experts intuition?
• Does positive “impact” preserve human preferences in realistic HRI, or can it create manipulation-like behavior by optimizing another agent’s task model?
• Which benchmark families best distinguish coexistence from fully cooperative ad hoc teamwork?

Citation

Rother, David, Thomas H. Weisswange, and Jan Peters. 2023. “Disentangling Interaction using Maximum Entropy Reinforcement Learning in Multi-Agent Systems.” ECAI 2023.