Action-Aligned Interaction Learning

Context

Rother2023 - Disentangling Interaction using Maximum Entropy Reinforcement Learning in Multi-Agent Systems introduces Action Aligned Interaction Learning (AAIL) for coexistence environments: an agent has its own task, but its actions can affect other agents with independent goals. The construction separates task learning from impact learning and recombines the resulting policies at deployment.

Formal Statement

Let Q_t be the soft Q-function for the ego task and let Q_i be impact-policy Q-functions for the effect on other agents. AAIL constructs a compound Q-function

$$Q_C=w_t{Q}_t+(1-w_t)\sum _i\frac{w_i}{{\sum }_j{w}_j}{Q}_i.$$

The impact reward for agent i acting on agent j is the change in the estimated value of j’s task:

$$I_i^j(s_t,a_t^i)=V^j(s_{t+1})-V^j(s_t).$$

The task-policy weight w_t increases when the task policy has low entropy, while the impact weights w_i use the Jensen-Shannon distance between the task action distribution and each impact-policy distribution.

Derivation / Construction

• Train task policies with maximum-entropy RL so each task has a soft Q-function and an analytically available action distribution.
• For each other-agent task, train an impact policy using impact I_i^j as the reward signal.
• At deployment, select the ego task policy and the relevant impact policies for nearby agents’ goals.
• Normalize Q-value distributions, compute the task entropy, compute pairwise Jensen-Shannon distances to the impact policies, and form the compound Q-function.
• Sample or choose actions from the compound policy induced by Q_C.
• The entropy/JSD rule is intended to avoid giving high influence to impact policies when the ego task policy is decisive or when the impact policy is poorly aligned with task actions.

Implications

AAIL is a useful middle point between fully cooperative MARL and purely egoistic single-agent RL. It does not assume a shared team reward, but it does assume that another agent’s task value can be estimated well enough to learn positive-impact behavior. In safety terms, this makes the quality of the other-agent value model important: optimizing “impact” against the wrong model can become interference or manipulation.

Links

• Rother2023 - Disentangling Interaction using Maximum Entropy Reinforcement Learning in Multi-Agent Systems
• Multi-Agent Coordination
• Ad Hoc Teamwork
• Cooperative Multi-Agent Reinforcement Learning
• Multi-Agent Non-Stationarity