Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective

Summary

Chelarescu reviews deception risks in MARL social learning, where agents can directly reshape other agents’ reward functions through incentives, causal-influence rewards, or reward-giving mechanisms. The review emphasizes the safety tension: social incentives can speed cooperation in social dilemmas, but they also give agents access to each other’s learning process and may enable manipulation when goals are heterogeneous or misaligned.

This is a partial ingest based on the full extracted text, with emphasis on definitions, risk framing, and open problems.

Key Claims

• Classical MARL lets agents influence each other indirectly through environment actions; social-learning mechanisms add direct influence over another agent’s reward signal or learning process.
• Reward-giving and incentive-learning methods can improve cooperation in mixed-motive games such as sequential social dilemmas.
• Most evidence assumes homogeneous or similarly aligned agent populations; heterogeneous agents with different goals raise a larger deception risk.
• Trust becomes a technical safety issue once an agent can accept incentives from another agent that may benefit from reshaping its policy.
• The literature has few guarantees against deception, so social learning should be evaluated not only by collective return but also by manipulation resistance, incentive provenance, and the ability to reject bad incentives.

Methods / Formalism

• The review defines social learning broadly as MARL algorithms that allow agents to reshape reward functions assigned to other agents.
• A representative causal-influence formulation modifies agent k’s immediate reward as

$$r_t^k=\alpha e_t^k+\beta c_t^k,$$

where e_t^k is extrinsic environment reward and c_t^k is a causal-influence reward.

• A reward-giving formulation modifies agent j’s reward by adding incentives supplied by other agents:

$$r_j(s_t,a_t,{\eta }^{-j})=r_{j,env}(s_t,a_t)+\sum _{i\ne j}{r}_{{\eta }_i}(o_t^i,a_t^{-i}).$$

• The source of incentive rewards matters: incentives may be paid from the giver’s budget, separated from the giver’s reward-maximization policy, or mediated by a planner.
• The review distinguishes cooperative benefits from deception risk by asking whether the recipient can detect, reject, or discount incentives that would move it toward policies against its own interests.

Evidence / Experiments

• This is a review, not a new empirical benchmark.
• It synthesizes work on social influence, reward-giving, inequity aversion, opponent modelling, reward tampering, and deception in multi-agent settings.
• The review highlights several open-problem scenarios, including Harvest-style sequential social dilemmas where a reward-giving agent could induce others to under-explore while keeping common resources for itself.
• It argues that probing agents for deception, classifying trustworthiness, and using inequity aversion or reputation mechanisms are possible but underdeveloped responses.

Connections

• Seeds Social Learning in MARL as the wiki concept for direct incentive and reward-reshaping channels between agents.
• Connects to Opponent Shaping: both study agents that influence other learners, but social learning often uses explicit reward-channel manipulation rather than differentiating through another learner’s update.
• Extends Multi-Agent Non-Stationarity from “teammates are learning” to “teammates may alter the learning signal itself.”
• Belongs in Safe Multi-Agent Reinforcement Learning because safety claims about cooperation can fail when incentive channels are exploitable.
• Supports Strategic Reasoning by making trust, reputation, deception, and incentive provenance part of the formal multi-agent problem.

Open Questions

• What concrete mechanisms let an agent detect deception without unethical or unrealistic probing?
• Can agents safely reject or discount incentives based on giver reputation, incentive source, or downstream policy impact?
• How should social-learning systems be benchmarked under heterogeneous goals rather than homogeneous self-play?
• When does incentive shaping become reward tampering from the recipient’s perspective?
• Can deception-resistance guarantees be stated for MARL social-learning algorithms?

Citation

Chelarescu, Paul C. 2021. “Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective.” arXiv:2106.05402.