Safe Mean-Field UCRL

Context

Jusup2024 - Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning proposes Safe-M3-UCRL for large homogeneous multi-agent populations. The formal payload is the conservative mean-field safety constraint under unknown transitions and the log-barrier optimistic update.

Formal Statement

Let mu_t in P(S) be the mean-field distribution and let h_C(mu)=h(mu)-C. The safe mean-field control problem requires:

$$h_C({\mu }_{t+1})\ge 0$$

for the distribution induced by the policy profile and transition map.

When the transition model is estimated, the planned distribution \tilde\mu_{n,t} may differ from the true distribution \mu_{n,t}. If h is Lipschitz in Wasserstein distance and C_{n,t} upper-bounds the planned/true distribution mismatch, then:

$$|h_C({\tilde{\mu }}_{n,t})-h_C({\mu }_{n,t})|\le L_h{C}_{n,t}.$$

Thus the tightened planned constraint:

$$h_C({\tilde{\mu }}_{n,t})\ge L_h{C}_{n,t}$$

implies true safety h_C(mu_{n,t}) >= 0 on the confidence event.

Safe-M3-UCRL optimizes over policy profile pi_n and hallucinated transition-control eta:

$$\underset{{\pi }_n,\eta }{\max }E\sum _{t=0}^{T-1}\left[r({\tilde{z}}_{n,t})+\lambda \log \left(h_C({\tilde{\mu }}_{n,t+1})-L_h{C}_{n,t+1}\right)\right].$$

Derivation / Construction

• Build a calibrated statistical model of transitions with mean m_{n-1} and uncertainty Sigma_{n-1}.
• Use hallucinated UCRL:

$${\tilde{f}}_{n-1}(z)=m_{n-1}(z)+{\beta }_{n-1}{\Sigma }_{n-1}(z)\eta (z),$$

where eta searches optimistically within the confidence set.

• Propagate the planned mean-field distribution under the candidate policy profile and hallucinated transition model.
• Compute conservative slack L_h C_{n,t} from uncertainty and Lipschitz/Wasserstein bounds.
• Add a log barrier so optimization remains inside the tightened safe domain.
• Update the transition model after each episode from collected trajectories.

Implications

• Safety is global and distributional: the constraint can encode coverage, dispersion, capacity, or fairness over the population.
• The algorithm is optimistic about reward but pessimistic about constraints.
• Guarantees depend on model calibration, Lipschitz assumptions, and a safe initial mean-field distribution.
• In finite-agent deployment, the mean-field policy still needs a concentration or robustness argument to connect the population limit to sampled agents.

Links

• Source: Jusup2024 - Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning
• Concept: Mean-Field Reinforcement Learning