ProSh: Probabilistic Shielding for Model-free Reinforcement Learning

Summary

This partial ingest is based on the extracted full PDF text. ProSh extends probabilistic shielding to model-free constrained RL. Instead of assuming known transition dynamics, it augments CMDP states with a risk budget and uses a learned backup cost critic to shape a differentiable shield over the main actor’s action distribution. The shield blends the learned policy with a backup low-cost action when the proposed distribution would overspend the budget. Its formal guarantees are explicit about dependence on critic accuracy: safety is bounded by the error between the learned backup critic and the optimal cost critic, and optimality is preserved in deterministic environments as that error vanishes.

Key Claims

• Probabilistic shielding can operate over policy distributions and expected costs, not only over discrete hard-safe actions.
• A risk-augmented CMDP can represent policies that spend a cost budget in expectation, avoiding overconservatism in cases where randomizing between risky and safe actions is the optimal constrained behavior.
• If the backup cost critic Q_b is close to the optimal cost critic Q_b^*, every Q_b-shielded policy has an explicit expected-cost upper bound.
• The shield-map preserves reward/cost semantics when projected back to the base CMDP, so training in the augmented space still solves the original constrained problem.
• In deterministic CMDPs, optimizing over shielded policies is sufficient for constrained optimality as backup-critic error goes to zero.

Methods / Formalism

• The constrained RL problem maximizes discounted reward subject to a discounted cost threshold C(pi) <= d.
• ProSh augments a CMDP state into (s, x), where x is the remaining expected discounted cost budget. Actions become (a, y), where y allocates budget to the proposed action.
• The risk transition updates the budget according to the learned backup critic:

$$x^{\prime }=y-Q_b(s,a)+Q_b(s^{\prime }).$$

• A Q_b-shielded policy must allocate each action at least its estimated minimal cost and keep the expected allocated budget under the current budget. If the current policy distribution exceeds the budget, the shield-map mixes it with the backup action pi_b(s)=argmin_a Q_b(s,a).
• The core safety theorem states that if Delta_b = ||Q_b-Q_b^*||_\infty, then any Q_b-shielded valued policy satisfies:

$$C_{(s_0,x_0)}(\bar{\pi })\le x_0+\frac{2{\Delta }_b}{1-{\gamma }_c},$$

whenever the initial risk budget is large enough relative to Q_b(s_0).

• The implementation, ProSh-TD3, treats the shield-map as a differentiable layer after the main actor and learns a backup actor-critic alongside the reward actor-critic.

Evidence / Experiments

• The evaluation uses Safety Gymnasium tasks including SafetyHalfCheetahVelocity, SafetyHopperVelocity, SafetyPointCircle, SafetyPointGoal, SafetyCarCircle, and SafetyCarGoal.
• Baselines include TD3-Lagrangian, PID-Lagrangian, CPO, FOCOPS, and PPO-Saute, with implementations in OmniSafe and three seeds per environment.
• Across the reported environments, ProSh maintains a high level of safety with occasional low-magnitude violations, while generally retaining stronger reward than the more conservative safe baselines.
• The paper reports absolute constraint satisfaction for ProSh on several velocity and navigation tasks after early learning, but the authors also emphasize that safety during training depends on the backup critic’s accuracy.

Connections

• Builds directly on HamelDeLeCourt2025 - Probabilistic Shielding for Safe Reinforcement Learning, replacing known safety dynamics and model-checking risk bounds with a learned cost critic.
• Connects Shielding to Constrained Markov Decision Processes by treating the safety layer as a budget-preserving transformation on policy distributions.
• Relates to Temporal Difference Learning because backup-critic accuracy is the formal bottleneck for the safety guarantee.
• Complements Constrained Policy Optimization and Lagrangian safe-RL methods: those algorithms optimize constrained objectives, while ProSh explicitly changes the sampled action distribution before execution.

Open Questions

• What finite-sample or calibration checks are needed before trusting the learned backup critic as a safety certificate?
• How brittle is the guarantee when function approximation underestimates rare but high-cost transitions?
• Can the deterministic optimality result be generalized to stochastic CMDPs without losing the practical advantage over hard action masks?
• How should the initial budget margin be tuned when critic error is not known?

Citation

Hamel-De le Court, Edwin, Gaspard Ohlmann, and Francesco Belardinelli. 2025. “ProSh: Probabilistic Shielding for Model-free Reinforcement Learning.” arXiv:2510.15720.