Probabilistic Shielding for Safe Reinforcement Learning

Summary

This partial ingest is based on the extracted full PDF text. Hamel-De le Court, Belardinelli, and Goodall give a Probabilistic Shielding construction for finite MDPs with known safety dynamics and an undiscounted probabilistic avoidance constraint. The method computes an inductive upper bound on the minimal probability of ever reaching an unsafe state, augments the MDP state with a remaining probability budget, and trains an ordinary RL learner inside the resulting shielded MDP. The resulting projected policy is safe during both training and testing, and the shield is optimality-preserving as the upper bound converges to the true risk function.

Key Claims

• Strict safety during exploration can be obtained without solving the full constrained reward problem by linear programming, provided the safety-relevant dynamics are known.
• A shield over an augmented state (s, q) can represent probabilistic safety budgets more expressively than a hard action mask over the base state s.
• If beta is an inductive upper bound on the minimal unsafe-reachability probability beta_M, every memoryless policy in the shielded MDP satisfies the probability threshold.
• Learning in the shielded MDP preserves reward optimality in the limit where beta -> beta_M, under a Slater-style strict-feasibility assumption.
• The approach separates safety dynamics from reward dynamics: the shield only needs the safety-relevant quotient of the environment, while any RL algorithm can optimize reward inside the shield.

Methods / Formalism

• The reachability-constrained optimization problem asks for a policy maximizing discounted return among policies satisfying M_pi |= P_{\le p}(Reach(u)), where u labels unsafe states.
• The key risk function is:

$${\beta }_M(s)=\underset{\pi }{\min }P{r}_{M,\pi ,s}(Reach(u)).$$

• For safe states, beta_M is the least fixed point of the Bellman-style safety operator B_M(beta)(s)=min_a sum_{s'} P(s,a,s') beta(s'); unsafe states have value 1.
• The shield requires an inductive epsilon-upper bound beta with 0 <= beta(s)-beta_M(s) <= epsilon and B_M(beta)(s) <= beta(s). The paper points to interval iteration, Sound Value Iteration, and optimistic value iteration as suitable ways to obtain such bounds.
• The shield state is (s, q), where q in [beta(s), 1] is the current safety level or allowed unsafe-reachability budget. A shield action combines a base action distribution with a prediction alpha:S -> [0,1] for next-state budgets.
• The action constraints enforce:
  • predicted next budgets stay at least as large as beta(s');
  • the transition-weighted predicted budget is no larger than the current budget.
• Theorem 1 proves that any memoryless policy in the shielded MDP satisfies P_{\le p}(Reach(u)); the projection back to the original MDP is generally memoryful because it tracks the current risk budget.
• Probabilistic Risk-Budget Shields records the reusable construction.

Evidence / Experiments

• The implementation uses interval iteration to compute the risk upper bound and PPO to optimize reward in the shielded MDP.
• Experiments cover media streaming, two color-bomb gridworlds, two bridge-crossing gridworlds, and a Pacman-like environment with a safety abstraction.
• PPO-shield preserves safety throughout training and test time in all reported environments.
• In the gridworld and bridge tasks, PPO-shield reaches near-optimal reward while respecting the probabilistic safety bound; in the bridge tasks it converges more slowly because the unsafe but reward-short path must be avoided.
• PPO-shield substantially outperforms PPO-Lagrangian and CPO in reward on these case studies, while the authors note the known-dynamics assumption as the main practical limitation.

Connections

• Extends Alshiekh2018 - Safe Reinforcement Learning via Shielding from almost-sure temporal-logic safety toward probabilistic unsafe-reachability constraints.
• Uses Probabilistic Model Checking machinery because the shield depends on sound upper bounds for MDP reachability probabilities.
• Sound Value Iteration is directly relevant: the shield needs upper, not merely lower, risk estimates to claim safety.
• Complements Hashimoto2026 - Data-Driven Synthesis of Probabilistic Controlled Invariant Sets for Linear MDPs: both produce runtime safety filters, but this paper assumes known safety dynamics and reasons over infinite-horizon unsafe reachability rather than data-certified finite-horizon invariant sets.
• Precedes HamelDeLeCourt2025 - ProSh Probabilistic Shielding for Model-free Reinforcement Learning, which replaces the known safety model with a learned backup cost critic for CMDP-style expected cost constraints.

Open Questions

• How small must the safety abstraction be for the augmented shield to remain practical beyond gridworld-like or factorable safety dynamics?
• How should uncertainty in the safety model be represented when the safety quotient is approximate rather than known?
• Can the optimality-preserving argument be extended cleanly to continuous spaces, function approximation, or partially observed environments?
• When does the risk-budget augmentation help exploration, and when does it make the learner’s state/action space too large?

Citation

Hamel-De le Court, Edwin, Francesco Belardinelli, and Alexander W. Goodall. 2025. “Probabilistic Shielding for Safe Reinforcement Learning.” arXiv:2503.07671.