Explainable Shielding
Definition
Explainable shielding makes the decisions of a runtime safety shield legible, for example by explaining why a state is safety-critical, why an action is blocked, or which failure traces justify a risk estimate.
Why It Matters
Shielding can make learning systems safer while leaving humans uncertain about why actions are forbidden. Explaining the shield targets a smaller and more formal object than the whole learned policy, so it can support debugging and trust in the safety mechanism itself.
Formalism / Key Objects
- Given an MDP
M, safety propertyphi, horizonh, and risk thresholdepsilon, compute state and state-action risk using probabilistic model checking. - An action is safe in state
sif:
- States can be partitioned into: safe states where all actions are safe; critical states where some actions are safe; dangerous states where no action is below the threshold but no violation has occurred; unsafe states where the property is already violated.
- A hierarchy of explanation objects can be used: a state-category tree, action-availability trees, and execution trees for unsafe-action evidence.
- Shield Risk Decision Trees stores the formal version of this hierarchy.
Connections
- Rieder2025 - Explainably Safe Reinforcement Learning introduces the decision-tree explanation framework.
- Shielding supplies the runtime intervention layer being explained.
- Probabilistic Model Checking computes the risk quantities that drive both the shield and the explanations.
- Explainable AI is the broader area, but this concept is specifically about explaining formal safety decisions rather than general neural-policy behavior.
Common Confusions
- Explaining a shield is not the same as explaining the learned policy.
- A compact tree can be faithful to the shield’s abstraction while still omitting details of the real environment.
- Dangerous does not mean violated; it means every available action exceeds the risk threshold.