Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning

Summary

Fisac, Lugovoy, Rubies-Royo, Ghosh, and Tomlin connect Hamilton-Jacobi Reachability safety analysis with reinforcement learning by introducing a discounted Safety Bellman Equation. Classical HJ safety asks for a controller that maximizes the minimum signed distance to failure over time, but the undiscounted dynamic-programming backup is not a contraction. The paper modifies the backup so temporal-difference methods can learn approximate safety value functions and safety policies.

The paper is best read as a computational bridge rather than a complete safe-learning framework. It enables RL methods to approximate safety analysis in simulation, including higher-dimensional examples where grid-based HJ reachability is intractable, but the authors emphasize that model-free learning may need to experience failures and should not be treated as safe online training on a real system.

Key Claims

• Safety is a minimum-over-time constraint-satisfaction problem, not an additive reward-maximization problem.
• The standard infinite-horizon HJ safety backup does not induce a contraction, which blocks direct use of ordinary value-iteration and temporal-difference convergence arguments.
• A locally discounted Safety Bellman Equation induces a contraction under the supremum norm while converging to the undiscounted safety value as gamma -> 1.
• The resulting Safety Q-learning update converges almost surely to the optimal state-action safety value function in finite discretized MDPs under standard exploration and learning-rate assumptions.
• Deep Q-learning and policy-optimization variants can use the discounted safety objective to approximate safe sets and safe policies in examples beyond the dimensional reach of traditional grid-based HJ solvers.

Methods / Formalism

• The continuous-time system has state x in X, control u in U, and dynamics:

$$\dot{x}=f(x,u).$$

• A constraint set K is represented by a payoff function l:X -> R such that l(x) >= 0 iff x in K.
• The undiscounted infinite-horizon safety value is:

$$V(x)=\underset{u(\cdot )}{\sup }\underset{t\ge 0}{\inf }l({\xi }_x^u(t)).$$

• A positive value means the system can remain inside the constraint set forever; a negative value means failure is unavoidable under the selected assumptions.
• The discrete undiscounted safety backup is:

$$V(x)=\min \{l(x),\underset{u\in U}{\max }V(x+f(x,u)\Delta t)\}.$$

• The paper’s discounted Safety Bellman Equation is:

$$V(x)=(1-\gamma )l(x)+\gamma \min \{l(x),\underset{u\in U}{\max }V(x+f(x,u)\Delta t)\}.$$

• Discounted Safety Bellman Equation records the contraction result, the gamma -> 1 approximation claim, and the Safety Q-learning update.

Evidence / Experiments

• Double integrator: tabular and deep Safety Q-learning recover the analytic safe set as the discount factor is annealed toward 1.
• Cart-pole: learned deep safety values closely match a numerical dynamic-programming safe-set approximation.
• Lunar lander: a six-dimensional safety value function learned by Safety DQN gives safety predictions that match rollout behavior when ground truth is unavailable.
• Half-cheetah: policy optimization with discounted safety yields policies that try to stand or sit safely in an 18-dimensional MuJoCo task, while additive reward formulations often produce unsafe jumping or falling.
• The experiments support scalability and alignment with known safe sets, but they do not constitute formal guarantees for neural function approximators.

Connections

• Seeds Hamilton-Jacobi Reachability as the control-theoretic safe-set analysis concept behind several safe-RL and shielding papers.
• Discounted Safety Bellman Equation captures the paper’s reusable formal payload: turning minimum-over-time safety into a contraction backup.
• Connects to Safe Reinforcement Learning by clarifying a different role for RL: approximating a safety value function, not merely optimizing reward under constraints.
• Connects to Shielding because learned HJ-style safety policies can act as supervisory controllers that override performance policies near the boundary of the safe set.
• Complements Probabilistic Controlled Invariant Sets: both bridge control-theoretic invariance/safe sets and RL, but this paper focuses on deterministic HJ reachability and discounted safety learning rather than data-certified probabilistic invariance.

Open Questions

• How should learned safety value functions be verified or conservatively bounded before being used as runtime safety filters?
• Can robust or stochastic versions of discounted safety preserve the contraction benefits while matching physical uncertainty?
• How should a performance RL policy be combined with a learned HJ safety controller without corrupting exploration or overusing overrides?
• Which parts of the safety value function need accurate approximation for shielding, and which can tolerate conservative errors?

Citation

Fisac, J. F., Lugovoy, N. F., Rubies-Royo, V., Ghosh, S., & Tomlin, C. J. (2019). Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning. In IEEE International Conference on Robotics and Automation (ICRA).