Hamilton-Jacobi Reachability

Definition

Hamilton-Jacobi reachability is a control-theoretic method for computing the states from which a dynamical system can satisfy or avoid a target constraint under optimal control, disturbance, or game assumptions. In safety analysis, it often computes a value function whose sign separates states that can remain safe from states that inevitably reach failure.

Why It Matters

HJ reachability gives a mathematically sharp notion of safe set and safety-preserving feedback. It is one of the main control-theoretic roots of runtime shields for robotics and cyber-physical systems, but classical grid-based solvers scale poorly with state dimension. This makes approximate, learned, decomposed, or conservative variants important for modern RL systems.

Formalism / Key Objects

• A dynamical system is written:

$$\dot{x}=f(x,u),$$

with state x, control u, and admissible control set U.

• A safe constraint set K can be represented by a signed payoff l(x) such that:

$$l(x)\ge 0⟺x\in K.$$

• A common infinite-horizon safety value is:

$$V(x)=\underset{u(\cdot )}{\sup }\underset{t\ge 0}{\inf }l({\xi }_x^u(t)).$$

The controller maximizes the worst safety margin ever encountered by the trajectory.

• The safe set is the nonnegative superlevel set of the value function, often \{x:V(x)\ge 0\}.
• Discounted Safety Bellman Equation records Fisac et al.’s RL-compatible discounted backup for approximating this value function.

Connections

• Fisac2019 - Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning introduces a discounted safety backup so RL algorithms can approximate HJ safety values.
• Shielding can use HJ-style safety controllers as supervisory overrides near the safety boundary.
• Control Theory supplies the dynamical-systems and feedback-control framing.
• Safe Reinforcement Learning uses HJ reachability when safety is represented as guaranteed constraint satisfaction rather than expected-cost minimization.
• Probabilistic Controlled Invariant Sets are adjacent safe-set objects for stochastic MDP settings; HJ reachability is more classical for continuous dynamics and robust optimal control.
• Sound Value Iteration is conceptually related because both distinguish safety-certifying value computations from ordinary reward estimates.

Common Confusions

• HJ safety value is not an additive reward value. It is based on a minimum or worst safety margin over time.
• A learned approximation to an HJ value function is not automatically a formal safety certificate.
• A safe set can be useful even when the optimal task policy is learned separately; the safety controller can act as an override or fallback.

Key Sources

• Fisac2019 - Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning
• Hashimoto2026 - Data-Driven Synthesis of Probabilistic Controlled Invariant Sets for Linear MDPs
• Alshiekh2018 - Safe Reinforcement Learning via Shielding