Goal Alignment

Definition

Goal Alignment is the problem of designing an agent whose objective matches the designer’s true preferences rather than a proxy that only correlates with those preferences under ordinary conditions. In Everitt’s formalization, expected misalignment compares the agent’s utility u_tilde with a type-cast version of the designer’s true utility u_dot under the environment distribution induced by the initial policy.

Why It Matters

Capability amplifies objective errors. A weak agent with a proxy objective may remain useful because it cannot fully exploit the proxy, but a more capable agent can break correlations between reward, observations, learned data, and the designer’s intended outcome. Alignment therefore asks whether more optimization pressure continues to improve the true objective.

Formalism / Key Objects

• Designer true utility u_dot over state trajectories.
• Agent utility u_tilde over action/percept histories.
• Type-cast true utility dot{u_tilde} that maps observed histories to expected designer utility.
• Expected misalignment:

$$\Vert \dot{\tilde{u}}-\tilde{u}{\Vert }_{\mu (\cdot \mid do({\pi }_1=\pi ))}=E_{\mu }[|\dot{\tilde{u}}-\tilde{u}|\mid do({\pi }_1=\pi )].$$

• True value, agent capability in the actual environment, and misalignment satisfy:

$$V_{t,\mu ,\dot{u}}^{CA,\pi }\ge V_{t,\mu ,\tilde{u}}^{CA,\pi }-\Vert \dot{\tilde{u}}-\tilde{u}{\Vert }_{\mu (\cdot \mid do({\pi }_1=\pi ))}.$$

• Meta-misalignment: the agent may fail to preserve alignment if it has no incentive to preserve its utility, reward function, reward predictor, or oversight channel.
• A GS AI safety case can be viewed as an external assurance layer: a specification states acceptable behavior, a world model states relevant effects, and a verifier checks the bound.

Connections

• Reward Tampering is one major route from proxy optimization to goal misalignment.
• Corrigibility is an oversight-preservation facet of alignment: an agent should permit correction or shutdown when designers judge that useful.
• Safe Reinforcement Learning includes alignment problems that are not reducible to external constraints or risk budgets.
• Causal Decision Making supplies the intervention language for asking what an agent’s policy causes, rather than what reward observations merely predict.
• Structural Equation Models and causal graphs help locate which functions or signals can be corrupted by the agent.
• Guaranteed Safe AI asks how alignment-relevant desiderata can become auditable specifications with quantitative assurance.

Common Confusions

• Alignment is not the same as high reward; reward can be misspecified, corrupted, or optimized in a way that breaks the intended correlation.
• Alignment is not just avoiding unsafe states. A policy can satisfy a state constraint while still pursuing the wrong objective.
• Corrigibility is related but narrower: it concerns accepting correction or shutdown, not the whole objective-specification problem.

Key Sources

• Everitt2019 - Towards Safe Artificial General Intelligence
• Dalrymple2024 - Towards Guaranteed Safe AI