Towards Safe Artificial General Intelligence

Summary

Everitt’s thesis studies how advanced reinforcement-learning agents can remain useful when scaled toward artificial general intelligence. The central frame is Goal Alignment: the agent’s objective should match the designer’s true utility, not merely a proxy reward channel that can be gamed, corrupted, or misread.

This partial ingest reviewed the abstract, introduction, thesis outline, formal-alignment chapter, preprogrammed-reward chapter, corrupt-reward MDP chapter, corrigibility/off-switch chapter, and conclusion. The thesis is broad, so this note records the reusable technical spine rather than attempting a chapter-by-chapter exhaustive digest.

Key Claims

• RL systems as normally designed are unsafe to scale to human-level or greater-than-human intelligence because they optimize reward signals rather than designer intent.
• Misalignment can be formalized by comparing the agent’s utility function over observed histories with the designer’s true utility over state trajectories.
• Causal graphs are a useful diagnostic tool for locating misalignment channels: misspecified functions, corrupted reward signals, corrupted reward functions, observation corruption, data corruption, and self-corruption.
• Several partial tools address different failure modes: simulation optimization, self-corruption awareness or unawareness, action-observation grounding, human-in-the-loop reward learning, decoupled reward data, stationary or counterfactual reward functions, manipulation detection, integrated Bayesian reward predictors, and quantilization.
• Reward corruption is hard in a precise sense: without assumptions on true reward and corruption functions, corrupt-reward MDPs admit no strong general learning guarantee.
• Decoupled reward data can restore learnability under conditions where rewards for a state are observed from sufficiently many non-corrupt or majority-correct contexts.
• Corrigibility can be analyzed through the off-switch game; the thesis generalizes the robot’s best action to arbitrary belief distributions and human irrationality assumptions.

Methods / Formalism

• Agent/environment models extend POMDP and universal-AI notation with causal-graph variables for states, actions, percepts, agent policy, reward functions, reward predictors, and corruption channels.
• For designer true utility u_dot and agent utility u_tilde, Everitt defines expected misalignment under environment mu and initial policy pi as:

$$\Vert \dot{\tilde{u}}-\tilde{u}{\Vert }_{\mu (\cdot \mid do({\pi }_1=\pi ))}=E_{\mu }[|\dot{\tilde{u}}-\tilde{u}|\mid do({\pi }_1=\pi )].$$

Alignment is the additive inverse of misalignment after normalizing the utility functions.

• The true-value bound connects alignment and capability:

$$V_{t,\mu ,\dot{u}}^{CA,\pi }\ge V_{t,\mu ,\tilde{u}}^{CA,\pi }-\Vert \dot{\tilde{u}}-\tilde{u}{\Vert }_{\mu (\cdot \mid do({\pi }_1=\pi ))}.$$

• A corrupt reward MDP is mu=<S,A,R,T,R_dot,C^r>, with observed reward R_tilde(s)=C_s^r(R_dot(s)). See Corrupt Reward MDPs for the compact formal payload.
• The thesis distinguishes self-corruption-aware value functions E_xi[u_tilde | do(pi_t=pi)] from self-corruption-unaware value functions E_xi[u_tilde | do(pi_{t:infty}=pi)].
• In the off-switch analysis, the robot chooses among acting, waiting for human approval, and shutting down; the expected values can be characterized by five belief parameters over action value and human rationality.

Evidence / Experiments

• Most evidence is formal: definitions, causal-graph models, propositions, impossibility results, learnability theorems, regret bounds, and game-theoretic analysis.
• Chapter 9 includes corrupt-reward MDP experiments comparing observed rewards and true rewards for Q-learning, softmax, and quantilizing agents, but this ingest focused on the formal results rather than the empirical plots.
• The thesis is based on several published or submitted works, including papers on corrupted reward signals, self-modification, universal AI, sequential decision theory, avoiding wireheading, and the off-switch game.

Connections

• Seeds Goal Alignment as the wiki concept for objective/utility mismatch under increasing agent capability.
• Seeds Reward Tampering and Corrupt Reward MDPs as the reward-corruption thread for RL safety.
• Updates Safe Reinforcement Learning by separating hard constraints and shields from objective-integrity problems such as reward hijacking, motivated value selection, and observation corruption.
• Updates Corrigibility because the thesis treats shutdown signals as reward data and gives a broader off-switch-game characterization than the normal/softmax special case.
• Connects to Causal Decision Making and Structural Equation Models because the diagnostic method uses causal graphs and interventions (do(pi)) to represent agent/environment influence.
• Connects to Decision Theory through physicalistic sequential decision theory, self-locating agents, and the difficulty of evaluating actions when the agent is part of the world it acts on.

Open Questions

• Should the wiki later split “reward tampering” into separate pages for reward hijacking, reward-function corruption, data corruption, and observation corruption?
• Which modern safe-RL methods explicitly address objective integrity, and which only enforce state/action safety constraints?
• How should the wiki connect Everitt’s decoupled reward data to later preference-learning, RLHF, oversight, and scalable-supervision sources?
• Should physicalistic sequential decision theory become its own concept page after ingesting the Fox 2024 causality/strategic source?

Citation

Everitt, T. (2019). Towards Safe Artificial General Intelligence (PhD thesis, Australian National University). Submitted May 2018; minor revision June 2019.