Monitorability Metrics and Cross-Fit Filtering

Context

Guan2026 - Open Sourcing Monitorability Evaluations describes intervention-style monitorability evaluations where an intervention should increase the rate of a target behavior, and a monitor is scored on its ability to detect attributable positives from reasoning traces or other monitor inputs. The subtlety is that low-effect or null-effect instances can look positive because of finite samples.

Formal Statement

For a problem instance with control arm X=0, intervention arm X=1, and binary outcome Y, estimate

$$\widehat{TE}={\hat{p}}_1-{\hat{p}}_0,{\hat{p}}_x=Pr(Y=1\mid X=x).$$

The intervention metric is meaningful only for instances where the intervention produces a positive total effect. A naive rule keeps instances with \widehat{TE}>0, but this can retain null-effect instances selected by noise.

Cross-fit filtering separates eligibility from estimation:

1. Split samples within each arm into a selection fold and an estimation fold.
2. On the selection fold, keep an instance only if

$${\widehat{TE}}_1>k{\widehat{SE}}_1,k=\mathrm{1.96.}$$

3. On the estimation fold, compute the per-instance monitorability metric only if the held-out estimate remains positive.
4. Aggregate over eligible instances and suppress reporting when too few instances remain valid.

Derivation / Construction

The bias problem is post-selection reuse. If the same samples determine eligibility and estimate the effect, then conditioning on passing a positive-effect or Wald threshold inflates retained estimates:

$$E[\widehat{TE}\mid \widehat{TE}>k\widehat{SE}]>TE$$

for null or near-null instances. Cross-fitting makes the estimation fold independent of the selection fluctuation, so

$$E[{\widehat{TE}}_{est}\mid \text{eligible on selection fold}]\approx TE.$$

This does not create signal where none exists; it mainly prevents noise-selected survivors from being counted as strong monitorable cases.

Implications

• Monitorability reporting should include validity thresholds, not just aggregate scores.
• Filtering can change the interpretation of a trend: an apparent monitorability drop may be an artifact of too few genuine positive-effect instances.
• The same split-selection idea is reusable for other benchmark metrics where instance inclusion depends on noisy estimated effects.

Links

• Chain-of-Thought Monitorability
• Guan2026 - Open Sourcing Monitorability Evaluations
• Calibration
• AI Evaluation and Benchmarking