“Why study alignment interventions on pre-RL checkpoints?” by Edward James Young, Puria, Cam

This is a dual post that lays out our current research project where we compare pre-RL-training methods on their ability to prevent models from ‘proto-training gaming,’ which we predict is selected for over the course of production RL post-training. In this post, we outline what we mean by pre-RL ‘alignment checkpoints’, give our reasons for focussing on these stages of training, and suggest ways that our current viewpoint might be wrong.

In the next post, we define proto-training gaming and argue that i) it is a necessary precursor to adversarial misalignment, ii) it is ecologically selected for in, and competently performed by, current models, iii) the pre-RL alignment checkpoint can have an outsized mitigation of this selection.

What do we mean by the pre-RL alignment checkpoint?

By the pre-RL alignment checkpoint of a model, we mean the alignment-relevant properties of a model as conferred by all the training that happens prior to on-policy RL[1]. This encompasses the following stages, with corresponding levers that safety teams can intervene on:

• Pretraining – training a model from scratch on a general text corpus. Our team has previously studied data-filtering to remove dangerous capabilities, and more recently adding positive synthetic documents to [...]

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
July 8th, 2026

Source:
https://www.lesswrong.com/posts/nhjkHsppEk98xxmPe/why-study-alignment-interventions-on-pre-rl-checkpoints

---

Narrated by TYPE III AUDIO.