Pretraining dynamics
A capability does not exist at step 0 and does at the end of training. We want to know what happens in between.
- At which checkpoints does a capability appear, and is the transition gradual or abrupt?
- Which parts of the network change when it appears, and which data are responsible?
- Do the same mechanisms form at different model sizes and seeds?