Pre-training is the long, undirected first act — the P in GPT. A model is shown an enormous quantity of text: a meaningful fraction of the written internet, plus books, code, and transcripts. It is set a single, almost absurdly simple task. Given everything so far, predict what comes next. No labels, no lessons, nobody marking the work. It is closer to an apprenticeship by immersion than to a course: nobody explains the trade, you simply spend years in the workshop until the patterns are in your hands. The 'pre' means before anything specific — before the model has been aimed at a use, a task, or an audience.
Because nobody curates the immersion, the model acquires whatever the corpus contains: the careful reasoning and the sloppy reasoning, the consensus and the prejudice, the well-sourced claim and the confident nonsense sitting on the page beside it. It also acquires no sense of which is which. A model straight off a pre-training run is not a helpful assistant — it is something stranger and considerably less useful, a machine that will continue a sentence in whatever register it was started in, including registers nobody wanted. Everything that makes a model feel patient, or reluctant, or safe was added afterwards, in far cheaper stages.
This is where nearly all of the money goes, and it is the stage that cannot be repeated casually. It runs for months across tens of thousands of chips and consumes very nearly all the text anyone has managed to gather — which is itself becoming the constraint, since the supply of high-quality human writing is finite and largely already spent. The decisions that matter are all made before it begins: how much data, how large a model, how to divide a fixed compute budget between the two. Getting that division right is not guesswork. It is what scaling laws are for.
The human translation is the part of your education nobody designed. Not the syllabus — the reading you did anyway, the arguments you overheard, the years spent absorbing how people in your field talk before you had anything of your own to say. It is unglamorous, largely invisible, and impossible to reconstruct afterwards. It is also, for a model as for a person, where almost all of it came from.
