Language models & chatbots
Pretraining is the first and biggest stage of building a language model, where it learns general language and knowledge by predicting text across huge datasets.
Before a model can be a chatbot, it has to learn what language is. In pretraining, it reads an enormous amount of text, trillions of tokens from websites, books, and code, and practices one task over and over: predict the next token. No human labels are needed, because the text itself supplies the answers.
This stage is where most of the cost and most of the knowledge come from. It can run for months on thousands of GPUs. The result, sometimes called a base or foundation model, is a powerful text continuer that is not yet polite, safe, or good at following instructions.
Later, shorter stages shape it: fine-tuning on example conversations and RLHF to make it helpful and harmless. The 'P' in GPT stands for pre-trained. The model's knowledge cutoff, the date after which it knows nothing, is set by when the pretraining data was collected.
Given the text 'The capital of France is,' a pretrained model has seen that pattern so often that it confidently continues with 'Paris.'