1 Introduction Over the past decade, natural language processing has evolved from statistical prediction to deep learning systems that emulate complex, human-like behaviors. The invention of the Transformer architecture (Vaswani et al., 2017) has allowed the scale of training necessary for large language models, and new research on the technology has developed models that can reason, possess world knowledge, and be helpful chatbots. While architecture and scale have been major drivers of progress, many of the most interesting behavioral breakthroughs have occurred in post-training: the stage after self-supervised pre-training where models are refined through supervised fine-tuning, reinforcement learning from human feedback (RLHF), and other alignment procedures. The base objective of next-token prediction is a proxy for language understanding and world knowledge, but it is agnostic to conceptions of “truth”1 or ethical norms. Post-training injects human preference data into these systems, teaching them to not merely predict text, but to perform appropriately within human social and moral contexts. The post-training process of an LLM resembles socialization in human development: the transformation of raw cognitive potential into an assistant capable of following norms, persuading others, and explaining its own reasoning. Yet this human-like training also introduces epistemological hazards. Models are optimized for approval, not “truth”—a tension that mirrors classic psychological findings about human introspection and reasoning.
...