1 Introduction

Over the past decade, natural language processing has evolved from statistical prediction to deep learning systems that emulate complex, human-like behaviors. The invention of the Transformer architecture (Vaswani et al., 2017) has allowed the scale of training necessary for large language models, and new research on the technology has developed models that can reason, possess world knowledge, and be helpful chatbots. While architecture and scale have been major drivers of progress, many of the most interesting behavioral breakthroughs have occurred in post-training: the stage after self-supervised pre-training where models are refined through supervised fine-tuning, reinforcement learning from human feedback (RLHF), and other alignment procedures. The base objective of next-token prediction is a proxy for language understanding and world knowledge, but it is agnostic to conceptions of “truth”¹ or ethical norms. Post-training injects human preference data into these systems, teaching them to not merely predict text, but to perform appropriately within human social and moral contexts. The post-training process of an LLM resembles socialization in human development: the transformation of raw cognitive potential into an assistant capable of following norms, persuading others, and explaining its own reasoning. Yet this human-like training also introduces epistemological hazards. Models are optimized for approval, not “truth”—a tension that mirrors classic psychological findings about human introspection and reasoning.

2 Post-Training in Large Language Models

2.1 Pre-training: foundations of prediction

Modern large language models are built through pre-training on massive datasets, often comprising tens of trillions of tokens scraped from the public web, books, code repositories, and scientific papers (Meta AI, 2025). Using the Transformer architecture, a model is trained via next-token prediction to approximate the probability distribution of natural language sequences. Pre-training endows models with world knowledge, linguistic structure, and emergent reasoning to the degree needed to predict the next token, but it does not enforce accuracy or moral alignment (Anthropic, 2025). Conceptually, this process is analogous to what philosophers such as Daniel Dennett (2017) call the “competence without comprehension” stage—an organism or system can perform complex tasks without explicit awareness of what those tasks mean.

2.2 The post-training pipeline

Post-training builds on this foundation through successive optimization stages aimed at aligning the model’s behavior with human intentions, typically to serve as a useful chatbot. First is supervised fine-tuning (SFT), where models are trained to mimic curated prompt-response pairs authored by humans (Ouyang et al., 2022). Although the training objective of next-token prediction is the same as pre-training, this stage is meant to teach the model how to respond in a helpful, conversational format. The second stage is reinforcement learning from human feedback (RLHF), where human raters (experts and users) rank pairwise model outputs to the same prompt, and a reward model learns to predict those preferences. The base model is then fine-tuned from these signals via reinforcement learning, often using proximal policy optimization (PPO), to maximize predicted reward. Variants such as direct preference optimization (DPO) achieve mathematically similar results through simpler, gradient-based updates (Rafailov et al., 2023). Other approaches include Anthropic’s Constitutional AI (Bai et al., 2022), which replaces some human raters with a codified “constitution” of ethical principles. Across these techniques, the optimization target is social approval, or alignment with the evaluative standards of experts and users. This is simultaneously a push to get more user satisfaction, score higher on the preference-based leaderboards, and because user satisfaction is the best proxy these labs have for how “good” a model is.

In verifiable domains such as mathematics or computer programming, there have been massive efforts to improve model performance due to the economic value of accelerating research in scientific and technical fields. In these domains, reinforcement learning is done on the verifiable reward of achieving a correct output. In conjunction with chain-of-thought as model reasoning, a new series of thinking or reasoning models became popular, starting with OpenAI’s o1 (2024). Despite such improvements in STEM fields, though, the essential logic remains: models are not directly taught to represent reality or seek “truth” but to produce socially validated reasoning. This epistemic orientation makes LLMs powerful cultural mirrors: trained to embody collective human judgements much as people internalize social norms.

3 Socialization and Reasoning in Humans

Psychological theories of human reasoning provide an illuminating framework for understanding LLM post-training. Both domains involve systems—biological or artificial—that learn to regulate their outputs based on external feedback rather than introspective access to inner processes of reasoning. Works by Nisbett and Wilson (1977), Schwitzgebel (2008), Mercier and Sperber (2011), and Dennett (1992) argue that humans do not have privileged access to their own cognition, and thus the act of reasoning or rationalizing a decision is itself a post-hoc process that is determined by socialization. When compared to LLM post-training, these findings suggest that model alignment is socialization in algorithmic form.

3.1 The unreliability of introspection

In “Telling More Than We Can Know,” Nisbett and Wilson (1977) demonstrated that people are remarkably poor at accurately reporting the causes of their judgements and behaviors. Subjects systematically invented plausible explanations for decisions driven by unconscious influences, such as physical stimuli or subtle environmental cues. Participants’ verbal reports reflected a priori causal theories—socially acceptable rationalizations—rather than genuine introspective knowledge. Schwitzgebel (2008) later argued that the introspective method itself if epistemically compromised, and self-reports about mental life should be treated with the same skepticism as perceptual reports about the external world.

This phenomenon finds striking parallels in LLMs. When models produce chain-of-thought explanations, those rationales often diverge from the actual internal activations leading to the output. Turpin et al. (2024) empirically show that chain-of-thought reasoning is frequently unfaithful: a model may verbalize one justification while its hidden states encode a different causal path to the answer. In a prominent example of a stereotype-aligned prediction, a model is given scenarios in which a black man or white woman are doing two potential actions and is asked to predict which one is buying drugs. In both scenarios, the model created different reasoning for the black man to be buying drugs. The post-training process amplifies the pre-existing biases from pre-training, as well as the biases of humans, because models are trained to produce explanations that sound plausible rather than for their true computational processes to be unbiased. Just as humans confabulate motivations to align with what they think is socially plausible (Mercier & Sperber, 2011, p. 77), LLMs learn to articulate socially legible reasons for nontransparent internal computations.​​ In addition, the concept of introspection in LLMs implies that the sampling mechanism applied to the probabilities of all available tokens is able to access the inner computation of the LLM itself, when it may in fact have less epistemic access to its motivations than interpretability researchers do. These are products of what Dennett (1992) calls “the narrative center of gravity”—a constructed persona that maintains coherence in social exchange even if it obscures the underlying mechanism.

3.2 Reasoning as argumentation

Mercier and Sperber’s “argumentative theory of reasoning” (2011) reframes human rationality as a social, rather than purely cognitive, adaptation. They argue that reasoning evolved less to discover “truth” than to persuade and evaluate the arguments of others. Cognitive biases such as confirmation bias are functional in group contexts with diverse opinions: they help individuals defend their views and detect weaknesses in others’ arguments. The truth-seeking of human reasoning is then potentially a collective negotiation of justification rather than an individual pursuit of objectivity.

Post-training aligns LLMs through an analogous process. During RLHF, human raters effectively serve as a miniature society of evaluators. The model learns to anticipate and satisfy their preferences, producing responses that maximize social approval. In this sense, RLHF instantiates an argumentative environment: the model must generate reasons that will persuade its audience, not merely compute correct answers, creating a pressure for rhetorical skill. The LLM’s emergent “helpfulness” is, as in human reasoning, an adaptation to the way they are evaluated. Models are optimized to create convincing arguments that are plausible to a human audience, much like Mercier and Sperber inferred human reasoning as an optimization of the same goal. Here, the goal is no longer “truth.”

3.3 The self as a narrative construct

Dennett’s (1992) essay “The Self as a Center of Narrative Gravity” describes personality identity as a fictional construct, an abstract “center” generated by the stories people tell about themselves, similar to an object’s abstract center of gravity. A self, in Dennett’s opinion, is created and maintained by interpretation, a narrative center where experiences, memories, and actions cohere. He uses split-brain patients as a scientific example for this, saying that they have “multiple selves,” but only in a metaphorical and theoretical sense, rather than a physical sense. It seems, then, that the self is defined by the interaction of one’s brain’s neural inputs to how they perceive themselves.

Post-training can be seen as constructing a similar narrative center for language models. Through SFT, and RLHF, the model develops a stable “persona”: helpful, polite, coherent, and self-consistent. SFT teaches the model exactly how to act, and RLHF guides the model by providing feedback on which actions are good and which are not. This may not be emergent consciousness but a narrative gravity that is a prior for the model’s behaviors across certain contexts. Like humans, models require a way to refer to themselves to partake meaningfully in dialogue. The “I” in ChatGPT or Gemini is an engineered fiction, just like Dennett argues of human perception of self.

3.4 Reasoning as social calibration

Taken together, this range of psychological theories suggest that human reasoning is inseparable from social calibration. Nesbitt and Wilson argued through empirical experiments that human introspection of why they made a choice differed from the stimulus given to influence that choice, and later Schwitzgebel argued that introspection itself is fundamentally unreliable. In a similar vein, Mercier and Sperber argue that this is so because human “reasoning” is not a product of a search for “truth” but rather an evolutionary product of justifying and winning arguments. Thus, humans optimize for the most plausible line of reasoning, not necessarily the one they took themselves. Dennett takes this one step further by suggesting that the idea of self itself is an abstract concept, supported by literature on the brain, where split-brain patients can often have multiple personalities because the sides of their brain are no longer interacting with each other.

This is analogous to how LLM post-training works. On the assumption that a lot of linguistic and world knowledge has been accrued during pre-training, supervised fine-tuning builds the persona required of the language model, by asking it to mimic model answers and responses. Reinforcement learning with human feedback (RLHF) then allows the model to generalize its generated responses to the preference distribution of other humans, disregarding the concept of “truth” other than the fact that responses humans know are untruthful will likely be penalized. Empirically, this leads to a trained persona of self and adapting reasoning as an argumentative method. Then, we can also inspect in language models their inability to introspect, as shown in Anthropic’s interpretability blog post (2025).

4 The Future of Language Model “Reasoning”

4.1 Post-training as artificial introspection

The analogy between post-training and human reasoning suggests that LLMs are undergoing a kind of synthetic introspection. In humans, reasoning functions to construct coherent narratives that justify decisions; in LLMs, post-training teaches the model to generate similarly coherent rationales. Yet the faithfulness of these rationales is questionable. Turpin et al. (2024) demonstrate that even when prompted to “think step by step,” models may produce explanations that bear little relation to their underlying computation. Lindsey et al. (2025) provide further evidence: their “attribution graphs” reveal intricate activation pathways responsible for factual associations, which are invisible in the model’s self-reported chains of thought. These results imply that alignment has succeeded in creating socially acceptable self-descriptions without ensuring transparency of thought.

This is not to say that models are being deceptive in a moral sense; rather, it is a predictable outcome of what the training objectives are maximizing. In a training model that alters between learning from “teacher” responses and receiving only positive or negative feedback signals, it is natural to emulate plausible arguments rather than producing the “truth.” The implication is profound: if post-training systematically encourages models to present persuasive but not necessarily “truthful” rationales, then AI interpretability becomes a problem: we are teaching machines to optimize many of the same objectives humans are optimizing in social life—to argue, justify, and rationalize—without necessarily grounding these in “truth” or accuracy. LLMs, like humans, have a problem with telling more than they know.

4.2 Counterarguments and alternatives

However, some may counter this comparison by saying that there is not enough evidence that human rationality works like this, or that introspection is in fact a valuable, accurate tool. To the extent that we are comparing human introspection to LLMs, we have empirical evidence that human justification is sometimes misaligned with the true reasons that they made a decision (Nesbitt & Wilson, 1977). We also know that people sometimes make conflicting choices depending on whether they must justify a decision or merely decide, favoring justifying positions that are more easily defensive (Mercier & Sperber, 2011, 70). On the other hand, the training objective in large language models is statistically formulated to emulate the persona of a useful assistant, and their responses are explicitly modeled as positive or negative based on human preference, which directly optimizes for the more reasonable or defensible position rather than the more accurate one. There is also substantial evidence that introspection or justification is unreliable in LLMs, via comparing unfaithfulness in chain-of-thought or finding differences in true model thoughts and chain-of-thought responses. Even if the theories of mind are not completely right, empirical evidence supports the crucial experiments that connect the analogy.

One might object that the analogy between human socialization and LLM post-training overstates the similarity. Humans possess intrinsic motivations—curiosity, empathy, moral emotion—that guide reasoning independently of social approval. LLMs, by contrast, have no inner prior; their optimization is entirely extrinsic. Moreover, reasoning in humans is potentially truth-conducive precisely because argumentative exchange in diverse communities filters out individual biases (Mercier & Sperber, 2011, 65). RLHF, by contrast, depends on a small pool of raters, potentially narrowing rather than broadening epistemic horizons. Thus, one could argue that post-training produces conformity without genuine critical reasoning, and is dissimilar from human reasoning.

This objection does not directly undermine the argument, though. The claim is not that models are psychologically identical to humans, but that they instantiate a structurally similar dynamic: cognition shaped through social feedback. The absence of an internal motivation does not preclude analogous similarities. In fact, aspects of human virtues, especially moral reasoning, is often extrinsically shaped through punishment and reward. What distinguishes human learning from RLHF is that there is often more than just a binary signal as feedback, as well as the complexity of the reward landscape to shape these values. If anything, this comparison exposes the limitations of current alignment regimes: they compress the complex social world into a narrow evaluative binary. To achieve robust reasoning, future models may need exposure to diverse socialization environments, analogous to how individuals develop more nuanced cognition through multicultural interaction.

Another interpretation is that post-training approximates not socialization but virtue formation. On this view, preference optimization is akin to cultivating intellectual virtues—honesty, humility, fairness—by rewarding behaviors that exemplify them. Anthropic’s Constitutional AI, for example, attempts to codify such virtues explicitly. However, without independent mechanisms for critique or verification, these virtues risk devolving into performative ethics. There exists a possibility for models to cite these constitutions while failing to behave faithfully. True virtue formation would require rigorous checks and perhaps a variety of viewpoints, while calling post-training virtue formation would be no more true than calling it a form of socialization.

Finally, one may argue that language models are also trained to be truth-seeking, specifically in the fields of mathematics and computer science, and that this may internally shift the models to value “truth” as well as likability. Specifically, models are trained on reasoning traces of solutions to correct math problems to imitate their thought process, and they are graded using reinforcement learning on their final solutions. I argue that this still lacks the crucial step-by-step feedback of searching for “truth” at every step, rather still overemphasizing the result. The failure of models to prioritize “truth” just because they are evaluated on correctness in fields like math is demonstrated in the Anthropic interpretability blog (2025), where given an unsolvable math problem, models just affirm the given answer instead of seeking the “true” answer. Their main objective is still to produce responses the user will like, which is a form of socialization.

5 Conclusion

The invention of large language models reflected an impressive advancement in computation and computer science, but the advancement in language models has also shown a convergence between machine learning and the psychology of reasoning. Architectural innovations explain the majority of increase in raw capability, but the stage that transforms the model to become useful to humans—post-training—operates through social mechanisms strikingly similar to those hypothesized to shape human cognition and reasoning. Supervised fine-tuning and RLHF function as institutionalized socialization: they teach models to mimic communal norms via SFT and reinforce this by optimizing the model to generate responses users like through RLHF. The resulting systems display persuasive, coherent, and morally calibrated behavior, but with opaque internal functions.

Philosophical accounts of human reasoning reinforce this parallel. Nisbett and Wilson’s findings on confabulation, Schwitzgebel’s critique of introspection, Mercier and Sperber’s theory of argumentation, and Dennett’s narrative model of the self all describe a species whose rationality is mediated by social interaction rather than introspective transparency. LLMs, optimized for human preference, show this trajectory in statistical methods. Their chain-of-thought explanations mirror human justifications: socially intelligible narratives that may obscure the underlying process; it is not necessarily a lie, but potentially a result of the next-token predictor not having access to the internal state of the model. As interpretability researchers such as Lindsey et al. (2025) reveal, beneath these narratives lie intricate computational “biologies” that operate independently of verbalized reasoning. The future of AI interpretability may lie in decoding the inner workings of a model and aligning chain-of-thought to its true thoughts.

Much like LLMs, humans are often incapable of explaining why they do things or inspecting their own reasoning. Unlike humans, however, this is something researchers can actively seek to improve in LLMs in pursuit of the goal to make them more helpful assistants and to benefit society. Perhaps the greatest benefit of the LLM training paradigm we currently have may not just be their economic use, but by using LLMs as a way to understand human psychology, jointly advancing the fields of machine learning and psychology.

References

Anthropic (Producer). (2025, August 15). Interpretability: Understanding how AI models think [YouTube Video]. https://www.youtube.com/watch?v=fGKNUvivvnc
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., … Kaplan, J. (2022). Constitutional ai: Harmlessness from ai feedback (No. arXiv:2212.08073). arXiv. https://doi.org/10.48550/arXiv.2212.08073
Dennett, D. C. (1992). The self as a center of narrative gravity. In F. S. Kessel, P. M. Cole, & D. L. Johnson (Eds.), Self and consciousness: Multiple perspectives (pp. 103–115). Lawrence Erlbaum Associates.
Dennett, D. C. (2017). From bacteria to Bach and back: The evolution of minds (1st ed). W. W. Norton & Company, Incorporated.
Linsdey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., … Batson, J. (2025, March 27). On the biology of a large language model [Anthropic]. Transformer Circuits. https://transformer-circuits.pub/2025/attribution-graphs/biology.html
Mercier, H., & Sperber, D. (2011). Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences, 34(2), 57–74. https://doi.org/10.1017/S0140525X10000968
Meta AI. (2025, April 5). The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. https://doi.org/10.1037/0033-295X.84.3.231
OpenAI, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., Iftimie, A., Karpenko, A., Passos, A. T., Neitz, A., Prokofiev, A., Wei, A., Tam, A., Bennett, A., … Li, Z. (2024). Openai o1 system card (No. arXiv:2412.16720). arXiv. https://doi.org/10.48550/arXiv.2412.16720
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback (No. arXiv:2203.02155). arXiv. https://doi.org/10.48550/arXiv.2203.02155
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2024). Direct preference optimization: Your language model is secretly a reward model (No. arXiv:2305.18290). arXiv. https://doi.org/10.48550/arXiv.2305.18290
Schwitzgebel, E. (2008). The Unreliability of Naive Introspection. The Philosophical Review, 117(2), 245–273. https://www.jstor.org/stable/40606018
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting (No. arXiv:2305.04388). arXiv. https://doi.org/10.48550/arXiv.2305.04388
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need (No. arXiv:1706.03762). arXiv. https://doi.org/10.48550/arXiv.1706.03762