How Large Language Models Actually Work: The Science Behind ChatGPT, Claude and Gemini

There is an explanation most people have been given for how ChatGPT works. It goes something like this: it has read a lot of text on the internet, and it predicts the next word. Technically, this is not wrong. But it is about as complete as saying the human brain is a collection of cells that send electrical signals. Accurate at the most literal level. Entirely useless for understanding what is actually happening.

In June 2017, eight researchers at Google published a paper called Attention Is All You Need. It was 15 pages long. The abstract was modest. The title was almost playful.

It changed everything.

The architecture they described — the transformer — is the engine inside every large language model alive today. ChatGPT, Claude, Gemini, Llama, Mistral, Grok. Every model that has reshaped how humans interact with machines in the past three years runs on a direct descendant of what those eight researchers built.

You can watch the 2017 transformer paper reorganise an entire research literature in the archive. By 1 September 2026, arXiv held 33,797 papers with “transformer” in the title and 19,300 with “large language model”.

Both numbers describe work that did not exist before Attention Is All You Need was published. The architecture named in a fifteen-page paper now titles more arXiv papers than “dark matter” (20,400) or “entanglement” (21,225), the two largest physics literatures counted across this site.

Nothing about that proves the approach is correct or that it leads anywhere further. It measures how completely one idea absorbed a field, which is the phenomenon this article is actually about.

Before the Transformer: The Long Road from Neurons to Neural Networks

The story begins in 1943, when Warren McCulloch, a neurophysiologist, and Walter Pitts, a self-taught mathematical prodigy, published A Logical Calculus of the Ideas Immanent in Nervous Activity. It proposed the first mathematical model of a neuron: a threshold device that fires or does not, depending on whether the sum of its inputs passes a threshold.

This was the founding act of artificial neural networks. A biological mechanism, abstracted into mathematics. To understand how DNA encodes the genetic instructions that build and wire those biological neurons is to understand the real starting point of every AI system running today.

In 1958, Frank Rosenblatt at Cornell Aeronautical Laboratory built the Perceptron — the first trainable neural network, described in Psychological Review. It could learn to classify simple patterns by adjusting connection weights based on errors. It caused enormous excitement. It also hit a wall: the Perceptron could only solve linearly separable problems. XOR — a basic logical operation — defeated it entirely. By the early 1970s, AI funding had collapsed into what became known as the first “AI winter.”

The thaw came in 1986. Geoffrey Hinton at the University of Toronto, David Rumelhart at UC San Diego, and Ronald Williams published a paper in Nature — volume 323, pages 533–536 — demonstrating that backpropagation could be used to train multi-layer networks. The algorithm computed how much each weight in the network contributed to the final error, then adjusted every weight accordingly, propagating corrections backwards through the layers. Multi-layer networks could now learn complex, non-linear patterns.

Hinton shared the 2018 Turing Award with Yann LeCun, who developed convolutional neural networks for image recognition, and Yoshua Bengio, whose group pioneered neural language modelling. In 2024 Hinton shared the Nobel Prize in Physics for foundational work on neural networks, and has used the platform to warn about the technology’s dangers.

By the 2010s, the field was dominated by recurrent neural networks (RNNs) and specifically Long Short-Term Memory (LSTM) networks, developed by Sepp Hochreiter and Jürgen Schmidhuber in 1997 and published in Neural Computation. LSTMs introduced memory cells that could preserve information across long sequences — critical for language, where the meaning of a word can depend on context from dozens of words earlier.

But LSTMs processed text one token at a time, which could not be parallelised efficiently, and earlier context faded over long sequences. The field needed something fundamentally different. It arrived in June 2017.

The 2017 Paper That Changed Everything

The full author list is worth recording: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin, all at Google Brain or Google Research. The paper was submitted on 12 June 2017 and presented at NeurIPS that year.

The core proposal was radical in its simplicity: abandon recurrence entirely. Rather than processing language sequentially, process the entire sequence simultaneously. Rather than carrying a hidden state forward through time, use a mechanism called self-attention that lets every word in a sequence directly attend to every other word at once, computing how relevant each is to understanding the current word’s meaning.

The immediate results were stunning. On the WMT 2014 English-to-German translation benchmark, the transformer achieved a BLEU score of 28.4 — more than 2 points above the previous state of the art, using a fraction of the training time. The architecture was also, crucially, massively parallelisable: because all positions in a sequence were processed simultaneously, training could be distributed across thousands of GPU cores in ways that sequential RNN processing could not.

What followed was a cascade of breakthroughs, each building on the transformer architecture.

BERT (2018), from Google, introduced pre-training a transformer on a massive text corpus by hiding words and training the model to predict them, then fine-tuning it for specific tasks. It beat the previous best results on eleven language benchmarks on release, and its pre-train-then-fine-tune recipe became the template for what followed.

GPT (2018) and GPT-2 (2019) — OpenAI (Radford, Narasimhan, Salimans, Sutskever). Where BERT used the transformer encoder, GPT used the decoder — processing text left to right and training to predict the next token. This autoregressive approach turned out to be the key to generation. GPT-2 generated such fluent prose that OpenAI initially delayed its full release, citing concerns about misuse.

GPT-3 (2020) — 175 billion parameters, trained on approximately 300 billion tokens (Brown and 30 co-authors). GPT-3 produced human-quality writing, answered factual questions, wrote code, and performed few-shot learning — tasks it had never been explicitly trained on — from just a handful of examples in the prompt. The age of the large language model had arrived.

What “Attention” Actually Means

The word attention in machine learning was not invented by the transformer paper. It was pioneered by Dzmitry Bahdanau at Université de Montréal, working with Kyunghyun Cho and Yoshua Bengio, in a 2015 paper submitted to ICLR titled Neural Machine Translation by Jointly Learning to Align and Translate. Their attention mechanism allowed a translation system to dynamically focus on different parts of the input sentence when generating each word of the output. The transformer took this idea and made it the entire architecture, discarding everything else.

Here is what attention actually computes, in plain language. Consider the sentence: The animal didn’t cross the street because it was too tired. What does “it” refer to? The animal or the street? A human reader resolves this instantly by connecting “it” to “animal” based on the semantic relationship between “tired” and a living creature. A sequential model has to carry this information forward through hidden states across multiple timesteps, and over long distances that information degrades. Attention solves this by allowing “it” to directly look at “animal” and “street” simultaneously and compute which is more relevant, in a single operation.

Technically, for each token in the sequence, the transformer computes three vectors: a Query (what this token is looking for in others), a Key (what information this token offers), and a Value (the actual content this token contributes).

The attention score between two tokens is computed by taking the dot product of the first token’s Query vector with the second token’s Key vector, scaling it, and passing it through a softmax function to produce a probability distribution over all other tokens. Each token then receives a weighted sum of all other tokens’ Value vectors, where the weights are determined by those attention scores.

This happens not once but in parallel across multiple attention heads — each head learning to attend to different types of relationships simultaneously. One head might learn syntactic dependencies such as subject-verb agreement. Another might track coreference — what “it” refers to. Another might capture semantic similarity. The outputs of all heads are concatenated and projected into a final representation. This is multi-head attention.

Think of how a researcher reads the conclusion of a paper while holding the methods, results and caveats in mind at once. Attention gives the model the same ability, which is why a chatbot can answer a question that depends on something you said twenty messages ago.

Tokens, Embeddings and the Strange Way Language Models Read

Large language models do not read words. They read tokens — chunks of text that are often, but not always, complete words or parts of words. The tokenisation scheme is determined at training time and fixed thereafter. GPT-4 uses a tokeniser called Byte Pair Encoding with a vocabulary of approximately 100,000 tokens. The word “genetics” might be one token. “Epigenetics” might be two: “epi” and “genetics.”

Representing words as numerical vectors was formalised by Tomas Mikolov and colleagues at Google in 2013. Their Word2Vec placed words with related meanings close together in a vector space; the famous example is King − Man + Woman ≈ Queen. The model had learned semantic relationships as geometry, without human labelling.

In a modern transformer, each token is represented as an embedding vector of typically 768 to 12,288 numbers, depending on model size. Unlike Word2Vec, which gives each word a fixed representation, transformer embeddings are contextual: the embedding for “bank” in “river bank” is a different vector from “bank” in “central bank,” because the attention mechanism has incorporated context from surrounding tokens into each representation.

Because transformers process all tokens at once, they need a way to encode word order; otherwise “dog bites man” and “man bites dog” would look the same. The original transformer added sine and cosine signals of different frequencies to each token’s embedding; modern models use schemes such as RoPE and ALiBi, which cope better with long sequences.

The strangeness worth dwelling on is this: the embedding space is not designed by any human. Nobody decided that “Paris” and “France” should be related, or that “mitosis” and “meiosis” should be nearby but not identical. The geometry of meaning builds itself, entirely from training. The model learns what things mean because it learns to predict what comes next — and meaning, in the statistical structure of human language, leaves a recoverable fingerprint in patterns of co-occurrence.

Training: How an LLM Learns from Trillions of Words

The training of a large language model happens in two phases. The first is pretraining. The second is alignment. Both matter enormously, and they operate on radically different scales.

Pretraining: the scale of it

During pretraining, the model is given a massive corpus of text and trained on a single objective: predict the next token. The corpus for GPT-3 included Common Crawl (filtered web text), WebText2, Books1, Books2, and English Wikipedia — approximately 300 billion tokens. For GPT-4 and subsequent models, training data is estimated in the range of several trillion tokens.

For each token, the model sees all preceding tokens and must predict the next one. The prediction is compared to the true next token using cross-entropy loss. The error is propagated backwards through the network using the backpropagation algorithm that Hinton and Rumelhart demonstrated in 1986, and all the model’s weights are nudged in directions that reduce the error. This is done using a variant of stochastic gradient descent, most commonly the Adam optimiser (Kingma and Ba, 2015).

The numbers are difficult to hold in mind. GPT-3 has 175 billion parameters, each a single floating-point number, and training it took an estimated 3.14 × 10^23 floating-point operations.

Alignment: making the model useful and safe

A pretrained model is competent but uncontrolled. It generates plausible text continuations, which means it can write harmful content as readily as a recipe. It has no concept of what a helpful, honest, harmless response looks like — it has only learned statistical patterns.

RLHF (Reinforcement Learning from Human Feedback) was the technique that turned pretrained models into useful assistants. Described in a landmark paper by Long Ouyang and colleagues at OpenAI, published at NeurIPS 2022, it works in three steps: human raters compare pairs of model outputs and indicate which is better; a separate reward model is trained to predict which outputs humans prefer; and the main language model is then fine-tuned using reinforcement learning to maximise the reward model’s score. This is what makes the difference between a raw statistical next-token predictor and an assistant that maintains a coherent, useful persona.

Anthropic, the company behind Claude, developed an alternative approach: Constitutional AI, described by Yuntao Bai and colleagues in an arXiv paper published December 2022. Rather than relying entirely on human feedback, it gives the model a set of written principles — a constitution — and trains it to critique and revise its own outputs against those principles. The result is an alignment approach that is more scalable and, Anthropic argues, more interpretable than pure RLHF.

Emergence: The Abilities Nobody Programmed In

In August 2022, a team of researchers at Google Brain and Stanford, led by Jason Wei, published a paper in Transactions on Machine Learning Research titled Emergent Abilities of Large Language Models. It catalogued something that had been noticed informally but never formally characterised: certain capabilities of language models are essentially absent in small models and present in large models, appearing apparently suddenly as model scale increases.

Examples they catalogued include chain-of-thought reasoning (breaking a multi-step problem into intermediate steps, dramatically improving accuracy on maths and logic), multi-step arithmetic, word unscrambling (near-random below roughly 100 billion parameters, near-perfect above), and sharp improvements on many of the 204 tasks in the BIG-bench benchmark suite at particular model scales.

The idea echoes what biologists know from complex systems: properties of a large system that cannot be predicted from its parts. At what point do statistical patterns of language prediction give rise to something that looks like reasoning? The question connects to our article on the hard problem of consciousness.

The scientific counterargument

Not everyone accepts that emergence is real rather than an artefact of measurement. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo from Stanford published a paper at NeurIPS 2023 titled Are Emergent Abilities of Large Language Models a Mirage? They argued that the apparent sharpness of emergent transitions is an artefact of using discontinuous, threshold-based metrics. When you replace pass/fail benchmarks with continuous metrics, many emergent abilities look like smooth improvements that were present all along, merely invisible to the binary measurement scheme.

This debate has not been resolved, and it matters practically. If emergent abilities are smooth and predictable, the behaviour of future models can in principle be extrapolated from current ones. If they are genuinely sudden and discontinuous — real phase transitions in capability — then larger models may surprise us in ways we cannot anticipate.

How ChatGPT, Claude and Gemini Differ

All three major frontier models share the same foundational architecture — the transformer. But the choices made in training, alignment, and design philosophy produce models with genuinely different characteristics, strengths, and failure modes.

ChatGPT (OpenAI) uses a decoder-only transformer aligned with RLHF; GPT-4o, released in May 2024, processes images, audio and text in a single model.

Claude comes from Anthropic, founded in 2021 by Dario and Daniela Amodei with other former OpenAI researchers. It is aligned with Constitutional AI, and Anthropic has invested heavily in mechanistic interpretability: its 2021 paper A Mathematical Framework for Transformer Circuits identified induction heads, attention circuits that enable in-context learning.

Gemini (Google DeepMind) comes from the 2023 merger of Google Brain, which wrote Attention Is All You Need, and DeepMind, and was trained on text, images, audio, video and code from the start. Its December 2023 technical report said Gemini Ultra beat GPT-4 on 30 of 32 academic benchmarks.

The Biology Connection: How LLMs Are Reshaping Genetics Research

The architecture developed to understand human language turns out to be extraordinarily powerful for reading another kind of sequence: the nucleotides in DNA, the amino acids in proteins, the regulatory grammar of the genome. This is not a metaphor. The transformer architecture is being applied directly to biological sequences, with results that are transforming molecular biology — the defining throughline of this publication, where AI and genetics meet.

AlphaFold and the protein language model revolution

In November 2020, DeepMind’s AlphaFold 2 — led by John Jumper — solved the protein folding problem: predicting the three-dimensional structure of a protein from its amino acid sequence, a problem unsolved for 50 years. AlphaFold used a transformer-based architecture trained on known protein structures. The 2021 paper in Nature (Jumper et al., volume 596, pages 583–589) described a system that predicted protein structures with accuracy matching experimental methods. In 2024, Jumper and Hassabis shared the Nobel Prize in Chemistry for this work.

Meta AI’s ESMFold took this further: Lin et al. (2023, Science, vol. 379, 1123–1130) trained a pure language model on 250 million protein sequences, treating amino acid letters exactly as a language model treats text tokens. The model learned protein structure prediction from sequence alone — no evolutionary information required — at a fraction of AlphaFold’s computational cost. Protein sequences are a language, and LLMs can read them.

AlphaGenome: the transformer reading non-coding DNA

In 2025 Google DeepMind released AlphaGenome, a transformer model trained to decode the 98 per cent of the human genome that does not code for proteins: the regulatory regions that set when, where and how much each gene is expressed, sometimes from thousands of base pairs away.

This is exactly the kind of long-range dependency that attention was designed to capture. The biology of the genome and the architecture of the transformer are, at a fundamental level, solving the same problem: understanding meaning at a distance.

This has direct implications for understanding how CRISPR gene-editing tools can be designed to target specific genomic sequences with greater precision, since such models can predict the downstream effects of edits in regulatory regions that were previously opaque to researchers.

Nucleotide transformers and genomic foundation models

In 2023 Hugo Dalla-Torre and colleagues at InstaDeep and Nvidia described the Nucleotide Transformer in Nature Methods, trained on 3,202 human genomes and 850 genomes from other species. It could predict gene expression, chromatin accessibility and the effects of genetic variants, tasks that previously needed separate models.

The implications extend to cancer genetics: genomic foundation models can be fine-tuned to identify somatic mutations, predict which variants affect splicing, and model the effect of specific DNA changes on protein function — all from raw sequence, at a scale that manual analysis could never approach.

AI drug discovery

Generative models are also being used to propose drug candidates, which connects to how RNA biology is being leveraged in medicine: the same architecture that reads language can be pointed at the molecular language of drug-target binding.

What LLMs Cannot Do — the Honest Limits

The capabilities are real and growing, but an honest account needs a clear description of what these systems cannot do, because the gap between appearance and mechanism causes more confusion than almost anything else in science communication.

They have no persistent memory. Without external tooling, each conversation starts from zero. A language model has no memory of previous conversations and cannot learn from interactions after training has ended. Its knowledge is frozen at the training data cutoff. This is why a model will tell you it has a knowledge cutoff, and why it may be unaware of events after that date.

They cannot do reliable arithmetic natively. They predict numerals that tend to follow arithmetic patterns, so simple sums are often right, but they do not apply precise rules the way a calculator does, which is why chatbots are usually given a calculator or code interpreter.

Nobody fully understands why they work. Mechanistic interpretability is still young. Anthropic’s 2021 framework identified algorithms implemented by attention heads, and in 2023 Wes Gurnee and Max Tegmark found that Llama-2 models learn internal representations of space and time. But understanding individual circuits is a long way from understanding a model with hundreds of billions of parameters. The Stochastic Parrots paper (Bender et al., FAccT 2021) argued that LLMs produce the form of language without grounded meaning.

Whether this critique is correct, partially correct, or substantially wrong remains one of the most actively contested questions in AI science. The honest answer is: we do not fully know.

These questions — about understanding, and what it means for a system to “know” something — connect directly to the deepest unsolved problems in neuroscience. The same difficulty that makes consciousness hard to define in biological organisms makes it equally hard to determine whether artificial ones possess anything analogous. For a deeper look, see our article on the hard problem of consciousness.

One clarification is needed before testing any of this on a chatbot. The limits above describe the model. What people use is a product that wraps the model in machinery it does not itself possess: a search tool, a calculator or code interpreter, a store that carries notes between conversations. A system that cites a source or gets long division right may be showing off a good tool rather than a smarter network, so the question worth asking is always which part of the assembly did the work.

Frequently Asked Questions

Sources

1. Vaswani, A. et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30. arxiv.org/abs/1706.03762
2. Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019. arxiv.org/abs/1810.04805
3. Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS, 33. arxiv.org/abs/2005.14165
4. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS, 35. arxiv.org/abs/2203.02155
5. Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. Anthropic. arxiv.org/abs/2212.08073
6. Wei, J. et al. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. arxiv.org/abs/2206.07682
7. Schaeffer, R., Miranda, B., Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS, 36. arxiv.org/abs/2304.15004
8. Jumper, J. et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589. doi.org/10.1038/s41586-021-03819-2
9. Lin, Z. et al. (2023). Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637), 1123–1130. doi.org/10.1126/science.ade2574
10. Dalla-Torre, H. et al. (2023). The Nucleotide Transformer. Nature Methods. doi.org/10.1101/2023.01.11.523679
11. Bahdanau, D., Cho, K., Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arxiv.org/abs/1409.0473
12. Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. doi.org/10.1145/3442188.3445922
13. McCulloch, W.S., Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115–133.
14. Hinton, G.E., Rumelhart, D.E., Williams, R.J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.
15. Hochreiter, S., Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
16. Mikolov, T. et al. (2013). Efficient Estimation of Word Representations in Vector Space. ICLR 2013. arxiv.org/abs/1301.3781
17. Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. transformer-circuits.pub
18. Gemini Team, Google (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805. arxiv.org/abs/2312.11805
19. OpenAI (2023). GPT-4 Technical Report. arxiv.org/abs/2303.08774
20. Kingma, D.P., Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. arxiv.org/abs/1412.6980