How Large Language Models Actually Work: The Science Behind ChatGPT, Claude and Gemini

There is an explanation most people have been given for how ChatGPT works. It goes something like this: it has read a lot of text on the internet, and it predicts the next word. Technically, this is not wrong. But it is about as complete as saying the human brain is a collection of cells that send electrical signals. Accurate at the most literal level. Entirely useless for understanding what is actually happening.

In June 2017, eight researchers at Google Brain — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin — published a paper called Attention Is All You Need. It was presented at NeurIPS, the world’s most prestigious machine learning conference. It was 15 pages long. The abstract was modest. The title was almost playful.

It changed everything.

The architecture they described — the transformer — is the engine inside every large language model alive today. ChatGPT, Claude, Gemini, Llama, Mistral, Grok. Every model that has reshaped how humans interact with machines in the past three years runs on a direct descendant of what those eight researchers built.

Understanding what a transformer actually does — at the level of the science — is understanding what this moment in history actually is. This article covers all of it: the science, the key researchers, the landmark papers, the biology connections, and the honest limits of what these systems can do.

Before the Transformer: The Long Road from Neurons to Neural Networks

The story of large language models does not begin in 2017. It begins in 1943, in a collaboration between a neuroscientist and a logician that most people have never heard of.

Warren McCulloch, a neurophysiologist at the University of Illinois, and Walter Pitts, a mathematical prodigy who had taught himself formal logic as a teenager, published a paper in the Bulletin of Mathematical Biophysics titled A Logical Calculus of the Ideas Immanent in Nervous Activity. In it, they proposed the first mathematical model of a neuron: a simple threshold device that either fires or does not fire, depending on whether the sum of its inputs exceeds a threshold value. Binary. All or nothing. Just like the biological neuron they were modelling.

This was the founding act of artificial neural networks. A biological mechanism, abstracted into mathematics. To understand how DNA encodes the genetic instructions that build and wire those biological neurons is to understand the real starting point of every AI system running today.

In 1958, Frank Rosenblatt at Cornell Aeronautical Laboratory built the Perceptron — the first trainable neural network, described in Psychological Review. It could learn to classify simple patterns by adjusting connection weights based on errors. It caused enormous excitement. It also hit a wall: the Perceptron could only solve linearly separable problems. XOR — a basic logical operation — defeated it entirely. By the early 1970s, AI funding had collapsed into what became known as the first “AI winter.”

The thaw came in 1986. Geoffrey Hinton at the University of Toronto, David Rumelhart at UC San Diego, and Ronald Williams published a paper in Nature — volume 323, pages 533–536 — demonstrating that backpropagation could be used to train multi-layer networks. The algorithm computed how much each weight in the network contributed to the final error, then adjusted every weight accordingly, propagating corrections backwards through the layers. Multi-layer networks could now learn complex, non-linear patterns.

Hinton shared the 2018 Turing Award — computing’s Nobel Prize — with Yann LeCun (Bell Labs, now Meta AI) and Yoshua Bengio (Université de Montréal). LeCun developed convolutional neural networks (CNNs) in 1989, which proved extraordinarily powerful for image recognition. Bengio’s group pioneered language modelling with neural networks and developed key ideas about word representations that would lead directly to modern language AI. In 2024, Hinton was awarded the Nobel Prize in Physics for his foundational contributions to the field he helped create — and then spent part of his platform warning the world about the dangers it presents.

By the 2010s, the field was dominated by recurrent neural networks (RNNs) and specifically Long Short-Term Memory (LSTM) networks, developed by Sepp Hochreiter and Jürgen Schmidhuber in 1997 and published in Neural Computation. LSTMs introduced memory cells that could preserve information across long sequences — critical for language, where the meaning of a word can depend on context from dozens of words earlier.

But LSTMs had a fundamental constraint: they processed sequences word by word, one token at a time, carrying a hidden state forward. This sequential processing could not be parallelised efficiently on modern hardware. Long sequences caused the hidden state to become saturated — earlier context was progressively lost. And training was slow. The field needed something fundamentally different. The solution arrived in June 2017.

The 2017 Paper That Changed Everything

The full author list of the paper that launched the modern AI era is worth recording: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. All were at Google Brain or Google Research at the time of publication. The paper was submitted on 12 June 2017, accepted at NeurIPS that same year, and has since been cited well over 130,000 times — making it one of the most influential scientific papers in the history of computer science.

The core proposal was radical in its simplicity: abandon recurrence entirely. Rather than processing language sequentially, process the entire sequence simultaneously. Rather than carrying a hidden state forward through time, use a mechanism called self-attention that lets every word in a sequence directly attend to every other word at once, computing how relevant each is to understanding the current word’s meaning.

The immediate results were stunning. On the WMT 2014 English-to-German translation benchmark, the transformer achieved a BLEU score of 28.4 — more than 2 points above the previous state of the art, using a fraction of the training time. The architecture was also, crucially, massively parallelisable: because all positions in a sequence were processed simultaneously, training could be distributed across thousands of GPU cores in ways that sequential RNN processing could not.

What followed was a cascade of breakthroughs, each building on the transformer architecture.

BERT (2018) — Bidirectional Encoder Representations from Transformers, from Google (Devlin, Chang, Lee, and Toutanova; published in the proceedings of NAACL-HLT 2019). BERT introduced the idea of pre-training a transformer on a massive text corpus using masked language modelling — randomly hiding words and training the model to predict them from context — then fine-tuning on specific downstream tasks. It demolished eleven NLP benchmarks simultaneously on release. The pre-train-then-fine-tune paradigm it established became the template for everything that followed.

GPT (2018) and GPT-2 (2019) — OpenAI (Radford, Narasimhan, Salimans, Sutskever). Where BERT used the transformer encoder, GPT used the decoder — processing text left to right and training to predict the next token. This autoregressive approach turned out to be the key to generation. GPT-2 generated such fluent prose that OpenAI initially delayed its full release, citing concerns about misuse.

GPT-3 (2020) — 175 billion parameters, trained on approximately 300 billion tokens (Brown and 30 co-authors). GPT-3 produced human-quality writing, answered factual questions, wrote code, and performed few-shot learning — tasks it had never been explicitly trained on — from just a handful of examples in the prompt. The age of the large language model had arrived.

What “Attention” Actually Means

The word attention in machine learning was not invented by the transformer paper. It was pioneered by Dzmitry Bahdanau at Université de Montréal, working with Kyunghyun Cho and Yoshua Bengio, in a 2015 paper submitted to ICLR titled Neural Machine Translation by Jointly Learning to Align and Translate. Their attention mechanism allowed a translation system to dynamically focus on different parts of the input sentence when generating each word of the output. The transformer took this idea and made it the entire architecture, discarding everything else.

Here is what attention actually computes, in plain language. Consider the sentence: The animal didn’t cross the street because it was too tired. What does “it” refer to? The animal or the street? A human reader resolves this instantly by connecting “it” to “animal” based on the semantic relationship between “tired” and a living creature. A sequential model has to carry this information forward through hidden states across multiple timesteps, and over long distances that information degrades. Attention solves this by allowing “it” to directly look at “animal” and “street” simultaneously and compute which is more relevant, in a single operation.

Technically, for each token in the sequence, the transformer computes three vectors: a Query (what this token is looking for in others), a Key (what information this token offers), and a Value (the actual content this token contributes).

The attention score between two tokens is computed by taking the dot product of the first token’s Query vector with the second token’s Key vector, scaling it, and passing it through a softmax function to produce a probability distribution over all other tokens. Each token then receives a weighted sum of all other tokens’ Value vectors, where the weights are determined by those attention scores.

This happens not once but in parallel across multiple attention heads — each head learning to attend to different types of relationships simultaneously. One head might learn syntactic dependencies such as subject-verb agreement. Another might track coreference — what “it” refers to. Another might capture semantic similarity. The outputs of all heads are concatenated and projected into a final representation. This is multi-head attention.

Think of it the way a researcher reads a scientific paper. When reading the conclusion, they are simultaneously holding in mind the methodology, the results tables, the original hypothesis stated in the abstract, and the caveats in the discussion — not reading the conclusion in isolation. Attention gives the model the same ability: to understand any part of a text in the context of every other part, simultaneously, in a single computational step. This is why, when you ask a chatbot a question that depends on something you said twenty messages ago, it can still answer accurately. The attention mechanism ranges across the entire context window at once.

Tokens, Embeddings and the Strange Way Language Models Read

Large language models do not read words. They read tokens — chunks of text that are often, but not always, complete words or parts of words. The tokenisation scheme is determined at training time and fixed thereafter. GPT-4 uses a tokeniser called Byte Pair Encoding with a vocabulary of approximately 100,000 tokens. The word “genetics” might be one token. “Epigenetics” might be two: “epi” and “genetics.”

The concept of representing words as dense numerical vectors was formalised by Tomas Mikolov and colleagues at Google in 2013, in a paper titled Efficient Estimation of Word Representations in Vector Space. Their system, Word2Vec, produced vectors of typically 100–300 numbers for each word. Words with related meanings ended up geometrically close to each other in this vector space. The now-famous example: King − Man + Woman ≈ Queen. The model had learned semantic relationships as geometry, without any human labelling.

In a modern transformer, each token is represented as an embedding vector of typically 768 to 12,288 numbers, depending on model size. Unlike Word2Vec, which gives each word a fixed representation, transformer embeddings are contextual: the embedding for “bank” in “river bank” is a different vector from “bank” in “central bank,” because the attention mechanism has incorporated context from surrounding tokens into each representation.

Because transformers process all tokens simultaneously rather than sequentially, they need a way to encode the order of tokens. Without this, “dog bites man” and “man bites dog” would produce the same representations. The original transformer solved this with positional encodings: sine and cosine functions of different frequencies added to each token’s embedding. Modern models use learned positional encodings or more sophisticated schemes like RoPE (Rotary Position Embedding) and ALiBi, which generalise better to sequences longer than those seen during training.

The strangeness worth dwelling on is this: the embedding space is not designed by any human. Nobody decided that “Paris” and “France” should be related, or that “mitosis” and “meiosis” should be nearby but not identical. The geometry of meaning builds itself, entirely from training. The model learns what things mean because it learns to predict what comes next — and meaning, in the statistical structure of human language, leaves a recoverable fingerprint in patterns of co-occurrence.

Training: How an LLM Learns from Trillions of Words

The training of a large language model happens in two phases. The first is pretraining. The second is alignment. Both matter enormously, and they operate on radically different scales.

Pretraining: the scale of it

During pretraining, the model is given a massive corpus of text and trained on a single objective: predict the next token. The corpus for GPT-3 included Common Crawl (filtered web text), WebText2, Books1, Books2, and English Wikipedia — approximately 300 billion tokens. For GPT-4 and subsequent models, training data is estimated in the range of several trillion tokens.

For each token, the model sees all preceding tokens and must predict the next one. The prediction is compared to the true next token using cross-entropy loss. The error is propagated backwards through the network using the backpropagation algorithm that Hinton and Rumelhart demonstrated in 1986, and all the model’s weights are nudged in directions that reduce the error. This is done using a variant of stochastic gradient descent, most commonly the Adam optimiser (Kingma and Ba, 2015).

The numbers are difficult to hold in mind. GPT-3 has 175 billion parameters — each a single floating-point number. Each forward pass performs roughly 350 billion floating-point operations. Training GPT-3 required an estimated 3.14 × 10²³ floating-point operations in total. The estimated compute cost of training GPT-4 exceeds 100 million dollars.

Alignment: making the model useful and safe

A pretrained model is competent but uncontrolled. It generates plausible text continuations, which means it can write harmful content as readily as a recipe. It has no concept of what a helpful, honest, harmless response looks like — it has only learned statistical patterns.

RLHF (Reinforcement Learning from Human Feedback) was the technique that turned pretrained models into useful assistants. Described in a landmark paper by Long Ouyang and colleagues at OpenAI, published at NeurIPS 2022, it works in three steps: human raters compare pairs of model outputs and indicate which is better; a separate reward model is trained to predict which outputs humans prefer; and the main language model is then fine-tuned using reinforcement learning to maximise the reward model’s score. This is what makes the difference between a raw statistical next-token predictor and an assistant that maintains a coherent, useful persona.

Anthropic, the company behind Claude, developed an alternative approach: Constitutional AI, described by Yuntao Bai and colleagues in an arXiv paper published December 2022. Rather than relying entirely on human feedback, it gives the model a set of written principles — a constitution — and trains it to critique and revise its own outputs against those principles. The result is an alignment approach that is more scalable and, Anthropic argues, more interpretable than pure RLHF.

Emergence: The Abilities Nobody Programmed In

In August 2022, a team of researchers at Google Brain and Stanford, led by Jason Wei, published a paper in Transactions on Machine Learning Research titled Emergent Abilities of Large Language Models. It catalogued something that had been noticed informally but never formally characterised: certain capabilities of language models are essentially absent in small models and present in large models, appearing apparently suddenly as model scale increases.

Examples they catalogued include chain-of-thought reasoning (breaking a multi-step problem into intermediate steps, dramatically improving accuracy on maths and logic), multi-step arithmetic, word unscrambling (near-random below roughly 100 billion parameters, near-perfect above), and sharp improvements on many of the 204 tasks in the BIG-bench benchmark suite at particular model scales.

The concept echoes something biologists know well from studying complex systems: emergent properties of a large system cannot be predicted by studying its components in isolation. The wetness of water cannot be derived from studying a single H²O molecule. The consciousness of a brain cannot be deduced from examining a single neuron. At what point do statistical patterns of language prediction give rise to something that looks, functionally, like reasoning? This connects directly to the questions raised in our article on the hard problem of consciousness — how complex systems develop new capabilities that none of their parts possess alone.

The scientific counterargument

Not everyone accepts that emergence is real rather than an artefact of measurement. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo from Stanford published a paper at NeurIPS 2023 titled Are Emergent Abilities of Large Language Models a Mirage? They argued that the apparent sharpness of emergent transitions is an artefact of using discontinuous, threshold-based metrics. When you replace pass/fail benchmarks with continuous metrics, many emergent abilities look like smooth improvements that were present all along, merely invisible to the binary measurement scheme.

This debate has not been resolved, and it matters practically. If emergent abilities are smooth and predictable, the behaviour of future models can in principle be extrapolated from current ones. If they are genuinely sudden and discontinuous — real phase transitions in capability — then larger models may surprise us in ways we cannot anticipate.

How ChatGPT, Claude and Gemini Differ

All three major frontier models share the same foundational architecture — the transformer. But the choices made in training, alignment, and design philosophy produce models with genuinely different characteristics, strengths, and failure modes.

ChatGPT / GPT-4 series (OpenAI) uses a decoder-only transformer trained with RLHF alignment. GPT-4o, released in May 2024, introduced native multimodality — processing images, audio, and text in a unified model. Key figures include Sam Altman (CEO), Greg Brockman (co-founder), and Ilya Sutskever (Chief Scientist until his departure in May 2024). The context window reaches up to 128,000 tokens in GPT-4 Turbo, and estimated parameters exceed one trillion, though OpenAI has not officially confirmed this. GPT-4 scored in the 90th percentile on the bar exam and numerous standardised tests, results reported in OpenAI’s technical report published March 2023.

Claude (Anthropic) was founded in 2021 by Dario Amodei and Daniela Amodei, along with several other former OpenAI researchers including Tom Brown (lead author of the GPT-3 paper) and Chris Olah (who leads Anthropic’s mechanistic interpretability research). Claude models use Constitutional AI alignment rather than pure RLHF. Anthropic has invested unusually heavily in mechanistic interpretability — the attempt to understand, at the level of individual circuits, what is actually happening inside the model. Olah’s team published A Mathematical Framework for Transformer Circuits (2021), identifying specific algorithms implemented by attention heads, such as induction heads that enable in-context learning. Claude 3 Opus reached a context window of 200,000 tokens, at the time one of the largest publicly available.

Gemini (Google DeepMind) comes from the team formed in April 2023 by the merger of Google Brain (which wrote Attention Is All You Need) and DeepMind (behind AlphaGo, AlphaFold, and AlphaGenome), led by Demis Hassabis. Gemini Ultra was designed with native multimodality from the ground up — trained simultaneously on text, images, audio, video, and code. The Gemini technical report (December 2023, arXiv:2312.11805) reported that Gemini Ultra outperformed GPT-4 on 30 of 32 academic benchmarks at launch, including becoming the first model to exceed human expert performance on the Massive Multitask Language Understanding (MMLU) benchmark.

The Biology Connection: How LLMs Are Reshaping Genetics Research

The architecture developed to understand human language turns out to be extraordinarily powerful for reading another kind of sequence: the nucleotides in DNA, the amino acids in proteins, the regulatory grammar of the genome. This is not a metaphor. The transformer architecture is being applied directly to biological sequences, with results that are transforming molecular biology — the defining throughline of this publication, where AI and genetics meet.

AlphaFold and the protein language model revolution

In November 2020, DeepMind’s AlphaFold 2 — led by John Jumper — solved the protein folding problem: predicting the three-dimensional structure of a protein from its amino acid sequence, a problem unsolved for 50 years. AlphaFold used a transformer-based architecture trained on known protein structures. The 2021 paper in Nature (Jumper et al., volume 596, pages 583–589) described a system that predicted protein structures with accuracy matching experimental methods. In 2024, Jumper and Hassabis shared the Nobel Prize in Chemistry for this work.

Meta AI’s ESMFold took this further: Lin et al. (2023, Science, vol. 379, 1123–1130) trained a pure language model on 250 million protein sequences, treating amino acid letters exactly as a language model treats text tokens. The model learned protein structure prediction from sequence alone — no evolutionary information required — at a fraction of AlphaFold’s computational cost. Protein sequences are a language, and LLMs can read them.

AlphaGenome: the transformer reading non-coding DNA

In 2025, Google DeepMind released AlphaGenome — a transformer model trained to decode what happens in the 98% of the human genome that does not code for proteins: the regulatory regions, enhancers, silencers, and non-coding RNA sequences that determine when, where, and how much of each gene is expressed. The model uses the same attention mechanism described in Attention Is All You Need to understand how a regulatory element far upstream influences the expression of a gene thousands of base pairs away.

This is exactly the kind of long-range dependency that attention was designed to capture. The biology of the genome and the architecture of the transformer are, at a fundamental level, solving the same problem: understanding meaning at a distance.

This has direct implications for understanding how CRISPR gene-editing tools can be designed to target specific genomic sequences with greater precision, since such models can predict the downstream effects of edits in regulatory regions that were previously opaque to researchers.

Nucleotide transformers and genomic foundation models

In 2023, Hugo Dalla-Torre and colleagues at InstaDeep and Nvidia published The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics in Nature Methods. They trained large language models on 3,202 human genomes and 850 additional genomes from diverse species, treating DNA as a sequence of nucleotide tokens. The resulting models could predict gene expression levels, chromatin accessibility, and the functional consequences of genetic variants — tasks that previously required separate, specialised models for each.

The implications extend to cancer genetics: genomic foundation models can be fine-tuned to identify somatic mutations, predict which variants affect splicing, and model the effect of specific DNA changes on protein function — all from raw sequence, at a scale that manual analysis could never approach.

AI drug discovery

Insilico Medicine used generative transformer models to design a novel drug candidate from scratch — identifying a target, generating a molecule to hit it, and moving it to Phase 2 clinical trials in under 30 months, a timeline that would typically take a decade and cost hundreds of millions of dollars. This connects directly to how RNA biology is being leveraged in medicine: the same AI systems that understand language can be directed to understand the molecular language of drug-target binding.

What Researchers Emphasise

The transformer’s designers have often expressed surprise at how much the attention mechanism alone could accomplish — that discarding recurrence and convolution entirely, and keeping only attention, was enough to surpass every previous approach. That sense of a simple idea outrunning expectations recurs throughout the field’s recent history.

Geoffrey Hinton has repeatedly stressed a more sobering point: that the field is deploying systems whose inner workings it cannot yet fully explain, and that this gap between capability and understanding should be treated as a central fact about modern AI rather than a footnote. Linguist Emily Bender, co-author of the influential Stochastic Parrots paper, has pressed a complementary challenge — that fluency is not the same as understanding, and that we should be careful about assuming a system that produces the form of language possesses the grounded meaning a human speaker has.

On the other side of that debate, interpretability researchers at Anthropic have reported that models trained only to predict text appear to develop internal representations of the world — of space, time, and entities — that nobody explicitly put there. And researchers building protein language models have emphasised that treating biological sequences as a language, and applying the very same transformer architecture that reads text, has become one of the most powerful ideas in computational biology of the past decade. These positions do not fully agree, which is precisely what makes the question of what these systems “understand” one of the most actively contested in the field.

What LLMs Cannot Do — the Honest Limits

The capabilities of large language models are real and rapidly expanding. But the honest scientific account requires a clear-eyed description of what these systems cannot do, because the gap between what they appear to do and what they are actually doing has generated more confusion than almost any other topic in contemporary science communication.

They have no persistent memory. Without external tooling, each conversation starts from zero. A language model has no memory of previous conversations and cannot learn from interactions after training has ended. Its knowledge is frozen at the training data cutoff. This is why a model will tell you it has a knowledge cutoff, and why it may be unaware of events after that date.

They cannot do reliable arithmetic natively. LLMs are pattern matchers trained on text. They predict numerals that tend to follow the patterns of arithmetic, which means they often get simple sums right because those patterns are strongly represented in training data. But they do not apply precise algorithmic rules the way a calculator does. Multi-step arithmetic without a code interpreter produces errors at a rate that makes native calculation unreliable — which is why every major LLM is given access to a code interpreter or calculator tool.

Nobody fully understands why they work. This is the most important limitation to acknowledge. The field of mechanistic interpretability is still in its early stages. Anthropic’s interpretability team has made real progress — their 2021 framework identified interpretable algorithms implemented by attention heads, and later work found that models develop internal representations of space, time, and other concepts that emerge from training without being explicitly defined. But the gap between understanding individual circuits and understanding the behaviour of a trillion-parameter model is vast. The Stochastic Parrots paper (Bender et al., FAccT 2021) argued that LLMs produce the form of language without grounded meaning — sophisticated pattern matching that mimics understanding without possessing it.

Whether this critique is correct, partially correct, or substantially wrong remains one of the most actively contested questions in AI science. The honest answer is: we do not fully know.

These questions — about understanding, and what it means for a system to “know” something — connect directly to the deepest unsolved problems in neuroscience. The same difficulty that makes consciousness hard to define in biological organisms makes it equally hard to determine whether artificial ones possess anything analogous. For a deeper look, see our article on the hard problem of consciousness.

Frequently Asked Questions

Sources

1. Vaswani, A. et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30. arxiv.org/abs/1706.03762
2. Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019. arxiv.org/abs/1810.04805
3. Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS, 33. arxiv.org/abs/2005.14165
4. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS, 35. arxiv.org/abs/2203.02155
5. Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. Anthropic. arxiv.org/abs/2212.08073
6. Wei, J. et al. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. arxiv.org/abs/2206.07682
7. Schaeffer, R., Miranda, B., Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS, 36. arxiv.org/abs/2304.15004
8. Jumper, J. et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589. doi.org/10.1038/s41586-021-03819-2
9. Lin, Z. et al. (2023). Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637), 1123–1130. doi.org/10.1126/science.ade2574
10. Dalla-Torre, H. et al. (2023). The Nucleotide Transformer. Nature Methods. doi.org/10.1101/2023.01.11.523679
11. Bahdanau, D., Cho, K., Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arxiv.org/abs/1409.0473
12. Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. doi.org/10.1145/3442188.3445922
13. McCulloch, W.S., Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115–133.
14. Hinton, G.E., Rumelhart, D.E., Williams, R.J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.
15. Hochreiter, S., Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
16. Mikolov, T. et al. (2013). Efficient Estimation of Word Representations in Vector Space. ICLR 2013. arxiv.org/abs/1301.3781
17. Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. transformer-circuits.pub
18. Gemini Team, Google (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805. arxiv.org/abs/2312.11805
19. OpenAI (2023). GPT-4 Technical Report. arxiv.org/abs/2303.08774
20. Kingma, D.P., Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. arxiv.org/abs/1412.6980