Chapter 4
How a language model works, in plain English
The chapter in a minute
- Training turns the dials of a model using a very large amount of text, long before you ever use it.
- A model reads and writes tokens, small pieces of text, and builds its reply one likely piece at a time.
- It writes likely text and doesn't look facts up by itself, so a smooth answer can still be wrong.
- What a model knows stops at its training, unless a tool such as search gives it fresh information.
- The context window limits how much it can take in at once, so in a long chat, repeat what matters.
Try it now
Find the edge
0 of 7 steps done
Step 1 of 7
Pick something that happened in the last few days and that you can check for yourself. It might be a local match result, a news headline or an event in your town. Aoife picks her duck race. Keep names and personal details out, as always.
Tools and settings, as of a date
The book points to the ‘How it works’ section of the companion page. Here it is in full, with the date each detail was checked. Open it on the companion page
Chapter 4 points here, with "rough token numbers, training cut-off dates and context window sizes" for current models.
Where training text comes from
Two providers' own accounts of their data sources, filters and user data settings:
- OpenAI, "How ChatGPT and our foundation models are developed" (opens in a new tab) (page showed "Updated: 5 days ago"). It names three main sources: information publicly available on the internet, information OpenAI partners with others to access, and information that users, human trainers and researchers provide or generate.
- Anthropic's Transparency Hub (opens in a new tab) (model report section dated 2 October 2026). Each model's summary has a "Training Data" part describing its mix of sources.
Checked: 2 November 2026. Chapter 8 of the book covers the user data settings. See the "Settings" section.
Tokens, by the numbers
- OpenAI. For English text, 1 token is about 4 characters, or about three quarters of a word, so 100 tokens are about 75 words. These are estimates, not exact counts, and other languages work out differently. OpenAI, "Understanding and counting tokens" (opens in a new tab) (page showed "Updated: 13 days ago").
- Anthropic. On Claude models before Claude Opus 4.7, a token is about 3.5 English characters. Claude Opus 4.7 and later models use a newer tokenizer that makes about 30 percent more tokens from the same text. Anthropic, Glossary, "Tokens" (opens in a new tab). For the current models, Anthropic's Models overview (opens in a new tab) puts 1 million tokens at roughly 555,000 words or 2.5 million characters, which is about 2.5 characters a token.
Counts depend on the model and the language, so treat all of these as rough guides.
Checked: 2 November 2026.
Training cut-off dates
A model's cut-off is the point its training text stops. Anthropic publishes a "reliable knowledge cutoff" for each model, which it describes as "the date through which the model's knowledge is most extensive and reliable". OpenAI publishes a "knowledge cutoff" for each model.
| Provider | Model | Cut-off |
|---|---|---|
| Anthropic | Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5 | June 2026 (reliable knowledge cut-off and training data cut-off) |
| OpenAI | GPT-6 Astra | 30 April 2026 |
| OpenAI | GPT-6.1 Sol | 30 April 2026 |
| OpenAI | GPT-6 Luna | 18 May 2026 |
| OpenAI | GPT-6.1 Terra | 1 June 2026 |
Sources: Anthropic, Models overview (opens in a new tab) and OpenAI, Models (opens in a new tab). Cut-off dates change with every new model.
Checked: 2 November 2026.
Context window sizes
The context window is how much a model can take into account at one time, counted in tokens.
- Anthropic. Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5 and Claude Haiku 5.5 each have a 1 million token context window, and can write up to 128,000 tokens in one reply. Some older Claude models have 200,000 tokens. Anthropic, "Context windows" (opens in a new tab).
- OpenAI. GPT-6 Astra has a context window of 1,050,000 tokens and can write up to 128,000 tokens in one reply (model page (opens in a new tab)). OpenAI's Models page (opens in a new tab) lists the same size for its other current flagship models. OpenAI's help page says these limits differ between models.
- Inside the chat apps the window can be smaller. ChatGPT's pricing page (opens in a new tab) gives 27,000 tokens (about 12 pages of text) for its quick "Instant" model on the Free plan, and says part of the window is used for instructions and memory, so less is left for what you type. Claude's pricing page (opens in a new tab) says "Up to 1M", varying by model, on every personal plan.
Checked: 2 November 2026.
Search settings
How search is switched on or chosen in two assistants: see "Web search inside an assistant" under Everyday AI.
The 2025 news study in full
European Broadcasting Union and BBC, News Integrity in AI Assistants: An international PSM study (opens in a new tab), report page dated 21 October 2025 (press release 22 October 2025).
- 45% of answers had at least one significant issue, 31% had sourcing problems and 20% had accuracy problems.
- 2,709 answers to core news questions, from 22 public service media organisations in 18 countries and 14 languages.
- Free versions of four named assistants. Answers were gathered in late May and early June 2025.
- Outdated information was one of the most common issues (page 15).
Checked: 2 November 2026. This is still the newest round: the EBU's report page lists no later edition.
From the book
It writes likely text. It does not look things up by itself.
Sources
- Context windows (opens in a new tab) · Anthropic, 2026
- Enable and use web search (opens in a new tab) · Anthropic, 2026
- Glossary (opens in a new tab) · Anthropic, 2026
- Models overview (opens in a new tab) · Anthropic, 2026
- Anthropic's Transparency Hub (opens in a new tab) · Anthropic, 2026
- Language Models are Few-Shot Learners (opens in a new tab) · Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and Winter, Clemens and Hesse, Christopher and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario, 2020
- News Integrity in AI Assistants: An international PSM study (opens in a new tab) · Fletcher, James and Verckist, Dorien, 2025
- Artificial Intelligence Playbook for the UK Government (opens in a new tab) · Government Digital Service, 2025
- The Curious Case of Neural Text Degeneration (opens in a new tab) · Holtzman, Ari and Buys, Jan and Du, Li and Forbes, Maxwell and Choi, Yejin, 2020
- Survey of Hallucination in Natural Language Generation (opens in a new tab) · Ji, Ziwei and Lee, Nayeon and Frieske, Rita and Yu, Tiezheng and Su, Dan and Xu, Yan and Ishii, Etsuko and Bang, Ye Jin and Madotto, Andrea and Fung, Pascale, 2023
- Lost in the Middle: How Language Models Use Long Contexts (opens in a new tab) · Liu, Nelson F. and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy, 2024
- What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (opens in a new tab) · Michael, Julian and Holtzman, Ari and Parrish, Alicia and Mueller, Aaron and Wang, Alex and Chen, Angelica and Madaan, Divyam and Nangia, Nikita and Pang, Richard Yuanzhe and Phang, Jason and Bowman, Samuel R., 2023
- The debate over understanding in AI's large language models (opens in a new tab) · Mitchell, Melanie and Krakauer, David C., 2023
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (opens in a new tab) · Modarressi, Ali and Deilamsalehy, Hanieh and Dernoncourt, Franck and Bui, Trung and Rossi, Ryan A. and Yoon, Seunghyun and Sch\"utze, Hinrich, 2025
- Introducing ChatGPT search (opens in a new tab) · OpenAI, 2024
- How ChatGPT and our foundation models are developed (opens in a new tab) · OpenAI, 2026
- Understanding and counting tokens (opens in a new tab) · OpenAI, 2026
- Neural Machine Translation of Rare Words with Subword Units (opens in a new tab) · Sennrich, Rico and Haddow, Barry and Birch, Alexandra, 2016