Skip to content

Chapter 4

How a language model works, in plain English


The chapter in a minute

  • Training turns the dials of a model using a very large amount of text, long before you ever use it.
  • A model reads and writes tokens, small pieces of text, and builds its reply one likely piece at a time.
  • It writes likely text and doesn't look facts up by itself, so a smooth answer can still be wrong.
  • What a model knows stops at its training, unless a tool such as search gives it fresh information.
  • The context window limits how much it can take in at once, so in a long chat, repeat what matters.

Try it now

Find the edge

0 of 7 steps done

Step 1 of 7

Pick something that happened in the last few days and that you can check for yourself. It might be a local match result, a news headline or an event in your town. Aoife picks her duck race. Keep names and personal details out, as always.

Progress is saved in this browser only.

Tools and settings, as of a date

The book points to the ‘How it works’ section of the companion page. Here it is in full, with the date each detail was checked. Open it on the companion page

Chapter 4 points here, with "rough token numbers, training cut-off dates and context window sizes" for current models.

Where training text comes from

Two providers' own accounts of their data sources, filters and user data settings:

Checked: 2 November 2026. Chapter 8 of the book covers the user data settings. See the "Settings" section.

Tokens, by the numbers

  • OpenAI. For English text, 1 token is about 4 characters, or about three quarters of a word, so 100 tokens are about 75 words. These are estimates, not exact counts, and other languages work out differently. OpenAI, "Understanding and counting tokens" (opens in a new tab) (page showed "Updated: 13 days ago").
  • Anthropic. On Claude models before Claude Opus 4.7, a token is about 3.5 English characters. Claude Opus 4.7 and later models use a newer tokenizer that makes about 30 percent more tokens from the same text. Anthropic, Glossary, "Tokens" (opens in a new tab). For the current models, Anthropic's Models overview (opens in a new tab) puts 1 million tokens at roughly 555,000 words or 2.5 million characters, which is about 2.5 characters a token.

Counts depend on the model and the language, so treat all of these as rough guides.

Checked: 2 November 2026.

Training cut-off dates

A model's cut-off is the point its training text stops. Anthropic publishes a "reliable knowledge cutoff" for each model, which it describes as "the date through which the model's knowledge is most extensive and reliable". OpenAI publishes a "knowledge cutoff" for each model.

ProviderModelCut-off
AnthropicClaude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5June 2026 (reliable knowledge cut-off and training data cut-off)
OpenAIGPT-6 Astra30 April 2026
OpenAIGPT-6.1 Sol30 April 2026
OpenAIGPT-6 Luna18 May 2026
OpenAIGPT-6.1 Terra1 June 2026

Sources: Anthropic, Models overview (opens in a new tab) and OpenAI, Models (opens in a new tab). Cut-off dates change with every new model.

Checked: 2 November 2026.

Context window sizes

The context window is how much a model can take into account at one time, counted in tokens.

  • Anthropic. Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5 and Claude Haiku 5.5 each have a 1 million token context window, and can write up to 128,000 tokens in one reply. Some older Claude models have 200,000 tokens. Anthropic, "Context windows" (opens in a new tab).
  • OpenAI. GPT-6 Astra has a context window of 1,050,000 tokens and can write up to 128,000 tokens in one reply (model page (opens in a new tab)). OpenAI's Models page (opens in a new tab) lists the same size for its other current flagship models. OpenAI's help page says these limits differ between models.
  • Inside the chat apps the window can be smaller. ChatGPT's pricing page (opens in a new tab) gives 27,000 tokens (about 12 pages of text) for its quick "Instant" model on the Free plan, and says part of the window is used for instructions and memory, so less is left for what you type. Claude's pricing page (opens in a new tab) says "Up to 1M", varying by model, on every personal plan.

Checked: 2 November 2026.

Search settings

How search is switched on or chosen in two assistants: see "Web search inside an assistant" under Everyday AI.

The 2025 news study in full

European Broadcasting Union and BBC, News Integrity in AI Assistants: An international PSM study (opens in a new tab), report page dated 21 October 2025 (press release 22 October 2025).

  • 45% of answers had at least one significant issue, 31% had sourcing problems and 20% had accuracy problems.
  • 2,709 answers to core news questions, from 22 public service media organisations in 18 countries and 14 languages.
  • Free versions of four named assistants. Answers were gathered in late May and early June 2025.
  • Outdated information was one of the most common issues (page 15).

Checked: 2 November 2026. This is still the newest round: the EBU's report page lists no later edition.

From the book

It writes likely text. It does not look things up by itself.
Your First Rung, chapter 4

Sources

  1. Context windows (opens in a new tab) · Anthropic, 2026
  2. Enable and use web search (opens in a new tab) · Anthropic, 2026
  3. Glossary (opens in a new tab) · Anthropic, 2026
  4. Models overview (opens in a new tab) · Anthropic, 2026
  5. Anthropic's Transparency Hub (opens in a new tab) · Anthropic, 2026
  6. Language Models are Few-Shot Learners (opens in a new tab) · Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and Winter, Clemens and Hesse, Christopher and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario, 2020
  7. News Integrity in AI Assistants: An international PSM study (opens in a new tab) · Fletcher, James and Verckist, Dorien, 2025
  8. Artificial Intelligence Playbook for the UK Government (opens in a new tab) · Government Digital Service, 2025
  9. The Curious Case of Neural Text Degeneration (opens in a new tab) · Holtzman, Ari and Buys, Jan and Du, Li and Forbes, Maxwell and Choi, Yejin, 2020
  10. Survey of Hallucination in Natural Language Generation (opens in a new tab) · Ji, Ziwei and Lee, Nayeon and Frieske, Rita and Yu, Tiezheng and Su, Dan and Xu, Yan and Ishii, Etsuko and Bang, Ye Jin and Madotto, Andrea and Fung, Pascale, 2023
  11. Lost in the Middle: How Language Models Use Long Contexts (opens in a new tab) · Liu, Nelson F. and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy, 2024
  12. What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (opens in a new tab) · Michael, Julian and Holtzman, Ari and Parrish, Alicia and Mueller, Aaron and Wang, Alex and Chen, Angelica and Madaan, Divyam and Nangia, Nikita and Pang, Richard Yuanzhe and Phang, Jason and Bowman, Samuel R., 2023
  13. The debate over understanding in AI's large language models (opens in a new tab) · Mitchell, Melanie and Krakauer, David C., 2023
  14. NoLiMa: Long-Context Evaluation Beyond Literal Matching (opens in a new tab) · Modarressi, Ali and Deilamsalehy, Hanieh and Dernoncourt, Franck and Bui, Trung and Rossi, Ryan A. and Yoon, Seunghyun and Sch\"utze, Hinrich, 2025
  15. Introducing ChatGPT search (opens in a new tab) · OpenAI, 2024
  16. How ChatGPT and our foundation models are developed (opens in a new tab) · OpenAI, 2026
  17. Understanding and counting tokens (opens in a new tab) · OpenAI, 2026
  18. Neural Machine Translation of Rare Words with Subword Units (opens in a new tab) · Sennrich, Rico and Haddow, Barry and Birch, Alexandra, 2016