Skip to content

Chapter 3

From neural networks to language models and agents


The chapter in a minute

  • A neural network is many simple units in layers. Training turns the strength of each connection a little after every mistake.
  • Deep learning uses many layers. After a deep network's clear win in a 2012 picture challenge, most teams in that challenge switched to deep networks within a year.
  • Translation moved from hand-written rules, to counting phrases in translated text, to neural networks that work on whole sentences.
  • A 2017 design built on attention, first made for translation, was the base for the large language models of the following years. Chat assistants opened to the public at the end of 2022.
  • Agents are the newest step. They can act for you as well as answer, and you approve first.

Try it now

Your own timeline

0 of 13 steps done

Step 1 of 13

Copy these eight milestones onto your paper, one per line, in the jumbled order shown here:

Progress is saved in this browser only.

Tools and settings, as of a date

The book points to the ‘Timeline’ section of the companion page. Here it is in full, with the date each detail was checked. Open it on the companion page

Chapters 2 and 3 both point here. Chapter 2 points to "further reading for this chapter". Chapter 3 points to "the names behind each milestone and further reading". Both are on this one section.

Chapter 2: further reading with a free copy

These are the sources that have an official or author-posted free copy. University course copies were used for checking only and are not linked.

Checked: 2 November 2026 (every link works).

Chapter 3: the names behind the milestones

Chapter 3 names no people or products in print. Here are the names and the papers. Every link was checked on 2 November 2026.

MilestoneWho and whatWhere to read it
1943, a simple mathematical model of a neuronWarren S. McCulloch and Walter Pitts, "A logical calculus of the ideas immanent in nervous activity"First published in The Bulletin of Mathematical Biophysics 5(4), December 1943, pp. 115 to 133, DOI 10.1007/BF02478259 (opens in a new tab); reprinted in Bulletin of Mathematical Biology 52(1/2), 1990
1958, the perceptron describedFrank Rosenblatt, "The perceptron: A probabilistic model for information storage and organization in the brain". Described in 1958, not built then.Psychological Review 65(6), 1958, pp. 386 to 408, DOI 10.1037/h0042519 (opens in a new tab)
1986, training networks with hidden layers made widely knownDavid E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, "Learning representations by back-propagating errors", published 9 October 1986Nature 323(6088), pp. 533 to 536, DOI 10.1038/323533a0 (opens in a new tab)
2012, a deep network wins the image-recognition challengeAlex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, "ImageNet classification with deep convolutional neural networks". The network is known as AlexNet. Top-5 error 15.3% against 26.2% for the second-best entry.NeurIPS 2012; republished in Communications of the ACM 60(6), 2017. The name and the original 2012 code: Computer History Museum, "CHM Releases AlexNet Source Code" (opens in a new tab), 20 March 2025, released with Google.
Around 1990, translation moves from rules to statisticsPeter F. Brown and colleagues, "A Statistical Approach to Machine Translation"ACL Anthology (opens in a new tab)
Mid 2010s, neural translationBahdanau, Cho and Bengio (2014), arXiv 1409.0473 (opens in a new tab), then Google's Neural Machine Translation system (GNMT), Wu and colleagues (2016), arXiv 1609.08144 (opens in a new tab)
2017, the transformer designVaswani and colleagues, "Attention Is All You Need", first posted 12 June 2017arXiv 1706.03762 (opens in a new tab)
2020, very large language models learn from a few examplesBrown and colleagues, "Language Models are Few-Shot Learners", first posted 28 May 2020. The paper's model is GPT-3 (OpenAI), which the abstract describes as having 175 billion parameters.arXiv 2005.14165 (opens in a new tab)
Late 2022, a chat assistant reaches the publicOpenAI, "Introducing ChatGPT", 30 November 2022OpenAI (opens in a new tab)
2016, a program beats a top Go professional (optional milestone)Google DeepMind's AlphaGo: a 4 to 1 win in Seoul in March 2016. The date and score come from the developer's own page, which carries no date of its own, so read them as the developer's account.Google DeepMind, AlphaGo (opens in a new tab)

Checked: 2 November 2026. DeepMind's page still gives the 4 to 1 score and March 2016 in Seoul.

Chapter 3: how quickly use grew

  • Pew Research Center, 25 June 2025. 34% of U.S. adults said they had ever used ChatGPT, roughly double the share in summer 2023. Survey of 5,123 U.S. adults, 24 February to 2 March 2025, American Trends Panel, recruited by random sampling of residential addresses and weighted to the U.S. adult population. One product, "ever used", U.S. only. Pew, 25 June 2025 (opens in a new tab).
  • Pew Research Center, 17 June 2026 (newer). 44% of U.S. adults now say they have used ChatGPT, up from 34% in 2025, 23% in 2024 and 18% in 2023. About half (49%) say they use AI chatbots of any kind. Survey of 5,119 U.S. adults, 17 to 23 February 2026. Pew notes the 2026 question was worded differently: earlier years asked only people who had heard of ChatGPT, and the 2026 question also asked about other chatbots, so the years don't line up exactly. Pew, "Americans and AI 2026" (opens in a new tab).

Checked: 2 November 2026.

Chapter 3: what assistants cost today

Chapter 3 says what assistants cost today is on this page. The current plans and monthly prices for four assistants, as shown on each provider's own pricing page on 2 November 2026, are under "Is it free?" in the Everyday AI section above.

From the book

Each step on this line fixed a limit of the one before and brought limits of its own. When a new AI tool arrives, ask what problem it solves, then ask what new problem it brings.
Your First Rung, chapter 3

Sources

  1. Neural Machine Translation by Jointly Learning to Align and Translate (opens in a new tab) · Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua, 2014
  2. A Statistical Approach to Machine Translation (opens in a new tab) · Brown, Peter F. and Cocke, John and Della Pietra, Stephen A. and Della Pietra, Vincent J. and Jelinek, Fredrick and Lafferty, John D. and Mercer, Robert L. and Roossin, Paul S., 1990
  3. Language Models are Few-Shot Learners (opens in a new tab) · Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and Winter, Clemens and Hesse, Christopher and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario, 2020
  4. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) (opens in a new tab) · European Parliament and Council of the European Union, 2024
  5. Explained: Neural networks (opens in a new tab) · Hardesty, Larry, 2017
  6. ImageNet classification with deep convolutional neural networks (opens in a new tab) · Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E., 2017
  7. A Neural Network for Machine Translation, at Production Scale (opens in a new tab) · Le, Quoc V. and Schuster, Mike, 2016
  8. Deep learning (opens in a new tab) · LeCun, Yann and Bengio, Yoshua and Hinton, Geoffrey, 2015
  9. A logical calculus of the ideas immanent in nervous activity (opens in a new tab) · McCulloch, Warren S. and Pitts, Walter, 1943
  10. Introducing ChatGPT (opens in a new tab) · OpenAI, 2022
  11. The perceptron: A probabilistic model for information storage and organization in the brain (opens in a new tab) · Rosenblatt, Frank, 1958
  12. Learning representations by back-propagating errors (opens in a new tab) · Rumelhart, David E. and Hinton, Geoffrey E. and Williams, Ronald J., 1986
  13. ImageNet Large Scale Visual Recognition Challenge (opens in a new tab) · Russakovsky, Olga and Deng, Jia and Su, Hao and Krause, Jonathan and Satheesh, Sanjeev and Ma, Sean and Huang, Zhiheng and Karpathy, Andrej and Khosla, Aditya and Bernstein, Michael and Berg, Alexander C. and Fei-Fei, Li, 2015
  14. Deep learning in neural networks: An overview (opens in a new tab) · Schmidhuber, J\"urgen, 2015
  15. Compute Trends Across Three Eras of Machine Learning (opens in a new tab) · Sevilla, Jaime and Heim, Lennart and Ho, Anson and Besiroglu, Tamay and Hobbhahn, Marius and Villalobos, Pablo, 2022
  16. 34% of U.S. adults have used ChatGPT, about double the share in 2023 (opens in a new tab) · Sidoti, Olivia and McClain, Colleen, 2025
  17. Mastering the game of Go with deep neural networks and tree search (opens in a new tab) · Silver, David and Huang, Aja and Maddison, Chris J. and Guez, Arthur and Sifre, Laurent and van den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and Dieleman, Sander and Grewe, Dominik and Nham, John and Kalchbrenner, Nal and Sutskever, Ilya and Lillicrap, Timothy and Leach, Madeleine and Kavukcuoglu, Koray and Graepel, Thore and Hassabis, Demis, 2016
  18. Mastering the game of Go without human knowledge (opens in a new tab) · Silver, David and Schrittwieser, Julian and Simonyan, Karen and Antonoglou, Ioannis and Huang, Aja and Guez, Arthur and Hubert, Thomas and Baker, Lucas and Lai, Matthew and Bolton, Adrian and Chen, Yutian and Lillicrap, Timothy and Hui, Fan and Sifre, Laurent and van den Driessche, George and Graepel, Thore and Hassabis, Demis, 2017
  19. Generative AI and News Report 2025: How People Think About AI's Role in Journalism and Society (opens in a new tab) · Simon, Felix M. and Nielsen, Rasmus Kleis and Fletcher, Richard, 2025
  20. Attention Is All You Need (opens in a new tab) · Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia, 2017