AI Timeline, 1676–2024: From the Chain Rule to ChatGPT

The chain rule — how to trace influence through nested calculations

While building his calculus, Gottfried Wilhelm Leibniz arrived at a rule for the case where one quantity affects a second, which in turn affects a third: the overall rate of change is the product of the rates at each stage. This is what is now called the chain rule. A neural network is a stack of simple calculations nested many layers deep, so how much each connection is to blame for an error at the output can be found by applying this rule backwards from the output. That procedure came to be called backpropagation some 290 years later (see 1970 and 1986).

• Until it was used to train neural networks ~290 years

People and institutions: Gottfried Wilhelm Leibniz

NoteThe date 1676 follows Schmidhuber's survey (2022), which cites J. M. Child's English translation (1920). Leibniz published his calculus in 1684, and the chain rule took its textbook form later still. The manuscript itself has not been consulted.

• Annotated History of Modern AI and Deep Learning — arXiv:2212.11279 (2022), J. Schmidhuber
• The Early Mathematical Manuscripts of Leibniz (translated by J. M. Child) — Chicago: Open Court (1920); HathiTrust catalog record

Reasoning backwards from results to causes — Bayes's essay

After the clergyman Thomas Bayes died, his friend Richard Price sent a manuscript he had left to the Royal Society. It tackles this problem: given how many times an event has and has not happened, how probable is it that the event is inherently likely? Instead of calculating effects from causes, it treated reasoning from effects back to causes as a matter of numbers for the first time. The idea of revising an estimate each time new data arrives became one of the foundations of machine learning as a whole.

People and institutions: Thomas Bayes / Richard Price

NoteThe general form now taught as Bayes' theorem was worked out independently by Pierre-Simon Laplace from 1774 onward.

• An Essay towards solving a Problem in the Doctrine of Chances — Phil. Trans. R. Soc. Lond. 53, 370–418 (1763), T. Bayes, communicated by R. Price

Least squares — minimise the sum of squared misses

In an appendix to a book on finding the orbits of comets, Adrien-Marie Legendre set out a method: square the gap between each observation and what the formula predicts, add the squares up, and choose the formula's coefficients so that the total is as small as possible. Observations always contain error, so no formula passes through every point. The idea is to boil down how far off you are to a single number and make it as small as you can. That pattern — turn the miss into a number, then shrink it — is exactly how machine-learning models are trained today. Fitting a straight line this way has the same form as a neural network with a single layer.

People and institutions: Adrien-Marie Legendre / Carl Friedrich Gauss

NoteIn his 1809 book Carl Friedrich Gauss stated that he had been using the method since 1795, and a priority dispute followed. Legendre published first.

• Nouvelles méthodes pour la détermination des orbites des comètes — Paris: Firmin Didot (1805); digitized copy, Internet Archive
• Gauss and the Invention of Least Squares — Ann. Statist. 9(3), 465–474 (1981), S. M. Stigler

Gradient descent — take small steps down the steepest slope

To solve the simultaneous equations that arise in computing the orbits of heavenly bodies, Augustin-Louis Cauchy published a note just three pages long. Replace the equations with a single quantity that is smaller the closer you are to the answer; from wherever you stand, move a little in the direction in which it falls fastest; repeat, and you approach the bottom of the valley. Cauchy himself did not prove that the procedure always arrives at the answer. Even so, treating the error as a slope and walking down it is exactly how neural networks are trained today.

• Length of the original note 3 pages

People and institutions: Augustin-Louis Cauchy

NoteThe original is Comptes Rendus de l'Académie des Sciences 25, 536–538 (1847). The source cited here is Lemaréchal's study of it.

• Cauchy and the gradient method — Documenta Mathematica, Extra Volume ISMP, 251–254 (2012), C. Lemaréchal

Counting 20,000 letters of Pushkin — predicting what comes next from what came before

The Russian mathematician Andrey Markov took the first 20,000 letters of Pushkin's verse novel Eugene Onegin, sorted them into vowels and consonants, and counted how sharply the chance of a vowel depends on whether the previous letter was a vowel or a consonant. It was a worked example of what is now called a Markov chain: a mathematics of sequences in which each event is not independent but depends on the state just before it. It was also the first time a text was treated as a problem of predicting, as a probability, what comes next from what came before. Today's language models are descendants that widen the view from one preceding letter to thousands of words or more.

• Letters counted 20,000

People and institutions: Andrey Markov (Андрей Марков)

NoteThe source is an English translation of the 1913 paper.

• An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in Chains — Science in Context 19(4), 591–600 (2006), A. A. Markov; English translation of the 1913 paper

A row of tiny magnets pointing up or down — an equation that later became a model of memory

To explain why iron becomes magnetic, Ernst Ising solved a model in which tiny magnets, each able to point only up or down, sit in a row and try to line up with their neighbours. The result was a disappointment — a single row never becomes a magnet — and Ising left the subject. Yet the model became a standard tool for studying how simple units that influence one another behave as a whole. Fifty-seven years later it was shown that if the units are read as nerve cells, it becomes a model of memory (see 1982).

• Until Hopfield's network 57 years

People and institutions: Ernst Ising / Wilhelm Lenz

NoteThe model itself was proposed in 1920 by Ising's supervisor Wilhelm Lenz, and physicists also call it the Lenz–Ising model.

• Beitrag zur Theorie des Ferromagnetismus — Z. Physik 31, 253–258 (1925), E. Ising
• History of the Lenz-Ising Model — Rev. Mod. Phys. 39(4), 883–893 (1967), S. G. Brush

What does it mean to compute? — defining one machine that can compute anything

Alan Turing imagined a simple abstract machine that does nothing but write and erase symbols on a paper tape, and argued that anything a person can calculate by following a procedure can be calculated by such a machine. He went on to show that a single universal machine suffices: given a description of any other machine, it can imitate it. This is the idea behind today's computers, which become anything you like when the program is changed. The premise of artificial intelligence — that if intelligence is some kind of computation, a machine can in principle do it — rests on this paper.

People and institutions: Alan Turing

NoteThe paper was received and read in 1936; the volume it appears in was published in 1937.

• On Computable Numbers, with an Application to the Entscheidungsproblem — Proc. London Math. Soc. s2-42(1), 230–265 (1937), A. M. Turing

Writing the nerve cell as a component of logic

The neurophysiologist Warren McCulloch and the young logician Walter Pitts described the nerve cell as a simple element: it fires if the sum of the signals coming in exceeds a threshold, and stays silent if not. They then showed that by combining such elements you can realise any logic that can be written with and, or and not. It was the first paper to let the workings of the brain and the calculations of logic be discussed in the same language. But the wiring in their model is set by hand; there is as yet no mechanism for learning from experience.

People and institutions: Warren McCulloch / Walter Pitts

• A logical calculus of the ideas immanent in nervous activity — Bull. Math. Biophys. 5(4), 115–133 (1943), W. S. McCulloch and W. Pitts

Measuring information with numbers — and producing English-like text from probabilities alone

Claude Shannon of Bell Laboratories defined the amount of information in a message by a formula in probabilities: how hard it is to predict. The paper also contains an experiment. He tabulated how likely each English letter or word is to follow the few before it, then strung symbols together as if drawing lots according to those odds; the wider the window, the more English-like the output became. In 1951 he measured how predictable English is by having people guess the next letter. The quantity that today's language models are trained to make small belongs to the family he defined here.

People and institutions: Claude Shannon / Bell Telephone Laboratories

• A Mathematical Theory of Communication — Bell Syst. Tech. J. 27(3), 379–423 (1948), C. E. Shannon
• Prediction and Entropy of Printed English — Bell Syst. Tech. J. 30(1), 50–64 (1951), C. E. Shannon

Learning is a change in the strength of connections — Hebb's hypothesis

In his book The Organization of Behavior, the Canadian psychologist Donald Hebb proposed that when one nerve cell repeatedly makes another fire, the connection between the two grows stronger. Memory and learning, on this view, are stored not in the cells themselves but in the strength of the links between them. At the time it was a guess with no means of testing it; something close to it was later found in real brains. The starting point for artificial neural networks — that to learn is to change the numbers standing for connection strengths — is here.

People and institutions: Donald O. Hebb

NoteThe source is a reissue of the 1949 original (New York: Wiley).

• The Organization of Behavior: A Neuropsychological Theory — New York: Wiley (1949); reissued by Psychology Press (2005), D. O. Hebb

Replacing 'Can machines think?' with a question that can be tested

Sidestepping arguments about what the word think means, Turing proposed a test: if, in an exchange carried on in writing alone, a machine cannot be told apart from a person, let us grant that it thinks. Most of the paper is spent answering, one by one, the objections that a machine could never do this. Near the end he also suggests that rather than building an adult mind outright, one might build a machine that corresponds to a child's mind and educate it. It foreshadowed what is now the mainstream approach: not writing rules in, but making a machine clever by having it learn.

People and institutions: Alan Turing

• Computing Machinery and Intelligence — Mind LIX(236), 433–460 (1950), A. M. Turing

Even with noisy measurements, you arrive if you keep shortening your step

The statisticians Herbert Robbins and Sutton Monro proved that even when every measurement is contaminated by error, you can correct your estimate a little after each one, make the corrections smaller as you go, and the estimate will close in on the right value. Training today's AI does not look at an enormous dataset all at once; it repeatedly pulls out a small sample and adjusts a little. Because only a sample is seen, each step points in a slightly wrong direction, yet overall the process heads the right way. This paper supplied the mathematical backing for that method, known as stochastic gradient descent.

People and institutions: Herbert Robbins / Sutton Monro

• A Stochastic Approximation Method — Ann. Math. Statist. 22(3), 400–407 (1951), H. Robbins and S. Monro

The perceptron — a machine that adjusts its connections every time it is wrong

Frank Rosenblatt, a psychologist at the Cornell Aeronautical Laboratory, combined the elements of McCulloch and Pitts with Hebb's idea of changing connection strengths to produce a model that learns from experience. Multiply the inputs by weights and add them up; if the answer is wrong, nudge the weights. Repeat, and the system comes to tell shapes apart without anyone giving it rules. But only the one layer of weights feeding the output could be learned; nobody knew how to adjust the layers in between. That weakness was attacked hard about ten years later (see 1969).

People and institutions: Frank Rosenblatt / Cornell Aeronautical Laboratory

• The perceptron: A probabilistic model for information storage and organization in the brain — Psychol. Rev. 65(6), 386–408 (1958), F. Rosenblatt

The layers in between can be learned by going downhill too — Shun'ichi Amari's theory

Shun'ichi Amari, then at Kyushu University, set out a general theory for training a pattern-classifying machine by stochastic gradient descent — adjusting the weights a little, each time one piece of data is seen, in the direction that reduces the error. The method also applies when there are layers between input and output. In principle, then, the perceptron's supposed weakness — how to adjust the layers in between — already had an answer. But the computers of the day could not run large experiments, the work stayed almost unknown in the English-speaking world, and the same idea was rediscovered about twenty years later (see 1986).

People and institutions: Shun'ichi Amari (甘利俊一)

NoteOn learning in several layers, Alexey Ivakhnenko and colleagues in the Soviet Union (now Ukraine) had from 1965 published a different method (GMDH) that builds up layers one at a time. The view that this paper is the first to train multiple layers by stochastic gradient descent rests mainly on Schmidhuber's survey.

• A Theory of Adaptive Pattern Classifiers — IEEE Trans. Electron. Comput. EC-16(3), 299–307 (1967), S. Amari
• Polynomial Theory of Complex Systems — IEEE Trans. Syst. Man Cybern. SMC-1(4), 364–378 (1971), A. G. Ivakhnenko

Perceptrons — a proof that some problems cannot be solved with one layer

Marvin Minsky and Seymour Papert showed mathematically that a perceptron with only one learnable layer cannot, in principle, solve certain problems. The famous example is a simple decision: output 1 when exactly one of two inputs is 1. It was understood that stacking layers would solve it, but no practical way of training stacked layers was widely known. The book cooled expectations for neural networks and is regarded as one reason research interest and money shifted to approaches based on symbols and rules.

People and institutions: Marvin Minsky / Seymour Papert

NoteThe source is the 2017 reissue of the 1969 original (MIT Press).

• Perceptrons: An Introduction to Computational Geometry — Cambridge, MA: MIT Press (1969; reissued 2017), M. Minsky and S. A. Papert

A method for studying rounding error turned out to be backpropagation itself

A Finnish student, Seppo Linnainmaa, wrote a master's thesis on how to find out how much the small rounding errors made along the way in a long calculation affect the final answer. Break the calculation into small steps, apply the chain rule from the last step back to the first, and the influence of every step falls out of a single backward pass. Neural networks are nowhere mentioned. Yet this is the same calculation that is now called backpropagation. Born as a tool for numerical computing, it was put to work as a way of learning more than a decade later.

People and institutions: Seppo Linnainmaa

NoteThe 1970 work is a master's thesis at the University of Helsinki, in Finnish; its content was published in English in 1976. Applying the same calculation to neural networks was proposed by Paul Werbos in his 1974 doctoral thesis and published in 1982. There are several lines of priority, which Griewank's essay sorts out.

• Taylor expansion of the accumulated rounding error — BIT 16(2), 146–160 (1976), S. Linnainmaa
• Who invented the reverse mode of differentiation? — Documenta Mathematica, Extra Volume ISMP, 389–400 (2012), A. Griewank

The neocognitron — a layered network that recognises a pattern even when it shifts

Kunihiko Fukushima of NHK's Broadcasting Science Research Laboratories built a network that alternates two kinds of layer, modelled on the visual cells Hubel and Wiesel had found. One kind picks up patterns within a small patch; the other absorbs small shifts in position. Stack several of these and a character is recognised as the same character even if it is displaced or distorted. Reusing the same small pattern detector across the whole picture is the prototype of the convolutional neural networks used for image recognition today. What it lacked was a way to train the whole thing together (see 1989).

People and institutions: Kunihiko Fukushima (福島邦彦) / NHK Broadcasting Science Research Laboratories

NoteA Japanese-language paper appeared in 1979 in the Transactions of the IECE of Japan; the English paper followed in 1980.

• Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position — Biol. Cybern. 36(4), 193–202 (1980), K. Fukushima

A network that holds memories as valleys — a physicist's way of seeing it

The physicist John Hopfield showed that for a collection of simple interconnected units one can define an energy of the same form as in the model of magnets (see 1925). Set the connection strengths so that the patterns to be remembered sit at the bottoms of valleys in that energy, and when a damaged or noisy input is supplied, the state rolls downhill and settles in the nearest valley. That is what memory does: recall the whole from a partial cue. The paper brought statistical physicists, and their tools, flooding into neural-network research.

People and institutions: John Hopfield

NoteThat a network of this type acquires memories through learning had been discussed by Shun'ichi Amari in 1972.

• Neural networks and physical systems with emergent collective computational abilities — Proc. Natl. Acad. Sci. USA 79(8), 2554–2558 (1982), J. J. Hopfield
• Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements — IEEE Trans. Comput. C-21(11), 1197–1206 (1972), S. Amari

The Boltzmann machine — teaching hidden units the features of the data

David Ackley, Geoffrey Hinton and Terrence Sejnowski changed the units of Hopfield's network into ones that behave probabilistically, and derived a learning rule using the formulas of statistical mechanics. The crucial move was to include hidden units, connected directly to neither input nor output. As learning proceeds, the hidden units come to represent regularities lurking in the data without being told what to look for. It was far too slow to be practical at the time, but the idea of letting hidden layers find features for themselves led to the relaunch of deep learning in 2006.

People and institutions: David Ackley / Geoffrey Hinton / Terrence Sejnowski

• A Learning Algorithm for Boltzmann Machines — Cogn. Sci. 9(1), 147–169 (1985), D. H. Ackley, G. E. Hinton and T. J. Sejnowski

Backpropagation spreads — the layers in between start building features of their own

In a four-page paper, David Rumelhart, Geoffrey Hinton and Ronald Williams showed how to send the error at the output back towards the input by the chain rule, adjusting every weight including those of the layers in between. What caught attention was the experimental result: the units in the middle layers came to represent features useful for the task without anyone telling them to. Problems said to be unsolvable with one layer were solved. The paper revived interest in neural networks, but a barrier remained for the next twenty years: learning went badly once the layers got deep.

People and institutions: David Rumelhart / Geoffrey Hinton / Ronald Williams

NoteThe procedure itself was not a new discovery. Reverse-mode differentiation is due to Linnainmaa (1970), its application to neural networks to Werbos (doctoral thesis, 1974), and a theory of multilayer learning to Amari (1967). This paper's contribution was to show by experiment that the layers in between build useful representations of their own, and to make that widely known.

• Learning representations by back-propagating errors — Nature 323(6088), 533–536 (1986), D. E. Rumelhart, G. E. Hinton and R. J. Williams
• Backpropagation through time: what it does and how to do it — Proc. IEEE 78(10), 1550–1560 (1990), P. J. Werbos

Learning without waiting for the outcome — using the gap between one prediction and the next

Richard Sutton showed as a matter of theory that learning need not wait for the final outcome: it can proceed from the disagreement between a prediction at one moment and the prediction one step later. Who wins a game is unknown until the end, but if the outlook changes after one move, that change is material for correcting the earlier outlook. It generalised what Samuel's checkers program (1959) had done by rule of thumb. It became the central method of reinforcement learning — learning by trial and error from rewards — and was later used in programs that play Go (see 2016).

People and institutions: Richard Sutton

NoteQ-learning, which also learns how to choose actions, was introduced by Christopher Watkins in his 1989 doctoral thesis and published with a convergence proof in 1992.

• Learning to predict by the methods of temporal differences — Mach. Learn. 3(1), 9–44 (1988), R. S. Sutton
• Q-learning — Mach. Learn. 8(3–4), 279–292 (1992), C. J. C. H. Watkins and P. Dayan

One layer in between is enough to get as close as you like to any function

In 1989 George Cybenko, Kurt Hornik and colleagues, and Ken-ichi Funahashi of Toyohashi University of Technology each independently proved a theorem to the same effect: put one layer between input and output, give it enough units, and the network can imitate any continuous function as accurately as you wish. It is a guarantee that, in principle, nothing is beyond what a neural network can represent. But the theorem says nothing about how many units are needed, or whether learning will ever find the answer. Being able to represent something and being able to find it are different problems.

People and institutions: George Cybenko / Kurt Hornik / Maxwell Stinchcombe / Halbert White / Ken-ichi Funahashi (船橋賢一)

• Approximation by superpositions of a sigmoidal function — Math. Control Signals Syst. 2(4), 303–314 (1989), G. Cybenko
• Multilayer feedforward networks are universal approximators — Neural Netw. 2(5), 359–366 (1989), K. Hornik, M. Stinchcombe and H. White
• On the approximate realization of continuous mappings by neural networks — Neural Netw. 2(3), 183–192 (1989), K. Funahashi

Support-vector machines — a method guaranteed to find the best answer becomes the mainstream

Corinna Cortes and Vladimir Vapnik set out a method for drawing the boundary between two kinds of data so that its distance from the nearest points on either side is as large as possible. The problem can be written in a form with only one valley, so the calculation is certain to reach the best answer. Training a neural network involves many valleys, with no guarantee of where it will settle. Because the theory was clean and, with the amounts of data then available, the results were good, the leading role in machine learning passed to this approach from the late 1990s through the 2000s, and neural networks entered a second winter.

People and institutions: Corinna Cortes / Vladimir Vapnik / AT&T Bell Laboratories

• Support-vector networks — Mach. Learn. 20(3), 273–297 (1995), C. Cortes and V. Vapnik

LSTM — carrying a clue from the distant past without losing it

Networks that handle ordered data such as text or speech had a problem: as the error is passed back through time the signal shrinks rapidly, so clues far apart cannot be learned. Sepp Hochreiter and Jürgen Schmidhuber avoided this by building into each unit a cell that can hold information unchanged, with gates that regulate what goes in and out. What to remember and what to forget is itself learned. It drew limited attention when published, but in the 2010s it became the workhorse of speech recognition and machine translation.

People and institutions: Sepp Hochreiter / Jürgen Schmidhuber

• Long Short-Term Memory — Neural Comput. 9(8), 1735–1780 (1997), S. Hochreiter and J. Schmidhuber

Representing words as lists of numbers and having a network predict the next word

Until then, language models worked by counting how often each sequence of words had been seen. That approach is helpless with a sequence it has never met. Yoshua Bengio and colleagues represented each word as a list of a few dozen numbers, and trained that representation together with a network that computes the probability of the word that comes next. Words used in similar ways end up with similar lists, so the model can reason by analogy from similar sentences to ones it has never seen. The skeleton of today's language models — turn words into lists of numbers, and learn by predicting the next word — was all in place here.

People and institutions: Yoshua Bengio / Réjean Ducharme / Pascal Vincent / Christian Jauvin

• A Neural Probabilistic Language Model — J. Mach. Learn. Res. 3, 1137–1155 (2003), Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin

Preparing a deep network one layer at a time before training it

It was well known that a network with many stacked layers does not train well by backpropagation alone. Geoffrey Hinton and colleagues set out a two-stage approach: first prepare the layers one at a time, letting each grasp the features of the data without being given right answers (using a simplified version of the Boltzmann machine), and only then finish the whole network off with backpropagation. This showed that deep networks could reach genuinely useful performance, and research got moving again under the name deep learning. The preparation step itself was replaced by other tricks within a few years.

People and institutions: Geoffrey Hinton / Ruslan Salakhutdinov / Simon Osindero / Yee-Whye Teh

• Reducing the Dimensionality of Data with Neural Networks — Science 313(5786), 504–507 (2006), G. E. Hinton and R. R. Salakhutdinov
• A Fast Learning Algorithm for Deep Belief Nets — Neural Comput. 18(7), 1527–1554 (2006), G. E. Hinton, S. Osindero and Y.-W. Teh

Attention — choosing where to look in the source for each word of the translation

Neural machine translation at the time squeezed the source sentence into a single list of numbers of fixed length and produced the translation from that. With long sentences not everything fits, and quality fell. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio added a mechanism that, each time a word of the translation is produced, computes on the spot how heavily to weigh each word of the source. Where to look is not taught by people but settled by learning. Quality held up much better on long sentences. Three years later this mechanism became the main component of the network itself (see 2017).

People and institutions: Dzmitry Bahdanau / Kyunghyun Cho / Yoshua Bengio

• Neural Machine Translation by Jointly Learning to Align and Translate — arXiv:1409.0473 (2014), D. Bahdanau, K. Cho and Y. Bengio

Turn an image gradually into static and learn to run it backwards — the prototype of diffusion models

Taking their cue from how physics treats the process of things diffusing, Jascha Sohl-Dickstein and colleagues proposed a new way to build a generative model. First add a tiny amount of noise to an image over and over until nothing but static remains. Then have a network learn to retrace a single step of that process in reverse — to remove just a little of the noise. Once trained, it can make a new image by starting from static and stepping back one stage at a time. On publication it trailed other methods in image quality and attracted little notice, but it became the foundation of image generation from 2020 onward (see 2022).

People and institutions: Jascha Sohl-Dickstein / Eric Weiss / Niru Maheswaranathan / Surya Ganguli / Stanford University

• Deep Unsupervised Learning using Nonequilibrium Thermodynamics — arXiv:1503.03585 (2015), J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan and S. Ganguli

ResNet — shortcuts that skip layers take networks past a hundred layers

Deeper networks ought to perform better, but in practice it was known that beyond a certain depth performance got worse even on the training data. Kaiming He and colleagues at Microsoft Research Asia added shortcuts that carry the input straight past a few layers. Each layer then no longer has to construct an answer from scratch; it need only learn a correction to what came before. A 152-layer network trained stably, and won the ImageNet competition with an error rate of 3.57 percent. These shortcuts are found in almost every large network today.

• Layers 152
• Top-5 error rate 3.57%

People and institutions: Kaiming He (何恺明) / Xiangyu Zhang / Shaoqing Ren / Jian Sun / Microsoft Research Asia

NoteThe paper was posted in December 2015 and presented at a conference in 2016.

• Deep Residual Learning for Image Recognition — 2016 IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (2016), K. He, X. Zhang, S. Ren and J. Sun

The Transformer — stop reading in order and build with attention alone

Until then, networks that handled text processed words one at a time from the front. Because they could only move forward in sequence, it was hard to speed things up by running calculations side by side, and relationships between distant words were hard to capture. Ashish Vaswani and seven colleagues at Google removed the sequential part altogether and built the network from attention alone, computing all at once how strongly every word in a sentence relates to every other. It set a new record in translation quality and trained far faster. This structure, well suited to running in parallel on GPUs, made the later growth in scale possible.

People and institutions: Ashish Vaswani / Noam Shazeer / Niki Parmar / Jakob Uszkoreit / Llion Jones / Aidan Gomez / Łukasz Kaiser / Illia Polosukhin / Google

• Attention Is All You Need — arXiv:1706.03762 (2017), A. Vaswani et al.

Lay a foundation on vast amounts of text first, then adapt to the task

In 2018 the standard way of handling language changed. In June OpenAI announced a method (GPT) in which a Transformer first reads a large body of text with the task of predicting the next word and is then finished off for a particular task. In October Jacob Devlin and colleagues at Google, using a method (BERT) that hides words here and there in a sentence and has the model guess them from both sides, set new records on eleven tasks including question answering. What the two share is that general knowledge of language is learned first, from plain text to which nobody has attached right answers. The era of building from scratch for each task ended here.

People and institutions: Jacob Devlin / Ming-Wei Chang / Kenton Lee / Kristina Toutanova / Google / OpenAI

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — arXiv:1810.04805 (2018), J. Devlin, M.-W. Chang, K. Lee and K. Toutanova
• Improving Language Understanding by Generative Pre-Training — OpenAI technical report (2018), A. Radford, K. Narasimhan, T. Salimans and I. Sutskever