TL;DR
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
At an MIT symposium in 2011, Noam Chomsky questioned whether statistical language models that reproduce observed data without explaining it count as scientific success. Peter Norvig’s response argued that engineering performance can provide evidence about a model, and that probabilistic approaches may help explain language as well as process it. The exchange remains a debate over what models should explain, not a settled finding that statistical learning succeeds or fails.
At a 2011 MIT symposium, linguist Noam Chomsky challenged the scientific value of statistical language models that can approximate observed linguistic data without explaining what that behavior means. In a response published by Peter Norvig, the computer scientist argued that such models can still offer scientific evidence and that probability may be needed to represent the complexity of language.
The exchange took place during the Brains, Minds, and Machines symposium, held as part of MIT’s 150th-anniversary events. According to Norvig’s account, Chomsky was responding to a question from Steven Pinker about the success of probabilistic models trained with statistical methods. Chomsky acknowledged that there had been some successes, alongside many failures, but objected to defining success as the ability to approximate data that had not been analyzed or explained.
Norvig interpreted Chomsky’s position as a distinction between engineering performance and scientific understanding. A system might reproduce patterns in language, Norvig wrote, without showing the principles that produce those patterns. He agreed that engineering success alone is not the goal of science, but argued that a model’s ability to work is evidence—though not proof—that it captures something useful. He also contended that researchers can learn from examining where a model succeeds or fails, and how performance changes with more data, even when its many parameters cannot be understood one by one.
The disagreement also concerned what kinds of models can represent language. Chomsky’s criticism, as Norvig presents it, included the view that a word-probability model cannot explain how people produce sentences, which may involve meanings, syntactic structures and the conversion of those structures into words. Norvig agreed that a simple Markov model cannot account for all language, but argued that a concise, non-probabilistic tree model also has limits. He favored models that could account for words, syntax, meaning and context, with probabilistic methods among the possible tools.
The Debate Over What Models Explain
The exchange addresses a continuing question in language research: is accurate prediction enough, or must a model also provide an account of the processes and principles behind its predictions? The answer affects how researchers judge language technologies. A model can be useful in an engineering task while leaving open whether it reflects how people understand or produce language.
Norvig’s response makes the case that practical results and scientific explanation need not be competing goals. Performance can help test a model, while analysis of its errors and behavior can guide further explanation. Chomsky’s objection, as reported by Norvig, cautions against treating a close fit to data as an explanation in itself. Neither position is established by the exchange as a final verdict: it lays out different standards for what counts as progress.
The stakes extend beyond a particular technique. If language learning is partly statistical, probabilistic models could contribute both to language-processing systems and to theories of human learning. Norvig noted that humans evidently have some innate capacity to acquire language, while arguing that the nature of that capacity—and whether it is language-specific or more general—was not known. The debate therefore leaves open how innate abilities and learning from experience may work together.
Popular right now
Dyson Cyclone V10 Origin Cordless Vacuum
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
De'Longhi Stilosa Compact Manual Espresso Machine with Milk Frother
As an affiliate, we earn on qualifying purchases.
From the MIT Symposium to Norvig’s Response
The original account on Norvig.com says the symposium was held during MIT’s 150th-birthday celebration in 2011 and that a transcript of Chomsky’s remarks had become available. Norvig used the transcript as the basis for a response, setting out questions about the meaning and limits of statistical language models. His essay is an argument in the debate, not a neutral experimental report testing both positions.
Norvig also distinguished a statistical model from a probabilistic model. In his explanation, a statistical model is a mathematical model trained or modified using data; statistical models are often, but not always, probabilistic. A probabilistic model represents a distribution over possible outcomes. That distinction matters because the exchange concerns not just data-driven training in general, but whether probabilistic representations are necessary or useful for language.
Norvig invoked a result associated with Gold’s 1967 theorem and said it concerned theoretical limitations on learning formal languages through logical deduction, not the full task of acquiring natural language. He further cited work by Horning in 1969 on the learnability of probabilistic context-free grammars. These historical points are part of Norvig’s case; the supplied account does not provide a full technical assessment of the results or their relevance to contemporary language learning.
““There is a notion of success … which I think is novel in the history of science. It interprets success as approximating unanalyzed data.””
— Noam Chomsky, as quoted in Peter Norvig’s account of the symposium transcript
What the Exchange Cannot Settle
The material describes a scholarly disagreement, not a controlled comparison that establishes which account of language learning is correct. It does not report new experiments, performance figures or a shared test of whether statistical models explain human language behavior. Norvig’s response presents his interpretation of Chomsky’s remarks and his own arguments; it should not be read as Chomsky endorsing those interpretations.
It also remains unclear from the account how far statistical approaches can explain the full range of language, including meaning, context and discourse, or how human innate capacities relate to statistical learning. Norvig explicitly says that researchers do not know enough about the innate ability to specify how it works or whether it is unique to language. The supplied material does not establish that all statistical models share the limitations Chomsky criticized, or that probabilistic models resolve them.
Questions for Later Research
The symposium exchange itself does not announce a research program, new study or scheduled follow-up. The next steps it points toward are questions rather than a stated milestone: test models against more than their ability to reproduce observed data, examine their successes and failures, and determine whether they can account for language features beyond local word patterns.
Further work would also need to clarify how statistical learning relates to human language acquisition and what role innate capacities play. Until research addresses those questions, the exchange remains useful as a disagreement over standards of explanation—not a resolution of whether statistical methods can provide a scientific account of language.
Key Questions
What did Chomsky criticize about statistical language models?
As quoted and summarized by Norvig, Chomsky objected to treating a model’s ability to approximate unanalyzed data as scientific success. He questioned whether statistical systems explain the underlying processes of language.
What was Peter Norvig’s response?
Norvig argued that engineering success can be evidence, though not proof, that a model captures something useful. He also said that examining a model’s performance and failures can provide insight, and that probabilistic methods may help represent language.
Was the exchange a new scientific study?
No. The source describes remarks at a 2011 symposium and Norvig’s response to them. It does not report a new experiment or a result settling the debate.
Did Norvig argue that statistical models can explain all of language?
No. Norvig acknowledged that a Markov model of word probabilities cannot model all language. He argued that non-probabilistic tree models also have limits and called for approaches that can address features such as syntax, meaning and context.
Source: norvig.com
