An autonomous debating system

an autonomous system that prepares and delivers spoken debate speeches against human debaters

Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Cohen-Karlik, Lena Dankin, Lilach Edelstein, Liat Ein-Dor, Roni Friedman-Melamed, Assaf Gavron, Ariel Gera, Martin Gleize, Shai Gretz, Dan Gutfreund, Alon Halfon, Daniel Hershcovich, Ron Hoory, Yufang Hou, Shay Hummel, Michal Jacovi, Charles Jochim, Yoav Kantor, Yoav Katz, David Konopnicki, Zvi Kons, Lili Kotlerman, Dalia Krieger, Dan Lahav, Tamar Lavee, Ran Levy, Naftali Liberman, Yosi Mass, Amir Menczel, Shachar Mirkin, Guy Moshkowich, Shila Ofek-Koifman, Matan Orbach, Ella Rabinovich, Ruty Rinott, Slava Shechtman, Dafna Sheinwald, Eyal Shnarch, Ilya Shnayderman, Aya Soffer, Artem Spector, Benjamin Sznajder, Assaf Toledo, Orith Toledo-Ronen, Elad Venezian, Ranit Aharonov · Nature · 2021

In one sentence

Project Debater is an autonomous debating system that decomposes competitive debate into modular tasks — argument mining over 400 million news articles, a curated argument knowledge base, speech-to-text rebuttal and rule-based speech construction — and delivers spoken speeches against expert human debaters.

Abstract

Artificial intelligence (AI) is defined as the ability of machines to perform tasks that are usually associated with intelligent beings. Argument and debate are fundamental capabilities of human intelligence, essential for a wide range of human activities, and common to all human societies. The development of computational argumentation technologies is therefore an important emerging discipline in AI research 1 . Here we present Project Debater, an autonomous debating system that can engage in a competitive debate with humans. We provide a complete description of the system's architecture, a thorough and systematic evaluation of its operation across a wide range of debate topics, and a detailed account of the system's performance in its public debut against three expert human debaters. We also highlight the fundamental differences between debating with humans as opposed to challenging humans in game competitions, the latter being the focus of classical 'grand challenges' pursued by the AI research community over the past few decades. We suggest that such challenges lie in the 'comfort zone' of AI, whereas debating with humans lies in a different territory, in which humans still prevail, and for which novel paradigms are required to make substantial progress.

Questions this paper answers

can a computer argue a debate topic well enough to hold its own against a skilled human debater?
how does an autonomous debating system's speech quality compare with that of expert human debaters under crowd evaluation?
how do I find out whether machine-generated debate speeches are judged as good as human ones?
if I want a system that can debate live against a person, is machine-generated argumentation good enough yet?
Project Debater's opening speeches score higher than a multi-document summarization system (Summit), a fine-tuned GPT-2 speech generator, GPT-2-generated arguments, ArgumenText-retrieved arguments and two human-curated argument baselines. The same speeches score lower than human expert debaters' speeches, both gaps being significant at P < 0.05.
Holds for: Crowd annotators rating agreement with 'This speech is a good opening speech for supporting the topic' on a 1-5 scale, 15 annotators per speech, over 78 motions unseen during development; the two human-argument baselines cover only 23 and 77 motions.
Crowd annotators perceived Project Debater as demonstrating 'decent performance' in at least 64% of debate motions. The average score was 4 or higher for 50 of 78 motions, and above the neutral 3 for all but 3 motions.
Holds for: 20 annotators per set reading 3 written speeches of unknown origin on a 1-5 agreement scale; only Project Debater's 2 speeches judged, against simple controls rather than a human debater.
Project Debater's public debut on 11 February 2019 debated debate champion H. Natarajan on whether preschool should be subsidized, a motion never included in the system's training data. The pre-debate audience vote was 79% in favour of subsidizing preschool and 13% against.
Holds for: A single live event; the unbalanced pre-debate vote left Project Debater at most 21% of the audience to win over versus 87% for the human debater, which is why audience voting is not used as the system's evaluation metric.
is a debate speech built by retrieving real arguments better than one written by a text-generating language model?
how do argument-mining opening speeches score against GPT-2 generation, multi-document summarization and retrieval baselines?
how do I pick between generating a persuasive speech with a language model and assembling one from mined arguments?
should I generate my argumentative speeches with a neural language model or mine and arrange existing arguments?
Project Debater's opening speeches score higher than a multi-document summarization system (Summit), a fine-tuned GPT-2 speech generator, GPT-2-generated arguments, ArgumenText-retrieved arguments and two human-curated argument baselines. The same speeches score lower than human expert debaters' speeches, both gaps being significant at P < 0.05.
Holds for: Crowd annotators rating agreement with 'This speech is a good opening speech for supporting the topic' on a 1-5 scale, 15 annotators per speech, over 78 motions unseen during development; the two human-argument baselines cover only 23 and 77 motions.
how often does an automatic debating system actually produce a speech people find acceptable?
on what proportion of motions did crowd annotators rate the autonomous debating system's full-debate performance as decent?
how do I judge how reliable an automatic debate speech generator is across many different topics?
if I run an automatic debater on a new motion, what are the odds it produces something usable?
Crowd annotators perceived Project Debater as demonstrating 'decent performance' in at least 64% of debate motions. The average score was 4 or higher for 50 of 78 motions, and above the neutral 3 for all but 3 motions.
Holds for: 20 annotators per set reading 3 written speeches of unknown origin on a 1-5 agreement scale; only Project Debater's 2 speeches judged, against simple controls rather than a human debater.
are the speeches an automatic debating system gives mostly pre-written filler, or real arguments about the topic?
what is the breakdown of an autonomous debating speech between canned text, mined arguments, a curated argument knowledge base and rebuttal?
how do I tell how much of a machine-written debate speech is boilerplate and how much is topic-specific content?
if I adopt an argument-mining pipeline for speech writing, how much of the output will I have to hand-author as templates?
Less than 18% of Project Debater's generated speech content is conventional canned text; the rest comes from mined arguments (41.8%), the argument knowledge base (27.0%), rebuttal (11.3%) and rebuttal leads (2.4%).
Holds for: Relative distribution of word content across all speeches for the 78 motions of the first evaluation set.
Project Debater's argument mining runs over a corpus of about 400 million newspaper articles from LexisNexis 2011-2018. The corpus is indexed offline by words, Wikipedia concepts, named entities and lexicon words, so claims and evidence for a new motion are retrieved at sentence level online.
Holds for: The motion's topic must be discussed in this news corpus; motions whose topics are sparsely covered are the ones that yield low-scoring speeches.
why does an automatic debating system do badly on some debate topics and well on others?
what distinguishes low-scoring from high-scoring motions in an autonomous debating system, and how do local errors differ from extensive ones?
how do I diagnose why an argument-mining debate system produces a weak speech on a particular motion?
before I run an automatic debater on my topic, what should I check to avoid a bad speech?
Speech quality in Project Debater tracks how much content the system managed to produce. Motions scored 'high' averaged 1,496 words across the three speeches, 'medium' 1,155 words and 'low' 793 words, with the largest gap in mined arguments.
Holds for: Independent evaluation set of 36 motions split by in-house annotator score into high (12, above 3.5), medium (11, between 3 and 3.5) and low (11, below 3); strict precision-oriented confidence thresholds filter content out.
Project Debater's extensive errors, in which one mistake such as off-topic argument-knowledge-base content recurs through a whole speech, occurred only in the lowest-scoring group of motions. Local errors appear in almost all motions, including the highest-scoring ones.
Holds for: Manual in-depth analysis of the independent set of 36 motions, grouped by in-house annotator scores; local errors counted are wrong argument stance, off-topic units and units incoherent without context.
how do you build a machine that debates without training one big model to do it all?
what module decomposition and corpus indexing underlie an autonomous debating system built without an end-to-end model?
how do I put together a debating pipeline from argument mining, a curated argument knowledge base and rebuttal components?
if I want to build a debating agent, do I need an end-to-end neural model or a set of narrow components?
Project Debater engages in competitive debate with no end-to-end model. Four modules do the work: argument mining over an indexed corpus, a manually authored argument knowledge base, speech-to-text-driven rebuttal, and rule-based debate construction with clustering.
Holds for: Architecture as deployed for a debate format of 4-minute opening and second speeches and 2-minute closings, with 15 minutes of preparation after the motion is announced; argument knowledge base texts are hand-authored or manually edited.
Project Debater's argument mining runs over a corpus of about 400 million newspaper articles from LexisNexis 2011-2018. The corpus is indexed offline by words, Wikipedia concepts, named entities and lexicon words, so claims and evidence for a new motion are retrieved at sentence level online.
Holds for: The motion's topic must be discussed in this news corpus; motions whose topics are sparsely covered are the ones that yield low-scoring speeches.
what should I read first about why arguing is a harder test for a computer than winning at chess?
which work framed competitive debate as a grand challenge outside the tractable-game paradigm of classical AI?
where do I start reading about breaking an open-ended human cognitive activity into narrow, solvable AI tasks?
Project Debater's authors argue that competitive debate lies outside the 'comfort zone' of classical AI grand challenges such as checkers, backgammon, chess, Jeopardy!, Go and StarCraft II. Debate has no clear winner, no enumerable moves, no room for strategies humans cannot follow, and no large body of structured training data.
Holds for: A position argued in the paper's discussion rather than an experimental finding, contrasting debate with game competitions as of 2021; no claim is made that humans prevail on all real-world language tasks.
Project Debater is presented as a case study in 'composite AI': breaking a broad human cognitive activity into a collection of tangible narrow tasks. Solutions are built for each narrow task rather than seeking a single end-to-end model.
Holds for: One system built between 2012 and 2019 for one task, competitive debate; the paper offers no evidence about whether end-to-end models could eventually do the task.
Context-dependent claim detection and context-dependent evidence detection were formulated in the course of Project Debater and have since become an active area of research in computational argumentation. Most of the system's underlying capabilities, including argument mining, are available as cloud services for academic research on request.
Holds for: As stated by the authors in 2021; datasets built during development are released, the full system code is not, and academic access is by request through IBM's early-access programme.
why is arguing a debate topic harder for a computer than winning at chess or Go?
what properties of competitive debate place it outside the comfort zone of game-playing AI grand challenges?
if my team beat humans at a board game, why would a debating task be a different kind of problem for us?
Project Debater's authors argue that competitive debate lies outside the 'comfort zone' of classical AI grand challenges such as checkers, backgammon, chess, Jeopardy!, Go and StarCraft II. Debate has no clear winner, no enumerable moves, no room for strategies humans cannot follow, and no large body of structured training data.
Holds for: A position argued in the paper's discussion rather than an experimental finding, contrasting debate with game competitions as of 2021; no claim is made that humans prevail on all real-world language tasks.
can researchers get access to the argument-mining components behind IBM's debating system?
which computational argumentation tasks, such as context-dependent claim and evidence detection, were formulated during the debating-system effort?
how do I get argument mining and evidence detection services for my own argumentation research?
can I use IBM's debating-system capabilities as cloud services for an academic project?
Context-dependent claim detection and context-dependent evidence detection were formulated in the course of Project Debater and have since become an active area of research in computational argumentation. Most of the system's underlying capabilities, including argument mining, are available as cloud services for academic research on request.
Holds for: As stated by the authors in 2021; datasets built during development are released, the full system code is not, and academic access is by request through IBM's early-access programme.
what topic did IBM's debating computer argue in its live 2019 event against a champion debater?
what was the motion and the pre-debate audience vote in the autonomous debating system's February 2019 public debut?
was the live debut motion something the debating system had already been trained on, or genuinely new to it?
Project Debater's public debut on 11 February 2019 debated debate champion H. Natarajan on whether preschool should be subsidized, a motion never included in the system's training data. The pre-debate audience vote was 79% in favour of subsidizing preschool and 13% against.
Holds for: A single live event; the unbalanced pre-debate vote left Project Debater at most 21% of the audience to win over versus 87% for the human debater, which is why audience voting is not used as the system's evaluation metric.

Claims and scope

Common misreadings

Terminology in this paper

motion
The resolution being debated, announced at the start of a debate; in IBM's released cloud services the same thing is called a 'topic'.
argument knowledge base (AKB)
A manually authored or manually edited collection of principled arguments, counter-arguments, analogies, quotes and framings grouped into thematic classes, matched to a new debate resolution by a classifier so that content reusable across debates can be inserted into a speech.
leads
Claims that a debating system predicts its opponent might make, compiled in advance from argument mining, an argument knowledge base and iDebate, and then detected in the opponent's transcribed speech to trigger a prepared rebuttal.
context-dependent claim detection
The task of finding, in a large text corpus, concise statements that take a clear stance towards a given debate resolution, as opposed to detecting claims independently of any topic.
context-dependent evidence detection
The task of finding single sentences that support or contest a given debate resolution by indicating whether a relevant claim is true, rather than merely expressing a belief or claim.
composite AI
Tasks tied to broad human cognitive activities that require several skills applied simultaneously, contrasted with 'narrow AI' tasks that are individually well defined and amenable to end-to-end solutions.
extensive error
A speech-generation error in which the same type of mistake recurs throughout a whole speech and propagates across multiple content units, as opposed to a local error affecting one unit.

How to cite

@article{DBLP:journals/nature/SlonimBABBBCCDE21,author       = {Noam Slonim and
                  Yonatan Bilu and
                  Carlos Alzate and
                  Roy Bar{-}Haim and
                  Ben Bogin and
                  Francesca Bonin and
                  Leshem Choshen and
                  Edo Cohen{-}Karlik and
                  Lena Dankin and
                  Lilach Edelstein and
                  Liat Ein{-}Dor and
                  Roni Friedman{-}Melamed and
                  Assaf Gavron and
                  Ariel Gera and
                  Martin Gleize and
                  Shai Gretz and
                  Dan Gutfreund and
                  Alon Halfon and
                  Daniel Hershcovich and
                  Ron Hoory and
                  Yufang Hou and
                  Shay Hummel and
                  Michal Jacovi and
                  Charles Jochim and
                  Yoav Kantor and
                  Yoav Katz and
                  David Konopnicki and
                  Zvi Kons and
                  Lili Kotlerman and
                  Dalia Krieger and
                  Dan Lahav and
                  Tamar Lavee and
                  Ran Levy and
                  Naftali Liberman and
                  Yosi Mass and
                  Amir Menczel and
                  Shachar Mirkin and
                  Guy Moshkowich and
                  Shila Ofek{-}Koifman and
                  Matan Orbach and
                  Ella Rabinovich and
                  Ruty Rinott and
                  Slava Shechtman and
                  Dafna Sheinwald and
                  Eyal Shnarch and
                  Ilya Shnayderman and
                  Aya Soffer and
                  Artem Spector and
                  Benjamin Sznajder and
                  Assaf Toledo and
                  Orith Toledo{-}Ronen and
                  Elad Venezian and
                  Ranit Aharonov},
  title        = {An autonomous debating system},
  journal      = {Nat.},
  volume       = {591},
  number       = {7850},
  pages        = {379--384},
  year         = {2021},
  url          = {https://doi.org/10.1038/s41586-021-03215-w},
  doi          = {10.1038/S41586-021-03215-W},
  timestamp    = {Sat, 30 Sep 2023 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/nature/SlonimBABBBCCDE21.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.