Will it Blend? Blending Weak and Strong Labeled Data in a Neural Network for Argumentation Mining

Q: how many hand-labelled sentences can noisy automatic labels replace when training an evidence classifier?

With Webis-Debate-16 as weak labeled data, blending lets 1,000 strong labeled instances reach accuracy comparable to 2,500 strong instances alone, and 2,000 strong instances plus weak data match 3,000 strong instances alone. Holds for: Micro-averaged accuracy on the 35-topic test split of the released 5,785-sentence evidence dataset; the equivalence is smaller for the 'that + topic concept' weak data.

Q: can noisy labelled sentences just be dumped into the training set together with the clean ones?

Unifying weak and strong labeled data into one training set with no blending schedule does not help. Accuracy falls well below strong-data-only for the 'that + topic concept' weak data, and neither helps nor harms for Webis-Debate-16. Holds for: Evidence detection with the 2 weak-data sources of the paper, at full strong-data size, shown as the single square points on the right border of each plot. Training on the weak labeled data alone yields accuracy much lower than training on the full strong labeled data. The blending gain is therefore not simply the effect of adding more labeled examples. Holds for: Topic-dependent evidence detection, both weak-data sources, shown as the single triangle points on the Y-axis of each plot.

Q: how quickly should the share of noisy labelled examples be reduced from one training pass to the next?

A blend factor larger than 0.05 is typically ineffective for decaying the weak-data fraction across epochs. A single initialization epoch on weak data alone suffices, and more than 1 gives slightly worse accuracy. Holds for: Blend factors tried were 0, 0.05 and 0.2; stopping point fixed so the last 4 blending epochs were at least 95% strong labeled data. Initialization-epoch count explored on a different dataset.

Q: is there a labelled collection of sentences marked as supporting or contesting a debate topic?

A manually annotated dataset of 5,785 sentences for topic-dependent evidence detection is released, split by topic into 4,066 training sentences over 83 topics and 1,719 test sentences over 35 topics. About 40% of each split is positive. Holds for: Wikipedia sentences over 118 debate topics, 10 crowd annotators each, majority label with ties as non-evidence, Fleiss' kappa 0.45. Topic concepts replaced by a common token. BlendNet's evidence-detection setup is corpus-wide: evidence is detected as directly supporting or contesting a topic, with no intermediate claim. No small set of relevant articles is pre-selected, unlike earlier evidence-detection work. Holds for: English Wikipedia as the corpus and 118 debate topics with clearly identifiable concepts; a bound on the released data, not a claim about other languages or argument types.

Q: can a simple word pattern find sentences that argue for or against a topic without any human labelling?

Sentences matching the query 'that + topic concept' are positive evidence 52% of the time, against a prior close to 40% over the whole training set. About 25% of retrieved Wikipedia sentences containing a topic concept match the query. Holds for: Wikipedia sentences for the 118 topics of the released evidence dataset; the query's own accuracy on the test set is only 17%.

Q: when adding noisy labelled text, does having more of it matter more than how well it matches the task?

Weak-data quality matters more than weak-data volume for blending: the smaller Webis-Debate-16 debate corpus helps more than the far larger 'that + topic concept' Wikipedia set. Holds for: 2 weak-data sources on topic-dependent evidence detection only, so this is a single observed contrast rather than a controlled study of size versus quality.

Q: what work proposed using noisy labelled data throughout training instead of only for pre-training?

BlendNet proposes blending weak and strong labeled data inside a single network throughout training, as a general recipe for language-understanding tasks where high-quality annotation is scarce. Earlier work used weak data only to pre-train, or trained separate networks per data type. Holds for: As of ACL 2018; the only task evaluated is topic-dependent evidence detection with a bi-LSTM, and the paper does not claim the schedule is optimal or the only option.

Eyal Shnarch; Carlos Alzate; Lena Dankin; Martin Gleize; Yufang Hou; Leshem Choshen; Ranit Aharonov; Noam Slonim

doi:10.18653/V1/P18-2095

Will it Blend? Blending Weak and Strong Labeled Data in a Neural Network for Argumentation Mining

a bi-LSTM trained on a decaying blend of noisy weak labels and scarce human labels

Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, Noam Slonim · ACL 2018 · 2018

In one sentence

BlendNet trains a neural network on an exponentially decaying mixture of abundant weak labeled data and scarce strong labeled data throughout training, rather than using weak data only for pre-training, and improves topic-dependent evidence detection most when strong data is scarce.

Abstract

The process of obtaining high quality labeled data for natural language understanding tasks is often slow, error-prone, complicated and expensive. With the vast usage of neural networks, this issue becomes more notorious since these networks require a large amount of labeled data to produce satisfactory results. We propose a methodology to blend high quality but scarce strong labeled data with noisy but abundant weak labeled data during the training of neural networks. Experiments in the context of topic-dependent evidence detection with two forms of weak labeled data show the advantages of the blending scheme. In addition, we provide a manually annotated data set for the task of topic-dependent evidence detection. We believe that blending weak and strong labeled data is a general notion that may be applicable to many language understanding tasks, and can especially assist researchers who wish to train a network but have a small amount of high quality labeled data for their task of interest.

Questions this paper answers

does mixing in automatically labelled sentences during training help when only a small hand-labelled set is available?
does interleaving weakly labelled data into every epoch outperform weak-label pre-training followed by fine-tuning on strong labels?
how do I use noisy machine-generated labels to improve a sentence classifier trained on a few thousand annotated examples?
I have a few thousand annotated sentences and a lot of noisy ones — should I pre-train on the noisy ones or feed them in throughout training?: Blending weak labeled data into every training epoch with an exponentially decaying fraction raises accuracy on topic-dependent evidence detection. It beats both using weak data only for initialization epochs and training on strong labeled data alone.
Holds for: A bi-LSTM with attention on GloVe embeddings, strong-data sizes from 500 to 4,000 sentences, and 2 weak-data sources: Webis-Debate-16 and a 'that + topic concept' Wikipedia query.; The accuracy gain from blending weak labeled data grows as the amount of strong labeled data shrinks, and is smallest at the largest strong-data size tested (4,000 sentences).
Holds for: Strong-data sizes from 500 to 4,000 sentences on topic-dependent evidence detection; unpaired t-test at p < 0.05 against strong-data-only.
how many hand-labelled sentences can noisy automatic labels replace when training an evidence classifier?
what is the measured strong-label equivalence of blending a weakly labelled debate corpus for topic-dependent evidence detection?
how much annotation budget can I save on evidence detection by adding weakly labelled data to training?
is it worth annotating another 1,000 evidence sentences, or can noisy weak labels get me the same accuracy?: With Webis-Debate-16 as weak labeled data, blending lets 1,000 strong labeled instances reach accuracy comparable to 2,500 strong instances alone, and 2,000 strong instances plus weak data match 3,000 strong instances alone.
Holds for: Micro-averaged accuracy on the 35-topic test split of the released 5,785-sentence evidence dataset; the equivalence is smaller for the 'that + topic concept' weak data.
can noisy labelled sentences just be dumped into the training set together with the clean ones?
does naive union of weakly and strongly labelled data match a decaying weak-data blending schedule for evidence classification?
do I need a decay schedule when adding weak labels, or can I concatenate the weak and strong training sets?
if I just concatenate my noisy and clean training data, will accuracy go up or down?: Unifying weak and strong labeled data into one training set with no blending schedule does not help. Accuracy falls well below strong-data-only for the 'that + topic concept' weak data, and neither helps nor harms for Webis-Debate-16.
Holds for: Evidence detection with the 2 weak-data sources of the paper, at full strong-data size, shown as the single square points on the right border of each plot.; Training on the weak labeled data alone yields accuracy much lower than training on the full strong labeled data. The blending gain is therefore not simply the effect of adding more labeled examples.
Holds for: Topic-dependent evidence detection, both weak-data sources, shown as the single triangle points on the Y-axis of each plot.
how quickly should the share of noisy labelled examples be reduced from one training pass to the next?
what blend factor and how many weak-data initialization epochs work best for decaying the weak-label fraction per epoch?
how do I set the decay rate for the weak-data fraction when training a classifier on mixed-quality labels?
what decay value should I start with for shrinking the noisy-data share across epochs?: A blend factor larger than 0.05 is typically ineffective for decaying the weak-data fraction across epochs. A single initialization epoch on weak data alone suffices, and more than 1 gives slightly worse accuracy.
Holds for: Blend factors tried were 0, 0.05 and 0.2; stopping point fixed so the last 4 blending epochs were at least 95% strong labeled data. Initialization-epoch count explored on a different dataset.
is there a labelled collection of sentences marked as supporting or contesting a debate topic?
what annotated corpus exists for corpus-wide topic-dependent evidence detection, and how are its topics split across train and test?
where do I get training and test data for detecting sentences that support or contest a given topic?
can I train an evidence detector on an existing annotated corpus instead of annotating sentences myself?: A manually annotated dataset of 5,785 sentences for topic-dependent evidence detection is released, split by topic into 4,066 training sentences over 83 topics and 1,719 test sentences over 35 topics. About 40% of each split is positive.
Holds for: Wikipedia sentences over 118 debate topics, 10 crowd annotators each, majority label with ties as non-evidence, Fleiss' kappa 0.45. Topic concepts replaced by a common token.; BlendNet's evidence-detection setup is corpus-wide: evidence is detected as directly supporting or contesting a topic, with no intermediate claim. No small set of relevant articles is pre-selected, unlike earlier evidence-detection work.
Holds for: English Wikipedia as the corpus and 118 debate topics with clearly identifiable concepts; a bound on the released data, not a claim about other languages or argument types.
can a simple word pattern find sentences that argue for or against a topic without any human labelling?
what is the positive rate of the 'that + topic concept' Wikipedia query as a weak-labelling heuristic for evidence sentences?
how do I generate weak labels for argument mining from Wikipedia without annotating anything?
is a keyword pattern over Wikipedia accurate enough to use as my source of noisy evidence labels?: Sentences matching the query 'that + topic concept' are positive evidence 52% of the time, against a prior close to 40% over the whole training set. About 25% of retrieved Wikipedia sentences containing a topic concept match the query.
Holds for: Wikipedia sentences for the 118 topics of the released evidence dataset; the query's own accuracy on the test set is only 17%.
when adding noisy labelled text, does having more of it matter more than how well it matches the task?
for weak-label blending in evidence detection, does weak-data quality dominate weak-data volume across sources?
which weak-supervision source should I pick for evidence detection, a large heuristic Wikipedia set or a smaller debate-portal corpus?
I can get a huge noisy corpus or a small well-matched one — which should I blend into training?: Weak-data quality matters more than weak-data volume for blending: the smaller Webis-Debate-16 debate corpus helps more than the far larger 'that + topic concept' Wikipedia set.
Holds for: 2 weak-data sources on topic-dependent evidence detection only, so this is a single observed contrast rather than a controlled study of size versus quality.
what work proposed using noisy labelled data throughout training instead of only for pre-training?
which work introduced single-network blending of weak and strong labelled data as a general recipe for low-annotation language understanding tasks?
what should I read first about training one network on both weak and strong labels for a task with scarce annotation?
is there a paper I can cite for keeping weak supervision in the loop rather than pre-training on it?: BlendNet proposes blending weak and strong labeled data inside a single network throughout training, as a general recipe for language-understanding tasks where high-quality annotation is scarce. Earlier work used weak data only to pre-train, or trained separate networks per data type.
Holds for: As of ACL 2018; the only task evaluated is topic-dependent evidence detection with a bi-LSTM, and the paper does not claim the schedule is optimal or the only option.
does the payoff from noisy extra labels fade once enough sentences have been annotated by hand?
how does the blending gain in evidence detection scale with strong-label set size, and is the gain reducible to added training volume?
at what annotated-data size should I stop bothering with weak supervision for evidence detection?
my annotated set keeps growing — will weak labels still buy me anything?: The accuracy gain from blending weak labeled data grows as the amount of strong labeled data shrinks, and is smallest at the largest strong-data size tested (4,000 sentences).
Holds for: Strong-data sizes from 500 to 4,000 sentences on topic-dependent evidence detection; unpaired t-test at p < 0.05 against strong-data-only.; Training on the weak labeled data alone yields accuracy much lower than training on the full strong labeled data. The blending gain is therefore not simply the effect of adding more labeled examples.
Holds for: Topic-dependent evidence detection, both weak-data sources, shown as the single triangle points on the Y-axis of each plot.

Claims and scope

Blending weak labeled data into every training epoch with an exponentially decaying fraction raises accuracy on topic-dependent evidence detection. It beats both using weak data only for initialization epochs and training on strong labeled data alone. (Figure 1)
Scope: A bi-LSTM with attention on GloVe embeddings, strong-data sizes from 500 to 4,000 sentences, and 2 weak-data sources: Webis-Debate-16 and a 'that + topic concept' Wikipedia query.
With Webis-Debate-16 as weak labeled data, blending lets 1,000 strong labeled instances reach accuracy comparable to 2,500 strong instances alone, and 2,000 strong instances plus weak data match 3,000 strong instances alone. (Figure 1 and Section 5)
Scope: Micro-averaged accuracy on the 35-topic test split of the released 5,785-sentence evidence dataset; the equivalence is smaller for the 'that + topic concept' weak data.
The accuracy gain from blending weak labeled data grows as the amount of strong labeled data shrinks, and is smallest at the largest strong-data size tested (4,000 sentences). (Figure 1)
Scope: Strong-data sizes from 500 to 4,000 sentences on topic-dependent evidence detection; unpaired t-test at p < 0.05 against strong-data-only.
Unifying weak and strong labeled data into one training set with no blending schedule does not help. Accuracy falls well below strong-data-only for the 'that + topic concept' weak data, and neither helps nor harms for Webis-Debate-16. (Figure 1 and Section 5)
Scope: Evidence detection with the 2 weak-data sources of the paper, at full strong-data size, shown as the single square points on the right border of each plot.
Training on the weak labeled data alone yields accuracy much lower than training on the full strong labeled data. The blending gain is therefore not simply the effect of adding more labeled examples. (Figure 1 and Section 5)
Scope: Topic-dependent evidence detection, both weak-data sources, shown as the single triangle points on the Y-axis of each plot.
A blend factor larger than 0.05 is typically ineffective for decaying the weak-data fraction across epochs. A single initialization epoch on weak data alone suffices, and more than 1 gives slightly worse accuracy. (Section 5)
Scope: Blend factors tried were 0, 0.05 and 0.2; stopping point fixed so the last 4 blending epochs were at least 95% strong labeled data. Initialization-epoch count explored on a different dataset.
A manually annotated dataset of 5,785 sentences for topic-dependent evidence detection is released, split by topic into 4,066 training sentences over 83 topics and 1,719 test sentences over 35 topics. About 40% of each split is positive. (Section 4.1)
Scope: Wikipedia sentences over 118 debate topics, 10 crowd annotators each, majority label with ties as non-evidence, Fleiss' kappa 0.45. Topic concepts replaced by a common token.
Sentences matching the query 'that + topic concept' are positive evidence 52% of the time, against a prior close to 40% over the whole training set. About 25% of retrieved Wikipedia sentences containing a topic concept match the query. (Section 4.2)
Scope: Wikipedia sentences for the 118 topics of the released evidence dataset; the query's own accuracy on the test set is only 17%.
Weak-data quality matters more than weak-data volume for blending: the smaller Webis-Debate-16 debate corpus helps more than the far larger 'that + topic concept' Wikipedia set. (Section 6)
Scope: 2 weak-data sources on topic-dependent evidence detection only, so this is a single observed contrast rather than a controlled study of size versus quality.
BlendNet proposes blending weak and strong labeled data inside a single network throughout training, as a general recipe for language-understanding tasks where high-quality annotation is scarce. Earlier work used weak data only to pre-train, or trained separate networks per data type. (Section 2.1)
Scope: As of ACL 2018; the only task evaluated is topic-dependent evidence detection with a bi-LSTM, and the paper does not claim the schedule is optimal or the only option.
BlendNet's evidence-detection setup is corpus-wide: evidence is detected as directly supporting or contesting a topic, with no intermediate claim. No small set of relevant articles is pre-selected, unlike earlier evidence-detection work. (Section 2.2)
Scope: English Wikipedia as the corpus and 118 debate topics with clearly identifiable concepts; a bound on the released data, not a claim about other languages or argument types.

Common misreadings

Blending weak labeled data is not shown to be the best possible schedule; the paper explicitly presents one method that works and reports that a schedule increasing the weak-data fraction gave similar results.
The gains from blending are not demonstrated across many tasks: the only evaluated task is topic-dependent evidence detection with a bi-LSTM plus attention, and generality to other language-understanding tasks is a conjecture.
The 'that + topic concept' query is not an evidence detector; as a standalone classifier its test-set accuracy is 17%, and it is useful only as a source of weakly labeled positives.
Weak labeled data is not a substitute for human annotation: training on weak data alone gives accuracy much lower than training on the full strong labeled set.
A larger weak-labeled corpus is not automatically better; the smaller Webis-Debate-16 set helped more than a far larger Wikipedia-query set.

Terminology in this paper

strong labeled data (SLD): High-quality, human-annotated labels for the target task, typically scarce and expensive to obtain.
weak labeled data (WLD): Freely obtainable but noisy labels, formed as two disjoint sets in which the positive set has a substantially higher probability of containing true positives than the negative set; that probability gap is the entire training signal.
blend factor: A value alpha in [0,1] controlling how the weak-data fraction decays across training: in the k-th blending epoch, alpha^k of the weak labeled data is mixed with all available strong labeled data.
initialization epochs: Epochs at the start of training that use only weak labeled data and no strong labeled data, before the blending epochs begin.
blending epochs: Epochs that feed the network all available strong labeled data plus a decaying fraction of the weak labeled data, in random order.

How to cite

@inproceedings{DBLP:conf/acl/ShnarchPDGHCAS18,author       = {Eyal Shnarch and
                  Carlos Alzate and
                  Lena Dankin and
                  Martin Gleize and
                  Yufang Hou and
                  Leshem Choshen and
                  Ranit Aharonov and
                  Noam Slonim},
  editor       = {Iryna Gurevych and
                  Yusuke Miyao},
  title        = {Will it Blend? Blending Weak and Strong Labeled Data in a Neural Network
                  for Argumentation Mining},
  booktitle    = {Proceedings of the 56th Annual Meeting of the Association for Computational
                  Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume
                  2: Short Papers},
  pages        = {599--605},
  publisher    = {Association for Computational Linguistics},
  year         = {2018},
  url          = {https://aclanthology.org/P18-2095/},
  doi          = {10.18653/V1/P18-2095},
  timestamp    = {Fri, 06 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/ShnarchPDGHCAS18.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.