Active Learning for BERT: An Empirical Study

Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim · EMNLP 2020 · 2020

In one sentence

A 2,520-run study of 7 active learning strategies on top of BERT for binary text classification with a 100-example seed and 50-example batches, showing the largest gains in the realistic case where the seed comes from a keyword query over a rare positive class.

Abstract

Real world scenarios present a challenge for text classification, since labels are usually expensive and the data is often characterized by class imbalance. Active Learning (AL) is a ubiquitous paradigm to cope with data scarcity. Recently, pre-trained NLP models, and BERT in particular, are receiving massive attention due to their outstanding performance in various NLP tasks. However, the use of AL with deep pre-trained models has so far received little consideration. Here, we present a large-scale empirical study on active learning techniques for BERT-based classification, addressing a diverse set of AL strategies and datasets. We focus on practical scenarios of binary text classification, where the annotation budget is very small, and the data is often skewed. Our results demonstrate that AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. We release our research framework, aiming to facilitate future research along the lines explored here.

Questions this paper answers

does choosing which examples to label help when there are only a few hundred labels?
how much F1 does active learning gain over random selection when fine-tuning BERT on imbalanced text classification?
how do I spend a tiny annotation budget on the examples that help a text classifier most?
I can label 500 examples -- should I choose them with active learning or at random?
Active learning strategies on top of BERT improve F1 over a random-selection baseline by 4-8% on average in the imbalanced-practical scenario. In that scenario the initial labeled seed of 100 examples is drawn from a keyword query.
Holds for: Binary classification on 4 datasets with a queryable positive class (ISEAR fear, TREC location, Wiki attack, AG's News-imb world); BERT-base, 5 iterations of 50 labels, dev set used for model selection.
All 6 non-random active learning strategies tested with BERT beat random selection significantly in both imbalanced scenarios, with Wilcoxon p < 10^-2 after Bonferroni correction. In the balanced scenario EGL and Perceptron Ensemble show no significant gain.
Holds for: 10 binary datasets split across 14 dataset-scenario combinations, 5 seeds x 5 iterations per strategy; p-values computed over all (strategy, random) result pairs; accuracy used for balanced, F1 for imbalanced scenarios.
which way of choosing examples to label works best with a transformer classifier?
is there a clear winner among uncertainty, core-set and discriminative active learning strategies for BERT?
how do I pick an acquisition strategy for active learning with BERT?
which active learning strategy should I implement first?
No single active learning strategy consistently outperforms the others when paired with BERT: pairwise Wilcoxon tests across the 3 scenarios find no significant performance difference between any pair of the 7 strategies.
Holds for: Least Confidence, Monte Carlo Dropout, Perceptron Ensemble, EGL, Core-Set, DAL and Random on binary tasks, 100-example seed plus 5 batches of 50; larger budgets and multi-class tasks untested.
Per-iteration batch selection cost with BERT differs by orders of magnitude across strategies: assuming 7,000 unlabeled examples, random selection takes under 1 second and EGL 1,106 seconds. Dropout takes 840 seconds, Perceptron Ensemble 370, DAL 167, Core-Set 98 and Least Confidence 84.
Holds for: Selection time only, excluding BERT fine-tuning; Intel Xeon E5-2699 v4 CPUs with a single Nvidia Tesla K80 GPU per run; dominated by BERT inference for every strategy except random.
can a classifier recover if its first labeled examples all came from one keyword search?
does a keyword-query seed permanently depress BERT classification quality relative to an unbiased positive sample?
how do I bootstrap a rare-class classifier when the only way to find positives is a keyword query?
my seed set came from a keyword search -- is my classifier stuck with that bias?
A keyword-query seed gives BERT a worse initial F1 than an unbiased sample of positives. After several active learning iterations of 50 labels each, the two seeding regimes converge to similar classification performance.
Holds for: Compared at iteration 0 versus later iterations on the datasets run in both imbalanced and imbalanced-practical scenarios; simple string-match queries whose own recall is low.
When BERT is seeded with a biased keyword-query sample of the minority class, the F1 gain from active learning is dominated entirely by rising recall. In the unbiased imbalanced seeding the gain comes mostly from precision.
Holds for: The 4 imbalanced-practical datasets versus the 6 imbalanced datasets, BERT-base, F1 at the default 0.5 threshold; precision and recall curves are in Appendix Figures 4 and 5.
The imbalanced-practical setting introduced in "Active Learning for BERT: An Empirical Study" is a low-resource evaluation protocol where the labeled seed comes from a simple keyword query. It replaces the usual assumption of an unbiased sample of the rare positive class.
Holds for: Applied to 4 datasets for which a simple string or sub-string match query with enough hits could be written; the queries are deliberately naive and better queries are likely to exist.
does choosing examples to label find more of the rare cases, or judge the found ones better?
does the active learning F1 gain on skewed data come from precision or from recall?
how do I tell whether active learning is improving coverage of a rare class or its precision?
When BERT is seeded with a biased keyword-query sample of the minority class, the F1 gain from active learning is dominated entirely by rising recall. In the unbiased imbalanced seeding the gain comes mostly from precision.
Holds for: The 4 imbalanced-practical datasets versus the 6 imbalanced datasets, BERT-base, F1 at the default 0.5 threshold; precision and recall curves are in Appendix Figures 4 and 5.
how much computation does each way of choosing examples to label cost per round?
what is the per-iteration selection runtime of EGL, Monte Carlo Dropout, DAL, Core-Set and Least Confidence over 7,000 unlabeled examples?
how do I keep the selection step from costing more than the fine-tuning step?
which acquisition strategies are cheap enough for me to run every round?
Per-iteration batch selection cost with BERT differs by orders of magnitude across strategies: assuming 7,000 unlabeled examples, random selection takes under 1 second and EGL 1,106 seconds. Dropout takes 840 seconds, Perceptron Ensemble 370, DAL 167, Core-Set 98 and Least Confidence 84.
Holds for: Selection time only, excluding BERT fine-tuning; Intel Xeon E5-2699 v4 CPUs with a single Nvidia Tesla K80 GPU per run; dominated by BERT inference for every strategy except random.
why does fine-tuning go unstable when a rare class barely appears in the first labeled batch?
what makes BERT fine-tuning unstable at a positive-class prior at or below 15% with a 100-instance random seed, and what stabilised it?
how do I stabilise fine-tuning on a small, highly imbalanced seed without buying more labels?
my seed set has almost no positives and training keeps collapsing -- what do I do?
For datasets whose positive class prior is at or below 15%, a random seed of 100 labeled instances led to unstable BERT fine-tuning runs. The study handled this by adding 100 further instances weakly labeled as negative.
Holds for: BERT-base fine-tuned for 5 epochs at learning rate 5e-5, batch size 50, maximum sequence length 100 tokens; observed on the 6 skewed datasets, with the instability data not shown in the paper.
do different ways of choosing examples to label end up picking the same ones?
how much do batches selected by different acquisition functions over the same BERT model overlap?
how do I tell whether combining two acquisition strategies could add anything?
Batches chosen by different active learning strategies over the same BERT model overlap little: expected pairwise batch overlap does not exceed 15%, with the highest overlap between EGL and Least Confidence.
Holds for: 50-example batches from the same unlabeled pool and the same trained BERT model, across 7 strategies and 14 dataset-scenario combinations; combining low-overlap strategies untested.
which ways of choosing examples cover the data broadly rather than picking oddities?
how do diversity and representativeness differ across acquisition strategies measured in BERT [CLS] space?
how do I check whether an acquisition strategy is selecting outliers?
Discriminative Active Learning selects the most diverse and the most representative batches of the 7 strategies compared using BERT [CLS] representations. Greedy Core-Set scores low on representativeness except in the imbalanced-practical scenario.
Holds for: Measured on the first 50-example batch after the initial BERT model, averaged over datasets and seeds per scenario; diversity per Zhdanov (2019) and representativeness from inverse KNN-density with K=10 on Euclidean [CLS] distances.
what is a good study comparing ways of choosing which text examples to label?
what work systematically compared classical and deep active learning strategies on top of BERT?
where should I start reading about active learning for text classification with transformers?
which paper should I cite for whether active learning helps BERT?
"Active Learning for BERT: An Empirical Study" is a systematic empirical comparison of classical and deep active learning strategies on top of BERT for text classification. It covers 10 binary datasets and 2,520 fine-tuning runs.
Holds for: As of EMNLP 2020: earlier deep-AL work on text either used non-BERT models or applied a narrow set of strategies to one or two tasks with specific BERT variants; binary classification only, English datasets, BERT-base.
"Active Learning for BERT: An Empirical Study" releases an open-source low-resource text classification framework with the 10 datasets, implementations of the 7 active learning strategies and an automatic evaluation harness. New AL strategies can be plugged in.
Holds for: Framework as released at publication in 2020, built around BERT-base and binary classification tasks; released at github.com/IBM/low-resource-text-classification-framework.
is there code and data for trying out example-selection strategies on text classification?
what open-source low-resource text classification framework, datasets and strategy implementations were released?
how do I benchmark a new acquisition strategy against existing ones without rebuilding a text classification harness from scratch?
can I plug my own selection strategy into an existing harness?
"Active Learning for BERT: An Empirical Study" releases an open-source low-resource text classification framework with the 10 datasets, implementations of the 7 active learning strategies and an automatic evaluation harness. New AL strategies can be plugged in.
Holds for: Framework as released at publication in 2020, built around BERT-base and binary classification tasks; released at github.com/IBM/low-resource-text-classification-framework.
how many datasets and training runs went into comparing ways of choosing which text examples to label?
how many binary datasets and BERT fine-tuning runs back the active learning comparison?
is the evidence broad enough for me to rely on these active learning conclusions?
"Active Learning for BERT: An Empirical Study" is a systematic empirical comparison of classical and deep active learning strategies on top of BERT for text classification. It covers 10 binary datasets and 2,520 fine-tuning runs.
Holds for: As of EMNLP 2020: earlier deep-AL work on text either used non-BERT models or applied a narrow set of strategies to one or two tasks with specific BERT variants; binary classification only, English datasets, BERT-base.
All 6 non-random active learning strategies tested with BERT beat random selection significantly in both imbalanced scenarios, with Wilcoxon p < 10^-2 after Bonferroni correction. In the balanced scenario EGL and Perceptron Ensemble show no significant gain.
Holds for: 10 binary datasets split across 14 dataset-scenario combinations, 5 seeds x 5 iterations per strategy; p-values computed over all (strategy, random) result pairs; accuracy used for balanced, F1 for imbalanced scenarios.

Claims and scope

Common misreadings

Terminology in this paper

imbalanced-practical scenario
An active learning setup for a rare target class in which the initial labeled seed is drawn from the results of a simple keyword query, yielding a positive-enriched but biased sample, rather than from an unbiased sample of positives.
imbalanced scenario
An active learning setup for a target class with prior at or below 15% in which the initial seed's 100 fully labeled examples are drawn at random from the dataset's positives, assumed available via high-precision heuristics.
Perceptron Ensemble (PE)
An uncertainty-sampling strategy that averages the predictions of 10 perceptrons trained on the [CLS] vectors of a fine-tuned BERT model, used as a cheap substitute for an ensemble of full BERT models.
Representativeness of a batch
One over the average KNN-density of the batch's instances, where density is the mean Euclidean distance from an instance to its K=10 nearest neighbours in the unlabeled pool in [CLS] representation space; low values indicate outlier selection.
Diversity of a batch
The inverse of the average, over all unlabeled instances, of the minimum Euclidean distance from that instance to any member of the selected batch in [CLS] representation space.

How to cite

@inproceedings{DBLP:conf/emnlp/Ein-DorHGSDCDAK20,author       = {Liat Ein{-}Dor and
                  Alon Halfon and
                  Ariel Gera and
                  Eyal Shnarch and
                  Lena Dankin and
                  Leshem Choshen and
                  Marina Danilevsky and
                  Ranit Aharonov and
                  Yoav Katz and
                  Noam Slonim},
  editor       = {Bonnie Webber and
                  Trevor Cohn and
                  Yulan He and
                  Yang Liu},
  title        = {Active Learning for {BERT:} An Empirical Study},
  booktitle    = {Proceedings of the 2020 Conference on Empirical Methods in Natural
                  Language Processing, {EMNLP} 2020, Online, November 16-20, 2020},
  pages        = {7949--7962},
  publisher    = {Association for Computational Linguistics},
  year         = {2020},
  url          = {https://doi.org/10.18653/v1/2020.emnlp-main.638},
  doi          = {10.18653/V1/2020.EMNLP-MAIN.638},
  timestamp    = {Tue, 20 Aug 2024 07:54:43 +0200},
  biburl       = {https://dblp.org/rec/conf/emnlp/Ein-DorHGSDCDAK20.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.