# Active Learning for BERT: An Empirical Study

Authors: Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim
Venue: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, {EMNLP} 2020, Online, November 16-20, 2020 (2020)

## Abstract
Real world scenarios present a challenge for text classification, since labels are usually expensive and the data is often characterized by class imbalance. Active Learning (AL) is a ubiquitous paradigm to cope with data scarcity. Recently, pre-trained NLP models, and BERT in particular, are receiving massive attention due to their outstanding performance in various NLP tasks. However, the use of AL with deep pre-trained models has so far received little consideration. Here, we present a large-scale empirical study on active learning techniques for BERT-based classification, addressing a diverse set of AL strategies and datasets. We focus on practical scenarios of binary text classification, where the annotation budget is very small, and the data is often skewed. Our results demonstrate that AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. We release our research framework, aiming to facilitate future research along the lines explored here.

## Links
- DOI: https://doi.org/10.18653/V1/2020.EMNLP-MAIN.638
- ACL Anthology: https://aclanthology.org/2020.emnlp-main.638/
- Semantic Scholar: https://www.semanticscholar.org/paper/226262191
- Publisher: https://doi.org/10.18653/v1/2020.emnlp-main.638

## How to cite
@inproceedings{DBLP:conf/emnlp/Ein-DorHGSDCDAK20,
author       = {Liat Ein{-}Dor and
                  Alon Halfon and
                  Ariel Gera and
                  Eyal Shnarch and
                  Lena Dankin and
                  Leshem Choshen and
                  Marina Danilevsky and
                  Ranit Aharonov and
                  Yoav Katz and
                  Noam Slonim},
  editor       = {Bonnie Webber and
                  Trevor Cohn and
                  Yulan He and
                  Yang Liu},
  title        = {Active Learning for {BERT:} An Empirical Study},
  booktitle    = {Proceedings of the 2020 Conference on Empirical Methods in Natural
                  Language Processing, {EMNLP} 2020, Online, November 16-20, 2020},
  pages        = {7949--7962},
  publisher    = {Association for Computational Linguistics},
  year         = {2020},
  url          = {https://doi.org/10.18653/v1/2020.emnlp-main.638},
  doi          = {10.18653/V1/2020.EMNLP-MAIN.638},
  timestamp    = {Tue, 20 Aug 2024 07:54:43 +0200},
  biburl       = {https://dblp.org/rec/conf/emnlp/Ein-DorHGSDCDAK20.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}