Real world scenarios present a challenge for text classification, since labels are usually expensive and the data is often characterized by class imbalance. Active Learning (AL) is a ubiquitous paradigm to cope with data scarcity. Recently, pre-trained NLP models, and BERT in particular, are receiving massive attention due to their outstanding performance in various NLP tasks. However, the use of AL with deep pre-trained models has so far received little consideration. Here, we present a large-scale empirical study on active learning techniques for BERT-based classification, addressing a diverse set of AL strategies and datasets. We focus on practical scenarios of binary text classification, where the annotation budget is very small, and the data is often skewed. Our results demonstrate that AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. We release our research framework, aiming to facilitate future research along the lines explored here.
@inproceedings{DBLP:conf/emnlp/Ein-DorHGSDCDAK20,
author = {Liat Ein{-}Dor and
Alon Halfon and
Ariel Gera and
Eyal Shnarch and
Lena Dankin and
Leshem Choshen and
Marina Danilevsky and
Ranit Aharonov and
Yoav Katz and
Noam Slonim},
editor = {Bonnie Webber and
Trevor Cohn and
Yulan He and
Yang Liu},
title = {Active Learning for {BERT:} An Empirical Study},
booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing, {EMNLP} 2020, Online, November 16-20, 2020},
pages = {7949--7962},
publisher = {Association for Computational Linguistics},
year = {2020},
url = {https://doi.org/10.18653/v1/2020.emnlp-main.638},
doi = {10.18653/V1/2020.EMNLP-MAIN.638},
timestamp = {Tue, 20 Aug 2024 07:54:43 +0200},
biburl = {https://dblp.org/rec/conf/emnlp/Ein-DorHGSDCDAK20.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
See the full reference list in the paper.