Cluster & Tune: Boost Cold Start Performance in Text Classification

Eyal Shnarch; Ariel Gera; Alon Halfon; Lena Dankin; Leshem Choshen; Ranit Aharonov; Noam Slonim

doi:10.18653/V1/2022.ACL-LONG.526

Cluster & Tune: Boost Cold Start Performance in Text Classification

BERT inter-trained to predict unsupervised cluster labels before fine-tuning on scarce labeled data

Eyal Shnarch, Ariel Gera, Alon Halfon, Lena Dankin, Leshem Choshen, Ranit Aharonov, Noam Slonim · ACL 2022 · 2022

In one sentence

Cluster & Tune inserts an unsupervised intermediate phase between pre-training and fine-tuning: cluster the unlabeled target-domain training data with sequential Information Bottleneck over bag-of-words, then train BERT to predict the cluster label before fine-tuning on the few available labels.

Abstract

In real-world scenarios, a text classification task often begins with a cold start, when labeled data is scarce. In such cases, the common practice of fine-tuning pre-trained models, such as BERT, for a target classification task, is prone to produce poor performance. We suggest a method to boost the performance of such models by adding an intermediate unsupervised classification task, between the pre-training and fine-tuning phases. As such an intermediate task, we perform clustering and train the pre-trained model on predicting the cluster labels.We test this hypothesis on various data sets, and show that this additional classification phase can significantly improve performance, mainly for topical classification tasks, when the number of labeled instances available for fine-tuning is only a couple of dozen to a few hundred.

Questions this paper answers

if I only have a few dozen labeled sentences, is there anything I can train a language model on first to make classification work better?
how much does inter-training BERT on unsupervised cluster pseudo-labels improve accuracy in a cold-start text classification setup with 64 labeled examples?
how do I get usable text classification accuracy when I can only afford to annotate a few dozen examples?
I have almost no annotated data for my classifier — should I add an unsupervised training phase before fine-tuning BERT?: Inter-training BERT to predict 50 sIB cluster labels raises average accuracy on 6 topical datasets from 26.9% to 49.6% with only 64 labeled fine-tuning examples. That is a 110% relative accuracy gain and a 33% error reduction over plain BERT fine-tuning.
Holds for: BERT-BASE (110M), 64 fine-tuning examples (128 for 20 newsgroups), 5 repetitions; no dev set, 50 clusters and 1 inter-training epoch fixed for all datasets.; The accuracy gain of cluster-based inter-training over plain BERT fine-tuning is statistically significant for labeling budgets up to 512 examples, and not significant beyond 512. Bonferroni-corrected paired-t-test p-values run from 1×10⁻⁶ at 64 samples to 9×10⁻³ at 512.
Holds for: Paired t-tests pooling all 9 datasets and 5 repetitions per labeling budget; budgets tested from 64 to 1024 examples with BERT-BASE.
is it better to keep training a language model on my own unlabeled text, or to train it to predict which group each document falls into?
does clustering-based inter-training outperform continued masked language model pre-training on the target corpus as an intermediate task?
I have a large unlabeled in-domain corpus — how do I use it best before fine-tuning on a tiny labeled set?
should I run domain-adaptive MLM pre-training, cluster-label inter-training, or both before fine-tuning?: Clustering as the intermediate task beats continued MLM pre-training on the same target corpus for labeling budgets of 64, 128 and 192 examples, with no significant difference at 256 or above. Bonferroni-corrected p-values are 8×10⁻⁵, 3×10⁻³ and 4×10⁻².
Holds for: BERT_IT:MLM baseline trained 30 epochs with replication rate 5 on the same unlabeled train set; comparison pools all 9 datasets and 5 repetitions per budget.; Running MLM inter-training and then cluster-label inter-training in sequence outperforms either intermediate task alone on topical datasets, showing the two phases are complementary at the cost of added runtime.
Holds for: Topical datasets, labeling budgets 64-1024, BERT-BASE, sIB clustering over bag-of-words; the paper reports no p-values for this specific pairwise comparison.
does grouping unlabeled documents before training help for spam or sentiment tasks, or only when the categories are about subject matter?
on which target tasks does cluster-based inter-training fail to yield gains, and does normalized mutual information between clusters and labels predict that?
how do I tell in advance whether an unsupervised clustering phase will help my particular classification task?
my labels are sentiment and subjectivity rather than topics — will a clustering pre-finetuning step do anything for me?: On the 3 non-topical datasets (SMS spam, Subjectivity, Polarity) cluster-based inter-training raises average accuracy only from 82.6% to 85.4% at 64 labeled examples, a 3% relative gain. Polarity shows 0% gain, and results stay comparable to plain BERT rather than worse.
Holds for: Sentiment, subjectivity and spam tasks whose distinction is stylistic rather than topical; BERT-BASE, 64 labeled examples, sIB clustering over stemmed bag-of-words with 50 clusters.; The benefit of cluster-based inter-training tracks the normalized mutual information between the 50-cluster partition and the target labels. Datasets with NMI near zero show no clear gain, and the 3 datasets with the lowest NMI are exactly those where inter-training did not help.
Holds for: 9 datasets, NMI computed over the full train set between sIB cluster labels and gold labels, error reduction measured at 64 fine-tuning samples; a correlation across 9 datasets, not a per-dataset predictive rule.
does it matter which algorithm I use to group the unlabeled documents into pseudo-labels?
does sequential Information Bottleneck over stemmed bag-of-words beat K-means or Hartigan's K-means over averaged GloVe embeddings as the source of inter-training pseudo-labels?
which clustering method should I use to produce the pseudo-labels for an intermediate BERT training phase?
can I just use K-means over sentence embeddings for the clustering step, or do I need a bag-of-words method?: sIB clustering over stemmed bag-of-words representations gives better downstream accuracy after inter-training than K-means or Hartigan's K-means over averaged GloVe embeddings on most of the datasets tested.
Holds for: 50 clusters, 1 inter-training epoch, sIB with 10 restarts and 15 iterations and a 10K-word vocabulary; "most cases" rather than all, and no significance test reported for this comparison.
with only a handful of labeled examples, could a simple word-count classifier or the document groups themselves do just as well as retraining BERT?
do Naive Bayes and SVM baselines over BOW or GloVe, or nearest-cluster label propagation, match cluster-inter-trained BERT under small labeling budgets?
how do I check whether a clustering-based BERT pipeline is worth it over a bag-of-words classifier on the same few labels?
should I bother fine-tuning BERT on cluster labels, or just label my clusters and classify by nearest cluster?: The gains of cluster-based inter-training do not come merely from bag-of-words information. Multinomial Naive Bayes and SVM classifiers over BOW or GloVe representations, trained on the same labeled samples, were all inferior to the cluster-inter-trained BERT.
Holds for: 4 reference settings (NB_BOW, NB_GloVe, SVM_BOW, SVM_GloVe) on the same 9 datasets and the same 5 label samplings per budget.; Using the 50 sIB clusters directly as a classifier is generally not on par with cluster-based BERT inter-training, though it is surprisingly effective where NMI is high and the budget small. Each cluster is labeled from a budget-proportional sample and each test example takes its nearest cluster's dominant label.
Holds for: Heuristic guarantees at least 1 labeled instance per cluster; evaluated on the same 9 datasets and budgets from 64 to 1024.
what actually changes inside a language model's sentence representations after training it to predict document groups?
how do BERT [CLS] embeddings change after cluster-label inter-training, measured by distance to class centroids?
is there evidence that cluster-label inter-training gives me a genuinely better starting point for fine-tuning, not just better numbers?: After cluster-label inter-training, BERT's [CLS] embeddings place same-class examples closer together in every dataset tested, measured by a permutation-normalized Euclidean distance to class centroids. t-SNE plots of topical datasets show visibly cleaner class separation.
Holds for: Embeddings over the full train set, gold labels used only for measurement; normalization uses a permutation test with 1000 repetitions. The visual separation is qualitative and clearest on topical data such as DBpedia, unlike Polarity.
how much extra compute and time does it cost to group unlabeled documents and train a model on those groups first?
what is the runtime overhead of sIB clustering plus one inter-training epoch on a single GPU?
how do I budget GPU time for adding a clustering-based intermediate training phase to a text classification pipeline?
I have one V100 and a 15K-document corpus — can I afford the clustering intermediate phase?: The clustering step for the intermediate task takes only a few seconds. The single inter-training epoch takes five and a half minutes for the largest 15K-instance train set on one Tesla V100-PCIE-16GB GPU.
Holds for: BERT-BASE, batch size 64, max sequence length 128, learning rate 3×10⁻⁵; datasets of 3.9K to 15K training instances.
which work first used grouping of unlabeled text as a training step between language model pre-training and fine-tuning on few labels?
which paper introduced unsupervised clustering pseudo-labels as an intermediate task for cold-start text classification in NLP?
where do I start reading about training text classifiers when almost nothing is labeled yet?
what should I read first if I need a text classifier and have no annotation budget to speak of?: Cluster & Tune introduces unsupervised text clustering as an intermediate training task for cold-start text classification, transferring to NLP the computer-vision practice of using cluster assignments as pseudo-labels for representation learning.
Holds for: As of publication at ACL 2022; earlier NLP work used clustering mainly for non-transfer purposes. Demonstrated for English single-label text classification with BERT-BASE only.; Cluster & Tune requires no additional labeled data and no per-task design, since the pseudo-labels come from clustering the unlabeled target-domain corpus. It is therefore an alternative to supervised intermediate tasks that need labeled data from another task.
Holds for: Requires an unlabeled corpus of the target domain on the order of several thousand examples; hyperparameters (50 clusters, 1 epoch) were fixed a priori rather than tuned per dataset, and no dev set is assumed.
once I can afford to label more examples, does an unsupervised pre-finetuning step stop being worth the trouble?
up to what labeling budget is the gain from cluster-based inter-training over plain BERT fine-tuning statistically significant?
how do I decide whether my annotation budget is small enough for a clustering intermediate phase to still pay off?
I can label about 1000 examples — is a cluster-based intermediate training phase still going to help me?: The accuracy gain of cluster-based inter-training over plain BERT fine-tuning is statistically significant for labeling budgets up to 512 examples, and not significant beyond 512. Bonferroni-corrected paired-t-test p-values run from 1×10⁻⁶ at 64 samples to 9×10⁻³ at 512.
Holds for: Paired t-tests pooling all 9 datasets and 5 repetitions per labeling budget; budgets tested from 64 to 1024 examples with BERT-BASE.; Inter-training BERT to predict 50 sIB cluster labels raises average accuracy on 6 topical datasets from 26.9% to 49.6% with only 64 labeled fine-tuning examples. That is a 110% relative accuracy gain and a 33% error reduction over plain BERT fine-tuning.
Holds for: BERT-BASE (110M), 64 fine-tuning examples (128 for 20 newsgroups), 5 repetitions; no dev set, 50 clusters and 1 inter-training epoch fixed for all datasets.
does training on document groups need any labeled data beyond the handful used for the final task?
what supervision does cluster-based inter-training require beyond the target-task labels, compared with supervised intermediate tasks?
how do I run an intermediate training phase when I have no labeled data from any other task to borrow?
I have no annotations at all and no related labeled dataset — can I still run the clustering intermediate phase, and what will it cost me?: Cluster & Tune requires no additional labeled data and no per-task design, since the pseudo-labels come from clustering the unlabeled target-domain corpus. It is therefore an alternative to supervised intermediate tasks that need labeled data from another task.
Holds for: Requires an unlabeled corpus of the target domain on the order of several thousand examples; hyperparameters (50 clusters, 1 epoch) were fixed a priori rather than tuned per dataset, and no dev set is assumed.; The clustering step for the intermediate task takes only a few seconds. The single inter-training epoch takes five and a half minutes for the largest 15K-instance train set on one Tesla V100-PCIE-16GB GPU.
Holds for: BERT-BASE, batch size 64, max sequence length 128, learning rate 3×10⁻⁵; datasets of 3.9K to 15K training instances.

Claims and scope

Inter-training BERT to predict 50 sIB cluster labels raises average accuracy on 6 topical datasets from 26.9% to 49.6% with only 64 labeled fine-tuning examples. That is a 110% relative accuracy gain and a 33% error reduction over plain BERT fine-tuning. (Table 2)
Scope: BERT-BASE (110M), 64 fine-tuning examples (128 for 20 newsgroups), 5 repetitions; no dev set, 50 clusters and 1 inter-training epoch fixed for all datasets.
On the 3 non-topical datasets (SMS spam, Subjectivity, Polarity) cluster-based inter-training raises average accuracy only from 82.6% to 85.4% at 64 labeled examples, a 3% relative gain. Polarity shows 0% gain, and results stay comparable to plain BERT rather than worse. (Table 2)
Scope: Sentiment, subjectivity and spam tasks whose distinction is stylistic rather than topical; BERT-BASE, 64 labeled examples, sIB clustering over stemmed bag-of-words with 50 clusters.
The accuracy gain of cluster-based inter-training over plain BERT fine-tuning is statistically significant for labeling budgets up to 512 examples, and not significant beyond 512. Bonferroni-corrected paired-t-test p-values run from 1×10⁻⁶ at 64 samples to 9×10⁻³ at 512. (Table 3)
Scope: Paired t-tests pooling all 9 datasets and 5 repetitions per labeling budget; budgets tested from 64 to 1024 examples with BERT-BASE.
Clustering as the intermediate task beats continued MLM pre-training on the same target corpus for labeling budgets of 64, 128 and 192 examples, with no significant difference at 256 or above. Bonferroni-corrected p-values are 8×10⁻⁵, 3×10⁻³ and 4×10⁻². (Table 3)
Scope: BERT_IT:MLM baseline trained 30 epochs with replication rate 5 on the same unlabeled train set; comparison pools all 9 datasets and 5 repetitions per budget.
Running MLM inter-training and then cluster-label inter-training in sequence outperforms either intermediate task alone on topical datasets, showing the two phases are complementary at the cost of added runtime. (Figure 2)
Scope: Topical datasets, labeling budgets 64-1024, BERT-BASE, sIB clustering over bag-of-words; the paper reports no p-values for this specific pairwise comparison.
The benefit of cluster-based inter-training tracks the normalized mutual information between the 50-cluster partition and the target labels. Datasets with NMI near zero show no clear gain, and the 3 datasets with the lowest NMI are exactly those where inter-training did not help. (Figure 4)
Scope: 9 datasets, NMI computed over the full train set between sIB cluster labels and gold labels, error reduction measured at 64 fine-tuning samples; a correlation across 9 datasets, not a per-dataset predictive rule.
sIB clustering over stemmed bag-of-words representations gives better downstream accuracy after inter-training than K-means or Hartigan's K-means over averaged GloVe embeddings on most of the datasets tested. (Figure 6, Appendix C)
Scope: 50 clusters, 1 inter-training epoch, sIB with 10 restarts and 15 iterations and a 10K-word vocabulary; "most cases" rather than all, and no significance test reported for this comparison.
The gains of cluster-based inter-training do not come merely from bag-of-words information. Multinomial Naive Bayes and SVM classifiers over BOW or GloVe representations, trained on the same labeled samples, were all inferior to the cluster-inter-trained BERT. (Figure 5, Appendix B)
Scope: 4 reference settings (NB_BOW, NB_GloVe, SVM_BOW, SVM_GloVe) on the same 9 datasets and the same 5 label samplings per budget.
Using the 50 sIB clusters directly as a classifier is generally not on par with cluster-based BERT inter-training, though it is surprisingly effective where NMI is high and the budget small. Each cluster is labeled from a budget-proportional sample and each test example takes its nearest cluster's dominant label. (Figure 5, Appendix B)
Scope: Heuristic guarantees at least 1 labeled instance per cluster; evaluated on the same 9 datasets and budgets from 64 to 1024.
After cluster-label inter-training, BERT's [CLS] embeddings place same-class examples closer together in every dataset tested, measured by a permutation-normalized Euclidean distance to class centroids. t-SNE plots of topical datasets show visibly cleaner class separation. (Figure 3, Section 5.3)
Scope: Embeddings over the full train set, gold labels used only for measurement; normalization uses a permutation test with 1000 repetitions. The visual separation is qualitative and clearest on topical data such as DBpedia, unlike Polarity.
The clustering step for the intermediate task takes only a few seconds. The single inter-training epoch takes five and a half minutes for the largest 15K-instance train set on one Tesla V100-PCIE-16GB GPU. (Section 3.2)
Scope: BERT-BASE, batch size 64, max sequence length 128, learning rate 3×10⁻⁵; datasets of 3.9K to 15K training instances.
Cluster & Tune introduces unsupervised text clustering as an intermediate training task for cold-start text classification, transferring to NLP the computer-vision practice of using cluster assignments as pseudo-labels for representation learning. (Section 6)
Scope: As of publication at ACL 2022; earlier NLP work used clustering mainly for non-transfer purposes. Demonstrated for English single-label text classification with BERT-BASE only.
Cluster & Tune requires no additional labeled data and no per-task design, since the pseudo-labels come from clustering the unlabeled target-domain corpus. It is therefore an alternative to supervised intermediate tasks that need labeled data from another task. (Section 2)
Scope: Requires an unlabeled corpus of the target domain on the order of several thousand examples; hyperparameters (50 clusters, 1 epoch) were fixed a priori rather than tuned per dataset, and no dev set is assumed.

Common misreadings

Cluster-based inter-training is not a general accuracy boost for text classification: on non-topical tasks such as Polarity, Subjectivity and SMS spam the average relative gain at 64 labels is 3%, and the advantage over plain BERT fine-tuning is not statistically significant above 512 labeled examples.
The number of clusters is not meant to match the number of target classes: setting it equal to the number of classes gave inferior accuracy, and 50 clusters was used for all 9 datasets regardless of their 2 to 20 classes.
Cluster & Tune is not a label-free classifier. The clustering phase produces pseudo-labels for inter-training only, and a final supervised fine-tuning phase on real target-task labels is still required.
The method does not replace continued MLM pre-training. Running MLM inter-training first and cluster inter-training second outperformed either one alone on topical datasets.
The clustering pseudo-labels are not tailored to the target task; they are generated independently of it, so no knowledge of the target label set is assumed when producing them.

Terminology in this paper

inter-training: An intermediate training phase inserted between a model's general pre-training and its supervised fine-tuning, which sees the target-task corpus or domain but none of its labeled instances.
BERT_IT:CLUST: BERT inter-trained as a 50-way classifier predicting sequential-Information-Bottleneck cluster assignments of the unlabeled target-domain train set, with the cluster-prediction head discarded before fine-tuning.
BERT_IT:MLM: BERT further pre-trained with masked language modeling on the unlabeled target-domain corpus before fine-tuning, also known as adaptive or further pre-training.
topical dataset: A classification dataset whose classes reflect a high-level distinction about what a text is about, such as sports versus economics, as opposed to non-topical datasets whose classes turn on style, fine details or negation.
Normalized Embeddings' Distance (NED): The average Euclidean distance of instance embeddings from their own class centroid, divided by the expected value of that distance under random permutations of the class labels.
cold start: The situation at the beginning of a text classification project in which unlabeled data from the target domain is available but labeled examples number only a couple of dozen to a few hundred.

How to cite

@inproceedings{DBLP:conf/acl/ShnarchGHDCAS22,author       = {Eyal Shnarch and
                  Ariel Gera and
                  Alon Halfon and
                  Lena Dankin and
                  Leshem Choshen and
                  Ranit Aharonov and
                  Noam Slonim},
  editor       = {Smaranda Muresan and
                  Preslav Nakov and
                  Aline Villavicencio},
  title        = {Cluster {\&} Tune: Boost Cold Start Performance in Text Classification},
  booktitle    = {Proceedings of the 60th Annual Meeting of the Association for Computational
                  Linguistics (Volume 1: Long Papers), {ACL} 2022, Dublin, Ireland,
                  May 22-27, 2022},
  pages        = {7639--7653},
  publisher    = {Association for Computational Linguistics},
  year         = {2022},
  url          = {https://doi.org/10.18653/v1/2022.acl-long.526},
  doi          = {10.18653/V1/2022.ACL-LONG.526},
  timestamp    = {Mon, 01 Aug 2022 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/ShnarchGHDCAS22.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.