Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours

Eyal Shnarch; Alon Halfon; Ariel Gera; Marina Danilevsky; Yannis Katsis; Leshem Choshen; Martín Santillán Cooper; Dina Epelboim; Zheng Zhang; Dakuo Wang

doi:10.18653/V1/2022.EMNLP-DEMOS.16

Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours

no-code system for labeling text and building a text classifier without ML expertise

Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martín Santillán Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang · EMNLP 2022 (Demo) · 2022

In one sentence

Label Sleuth is a free open-source no-code system that walks a domain expert from an unlabeled corpus to a custom binary text classifier in a few hours, training models in the background and telling the user what to label next.

Abstract

Text classification can be useful in many real-world scenarios, saving a lot of time for end users. However, building a custom classifier typically requires coding skills and ML knowledge, which poses a significant barrier for many potential users. To lift this barrier, we introduce Label Sleuth, a free open source system for labeling and creating text classifiers. This system is unique for (a) being a no-code system, making NLP accessible to non-experts, (b) guiding users through the entire labeling process until they obtain a custom classifier, making the process efficient -- from cold start to classifier in a few hours, and (c) being open for configuration and extension by developers. By open sourcing Label Sleuth we hope to build a community of users and developers that will broaden the utilization of NLP models.

Questions this paper answers

is there a free tool that lets someone with no coding skills build a classifier that sorts text into two categories?
what no-code system supports end-to-end binary text classifier construction by a domain expert over an unlabeled corpus?
how do I turn a pile of unlabeled documents into a custom text classifier without writing code or hiring a data scientist?
I have no machine learning background but need a classifier for my own documents, can Label Sleuth do that for me?: Label Sleuth is a free, open-source, no-code system in which a domain expert with no machine learning knowledge labels text and obtains a custom binary text classifier, starting from an unlabeled corpus.
Holds for: Released July 2022 under Apache 2.0; binary text classification of pre-split text elements only, with English as the default language.; Label Sleuth targets the gap between text classification techniques and non-technical practitioners such as lawyers, physicians and psychologists, who need custom classifiers but depend on ML experts to build them.
Holds for: As of publication in 2022; comparable labeling tools with ML support target data scientists and developers.
where should someone start reading about tools that let subject matter experts build text classifiers themselves?
which paper introduced a human-in-the-loop labeling system aimed at domain experts rather than machine learning practitioners?
what should I read first if I want to understand how non-technical experts can build their own text classifiers?
my team is lawyers and clinicians rather than engineers, is there published work on labeling systems built for people like us?: Label Sleuth targets the gap between text classification techniques and non-technical practitioners such as lawyers, physicians and psychologists, who need custom classifiers but depend on ML experts to build them.
Holds for: As of publication in 2022; comparable labeling tools with ML support target data scientists and developers.; Label Sleuth is a free, open-source, no-code system in which a domain expert with no machine learning knowledge labels text and obtains a custom binary text classifier, starting from an unlabeled corpus.
Holds for: Released July 2022 under Apache 2.0; binary text classification of pre-split text elements only, with English as the default language.
how does Label Sleuth differ from other text annotation tools people already use?
which text annotation platforms require no technical expertise and provide guidance on labeling errors?
how do I pick between labeling tools when nobody on my team can write code or configure a model?
we already use a commercial annotation tool, what would we gain or lose by switching to Label Sleuth?: Label Sleuth is the only one of 5 compared labeling tools that requires no technical expertise and gives ML guidance on label errors. It is also the only one of the 5 restricted to text classification.
Holds for: Prodigy, free and paid Label Studio, INCEpTION and Label Sleuth, chosen for similarity and popularity and assessed by the authors in 2022; a feature comparison rather than a usability study.
has anyone actually used a no-code labeling tool on a real job, and how much time did it take them?
what deployment case studies report annotation effort and downstream use of Label Sleuth classifiers?
how many hours should I budget to get a usable classifier for a niche category out of an interactive labeling system?
before I commit my own time, is there evidence that someone built something useful with Label Sleuth and saved effort?: A legal user built a Label Sleuth classifier for a category of high-risk contract clauses after 6 hours of work in the system. Reviewing only the highlighted clauses instead of entire contracts saved an estimated 80% of their time.
Holds for: A single early user's self-reported experience and self-estimated time saving on a contract-review task, not a controlled measurement.; The VIRA COVID-19 vaccine-hesitancy chatbot is 1 of the early real-world uses of Label Sleuth, which built its dialogue act classifier mapping user utterances into categories such as greeting, query and concern.
Holds for: One reported deployment by ML-expert users; no accuracy numbers for that classifier are reported.
how can a labeling tool keep the suggestions coming fast when a big language model is slow to train?
does swapping an SVM for BERT only in later active learning iterations preserve F1 relative to fine-tuning BERT every iteration?
how do I keep model retraining fast enough for interactive labeling without giving up final classifier quality?
should I let a heavier transformer train on every labeling round, or start with something light and switch later?: Using a light SVM for active learning iterations 0-4 and BERT only for iterations 5-6 reaches F1 comparable to fine-tuning BERT at all 6 iterations, while running substantially faster. Each iteration adds 30 examples.
Holds for: F1 averaged over 1 target class from each of 5 datasets (20 Newsgroups, AG News, DBPedia, ISEAR, Yahoo! Answers) and 5 seeds, with gold labels for added examples.
which model does Label Sleuth train in the background if nobody changes any settings?
what is Label Sleuth's default classifier architecture and default active learning strategy?
how do I know which model and example-selection strategy I am getting out of the box in an interactive labeling system?
do I need to configure a model before I start labeling in Label Sleuth, or is the default fine?: Label Sleuth's default classifier is an ensemble of 2 SVM classifiers, one over bag-of-words and one over GloVe representations, paired with uncertainty sampling as the default active learning strategy.
Holds for: The default policy of the initial open-source release, chosen empirically for typical text classification use cases; other models, including GPU-backed large models, can be configured.
how many examples do I have to mark before a labeling tool starts giving predictions?
what labeled-positive threshold triggers initial classifier training and subsequent retraining in Label Sleuth?
how do I get a first model out of an interactive labeling session without deciding when to hit train?
do I have to tell Label Sleuth when to train, or will it start on its own once I have labeled enough?: Label Sleuth's default policy trains the first classifier once 20 positively labeled examples exist, then retrains after every 20 further labels, so the user never invokes training manually.
Holds for: The default policy as of the initial open-source release, chosen empirically rather than claimed optimal; advanced users can reconfigure the trigger.
if only a tiny fraction of my texts belong to the category, do I have to label all the ones that do not?
how does Label Sleuth obtain negative examples for a rare positive class without explicit negative annotation?
how do I train a binary classifier for a rare category when I have only marked the positive examples?
can I skip labeling negatives in Label Sleuth and still get a working classifier for a rare clause type?: Label Sleuth spares users from labeling negatives by automatically sampling unlabeled elements as weak negative examples until there are 2 labeled negatives for every labeled positive example.
Holds for: The default policy of the initial release, and only where the positive prior is low; when positives are not rare the feature must be disabled.
how can I tell whether a classifier is any good if I never built a test set?
how does Label Sleuth estimate classifier precision without a held-out labeled test set?
how do I check the quality of a classifier I built by labeling as I go, with no separate evaluation set?
how will I know when the Label Sleuth model is accurate enough to stop labeling?: Label Sleuth estimates classifier quality by sampling 50 examples predicted positive for the user to label, then reports precision and folds those 50 labels into the training set.
Holds for: Default sample size of 50 in the initial release, user-invoked, and precision only; recall estimation is impractical under a low positive prior.
can a labeling tool tell me when I have contradicted myself while marking texts?
what mechanisms surface suspect annotations during interactive labeling, such as cross-validated disagreement or contradicting labels on similar texts?
how do I find and fix inconsistent labels in a dataset I am annotating as I go?
will Label Sleuth flag my own labeling mistakes, or do I need a separate quality check?: Label Sleuth surfaces suspect labels 2 ways: cross-validated classifiers disagreeing with the user's own label on held-out elements, and pairs of semantically similar texts the user gave contradicting labels.
Holds for: Pair similarity is the distance between average GloVe embeddings in the initial implementation; the user reviews and corrects the ranked lists.
can a researcher plug their own model into an open-source labeling tool, and how much code does it take?
how are new classification models and active learning strategies integrated into the Label Sleuth architecture?
how do I add my own active learning strategy to an existing annotation platform instead of building one from scratch?
I want to benchmark my own model inside a real labeling loop, is Label Sleuth worth extending for that?: Developers extend Label Sleuth by implementing 1 or 2 functions to add a classification model or active learning strategy, and can configure the system to switch models or strategies as labeling progresses.
Holds for: Python Flask backend with a React frontend; GPU-backed large models supported; data access is in-memory plus local disk in the current implementation.
what problems are still unsolved about letting non-experts build their own text classifiers?
what open research questions does Label Sleuth identify around system policy, evaluation without a held-out test set, and zero-shot warm start?
what should I work on if I want to research interactive classifier building for users with no machine learning background?
if I adopt a no-code classifier builder like Label Sleuth today, which parts are still open research rather than solved?: Label Sleuth names 3 open research problems arising from interactive classifier building for non-technical users: choosing the system policy, evaluating models without a held-out test set, and warm-starting from zero-shot classification.
Holds for: Framing offered as an invitation to the NLP and HCI communities in 2022; only initial experiments on the policy question are reported, and no solutions to the other two.; Label Sleuth estimates classifier quality by sampling 50 examples predicted positive for the user to label, then reports precision and folds those 50 labels into the training set.
Holds for: Default sample size of 50 in the initial release, user-invoked, and precision only; recall estimation is impractical under a low positive prior.
can a no-code labeling tool handle more than two categories, or tasks like tagging names in text?
is Label Sleuth restricted to binary text classification, or does it support multi-class labeling and sequence tagging such as NER?
how do I choose a labeling tool when my task is entity extraction or multi-class rather than a yes-no decision?
my task is multi-class annotation, is Label Sleuth the wrong tool for me?: Label Sleuth is a free, open-source, no-code system in which a domain expert with no machine learning knowledge labels text and obtains a custom binary text classifier, starting from an unlabeled corpus.
Holds for: Released July 2022 under Apache 2.0; binary text classification of pre-split text elements only, with English as the default language.; Label Sleuth is the only one of 5 compared labeling tools that requires no technical expertise and gives ML guidance on label errors. It is also the only one of the 5 restricted to text classification.
Holds for: Prodigy, free and paid Label Studio, INCEpTION and Label Sleuth, chosen for similarity and popularity and assessed by the authors in 2022; a feature comparison rather than a usability study.

Claims and scope

Label Sleuth is a free, open-source, no-code system in which a domain expert with no machine learning knowledge labels text and obtains a custom binary text classifier, starting from an unlabeled corpus.
Scope: Released July 2022 under Apache 2.0; binary text classification of pre-split text elements only, with English as the default language.
Label Sleuth targets the gap between text classification techniques and non-technical practitioners such as lawyers, physicians and psychologists, who need custom classifiers but depend on ML experts to build them.
Scope: As of publication in 2022; comparable labeling tools with ML support target data scientists and developers.
Label Sleuth is the only one of 5 compared labeling tools that requires no technical expertise and gives ML guidance on label errors. It is also the only one of the 5 restricted to text classification. (Table 1)
Scope: Prodigy, free and paid Label Studio, INCEpTION and Label Sleuth, chosen for similarity and popularity and assessed by the authors in 2022; a feature comparison rather than a usability study.
A legal user built a Label Sleuth classifier for a category of high-risk contract clauses after 6 hours of work in the system. Reviewing only the highlighted clauses instead of entire contracts saved an estimated 80% of their time. (Section 2.3)
Scope: A single early user's self-reported experience and self-estimated time saving on a contract-review task, not a controlled measurement.
The VIRA COVID-19 vaccine-hesitancy chatbot is 1 of the early real-world uses of Label Sleuth, which built its dialogue act classifier mapping user utterances into categories such as greeting, query and concern. (Section 2.3)
Scope: One reported deployment by ML-expert users; no accuracy numbers for that classifier are reported.
Using a light SVM for active learning iterations 0-4 and BERT only for iterations 5-6 reaches F1 comparable to fine-tuning BERT at all 6 iterations, while running substantially faster. Each iteration adds 30 examples. (Figure 2)
Scope: F1 averaged over 1 target class from each of 5 datasets (20 Newsgroups, AG News, DBPedia, ISEAR, Yahoo! Answers) and 5 seeds, with gold labels for added examples.
Label Sleuth's default policy trains the first classifier once 20 positively labeled examples exist, then retrains after every 20 further labels, so the user never invokes training manually. (Appendix A)
Scope: The default policy as of the initial open-source release, chosen empirically rather than claimed optimal; advanced users can reconfigure the trigger.
Label Sleuth spares users from labeling negatives by automatically sampling unlabeled elements as weak negative examples until there are 2 labeled negatives for every labeled positive example. (Appendix A)
Scope: The default policy of the initial release, and only where the positive prior is low; when positives are not rare the feature must be disabled.
Label Sleuth estimates classifier quality by sampling 50 examples predicted positive for the user to label, then reports precision and folds those 50 labels into the training set. (Appendix A)
Scope: Default sample size of 50 in the initial release, user-invoked, and precision only; recall estimation is impractical under a low positive prior.
Label Sleuth surfaces suspect labels 2 ways: cross-validated classifiers disagreeing with the user's own label on held-out elements, and pairs of semantically similar texts the user gave contradicting labels. (Appendix B)
Scope: Pair similarity is the distance between average GloVe embeddings in the initial implementation; the user reviews and corrects the ranked lists.
Label Sleuth's default classifier is an ensemble of 2 SVM classifiers, one over bag-of-words and one over GloVe representations, paired with uncertainty sampling as the default active learning strategy. (Appendix A)
Scope: The default policy of the initial open-source release, chosen empirically for typical text classification use cases; other models, including GPU-backed large models, can be configured.
Developers extend Label Sleuth by implementing 1 or 2 functions to add a classification model or active learning strategy, and can configure the system to switch models or strategies as labeling progresses. (Section 4)
Scope: Python Flask backend with a React frontend; GPU-backed large models supported; data access is in-memory plus local disk in the current implementation.
Label Sleuth names 3 open research problems arising from interactive classifier building for non-technical users: choosing the system policy, evaluating models without a held-out test set, and warm-starting from zero-shot classification.
Scope: Framing offered as an invitation to the NLP and HCI communities in 2022; only initial experiments on the policy question are reported, and no solutions to the other two.

Common misreadings

The 80% time saving reported by a legal user of Label Sleuth is that user's own estimate on one contract-review task, not a measured average across users or tasks.
Label Sleuth builds binary classifiers only: a multi-label category such as Emotions requires creating a separate binary category per label, and mutually exclusive multi-class models must be trained outside the system from exported labels.
The SVM-then-BERT result compares a schedule that uses SVM for iterations 0-4 against BERT at every iteration; it does not show that SVM alone matches BERT.
Label Sleuth's automatic weak negatives assume positives are rare; where the positive prior is high the feature should be disabled and negatives labeled by hand.
Being the only compared tool needing no technical expertise is a feature comparison in Table 1, not evidence that non-experts label faster or more accurately with Label Sleuth.

Terminology in this paper

Policy: The bundle of Label Sleuth configuration choices — classification model, active learning strategy, training-set selection, and the criterion that triggers training a new model — that together shape the classifier-building flow.
Precision Evaluation: A Label Sleuth procedure that samples examples predicted positive by the current classifier, asks the user to label them, and uses those labels both to estimate precision and to extend the training set.
Weak negative examples: Unlabeled text elements automatically added to the training set as negatives, on the assumption that positives are rare, so that the user does not have to label negatives explicitly.
Label Next list: A panel of unlabeled examples chosen by an active learning strategy that Label Sleuth presents as the examples most beneficial to label in the current iteration.
Domain expert: A practitioner with deep knowledge of the target subject matter — for example a lawyer, physician or psychologist — but typically without coding skills or machine learning knowledge.

How to cite

@inproceedings{DBLP:conf/emnlp/ShnarchHGDKCCEZW22,author       = {Eyal Shnarch and
                  Alon Halfon and
                  Ariel Gera and
                  Marina Danilevsky and
                  Yannis Katsis and
                  Leshem Choshen and
                  Mart{\'{\i}}n Santill{\'{a}}n Cooper and
                  Dina Epelboim and
                  Zheng Zhang and
                  Dakuo Wang},
  editor       = {Wanxiang Che and
                  Ekaterina Shutova},
  title        = {Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours},
  booktitle    = {Proceedings of the The 2022 Conference on Empirical Methods in Natural
                  Language Processing, {EMNLP} 2022 - System Demonstrations, Abu Dhabi,
                  UAE, December 7-11, 2022},
  pages        = {159--168},
  publisher    = {Association for Computational Linguistics},
  year         = {2022},
  url          = {https://doi.org/10.18653/v1/2022.emnlp-demos.16},
  doi          = {10.18653/V1/2022.EMNLP-DEMOS.16},
  timestamp    = {Mon, 22 Jul 2024 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/emnlp/ShnarchHGDKCCEZW22.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.