Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI

a Python library for modular, shareable prompt construction and evaluation of generative language models

Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman, Ofir Arviv, Matan Orbach, Shachar Don-Yehiya, Dafna Sheinwald, Ariel Gera, Leshem Choshen, Michal Shmueli-Scheuer, Yoav Katz · NAACL 2024 (Demo) · 2024

In one sentence

Unitxt decomposes LLM data preparation and evaluation into shareable components — resources, task, template, format and extensions — combined by a declarative "recipe" that produces a HuggingFace dataset of ready-to-send prompts and metric-ready targets.

Abstract

In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution.Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt

Questions this paper answers

Is there a library that turns a dataset plus a prompt wording into ready-to-score model inputs?
Which framework decomposes prompt-level data preparation into shareable cards, templates, formats and system prompts?
How do I hand a colleague my entire prompt-building and scoring setup as one shareable specification?
Should I adopt Unitxt instead of maintaining my own prompt formatting and metric code?
A Unitxt recipe specifies a dataset card, template, system prompt, format and number of in-context demonstrations in one declarative string, and the open catalog's ingredients combine into more than 100K such recipes. Loading it yields a dataset whose every instance already holds fully prepared model-input text and a metric-ready target.
Holds for: Textual (multilingual) data for generative language models; loading is via unitxt.load_dataset, and the recipe covers loading, verbalization, formatting and metric configuration. As described in the January 2024 release.
Unitxt is a library for prompt-level data preparation and evaluation that separates system prompts, task instructions, verbalizations and model-specific formats into independently shareable components. The authors argue that decomposition was missing from earlier pipeline frameworks.
Holds for: Positioning as of the January 2024 release, against Datasets/Evaluate, Tasksource, PromptSource, SeqIO and the pipelines inside OpenCompass, HELM and LM-eval-harness; a comparison of design properties, not a benchmark.
What paper should I read first about making language-model evaluation pipelines reproducible?
Which work introduced a modular, catalog-based library for prompt-level data processing and LLM evaluation?
Where do I start reading if I want to build a reproducible LLM evaluation pipeline?
I need a citable reference for reproducible prompt-level evaluation tooling — which one covers the modular pipeline design?
Unitxt is a library for prompt-level data preparation and evaluation that separates system prompts, task instructions, verbalizations and model-specific formats into independently shareable components. The authors argue that decomposition was missing from earlier pipeline frameworks.
Holds for: Positioning as of the January 2024 release, against Datasets/Evaluate, Tasksource, PromptSource, SeqIO and the pipelines inside OpenCompass, HELM and LM-eval-harness; a comparison of design properties, not a benchmark.
Unitxt is usable as a standalone data-preparation and evaluation layer rather than a full evaluation harness, which is what lets it be embedded inside other frameworks including LM-eval-harness instead of replacing them.
Holds for: Contrast drawn with evaluation frameworks whose pipelines are coupled to their inference engine; reflects the design as described at NAACL 2024 demo track.
How do you write down which wording, instruction and number of examples a prompt should use?
How is a Unitxt recipe declared, and are templates bound to a single dataset or reusable across tasks?
How do I swap the template, system prompt and shot count without editing my data-processing code?
Can I change prompt wording and in-context example count from a config string rather than rewriting my loader?
A Unitxt recipe specifies a dataset card, template, system prompt, format and number of in-context demonstrations in one declarative string, and the open catalog's ingredients combine into more than 100K such recipes. Loading it yields a dataset whose every instance already holds fully prepared model-input text and a metric-ready target.
Holds for: Textual (multilingual) data for generative language models; loading is via unitxt.load_dataset, and the recipe covers loading, verbalization, formatting and metric configuration. As described in the January 2024 release.
In Unitxt, templates, datasets and tasks are not exclusively tied: one task can use many templates and one template can serve many datasets. In PromptSource, by contrast, each prompt is bound to a single dataset.
Holds for: Unitxt tasks whose interface fixes input and output field names and types, so any metric accepting those fields applies; templates and datasets drawn from the open catalog.
How many different prompt setups can you build from the ready-made pieces in the Unitxt catalog?
How many pipeline configurations does the Unitxt Catalog yield by combining cards, tasks, templates, formats and extensions?
How many dataset, task and prompt-format combinations can I get without authoring anything myself?
Is the Unitxt catalog big enough that my task and prompt format are probably already in it?
The open-source Unitxt Catalog supports more than 100K possible pipeline configurations, obtained by mixing and matching cards, tasks, templates, formats and extensions.
Holds for: Combinatorial recipe configurations in the open catalog as of January 2024, not distinct datasets; private catalogs can add proprietary artifacts.
How much work is it to make an existing evaluation harness use Unitxt data and metrics?
What integration cost does adding Unitxt recipes and metrics to LM-eval-harness incur in lines of code?
How do I add a new task and a new metric to LM-eval-harness without rewriting its data loading?
I already run LM-eval-harness — how disruptive is wiring Unitxt into it?
Integrating Unitxt into LM-eval-harness required no code changes on the data side and about 30 lines of code to register Unitxt metrics. A Unitxt recipe becomes an LM-eval-harness task through a one-line change in the task YAML.
Holds for: LM-eval-harness as of the version used in January 2024; the data-side integration works because Unitxt recipes load as standard HuggingFace datasets, and the existing LM-eval-harness API is preserved.
The Unitxt data-preparation pipeline emits a HuggingFace dataset that can be saved or pushed to the hub, so Unitxt drops into existing HuggingFace-based codebases without rewriting downstream code. Integrating it into LM-eval-harness took about 30 lines of code, with one line changed per recipe in a yaml.
Holds for: The data-preparation pipeline output format; raw data and metrics arrive through Unitxt resource APIs covering HuggingFace Hub, local files and cloud storage.
Does the prepared prompt data come out in a format ordinary training code can read?
Does the Unitxt preparation pipeline emit a HuggingFace dataset that can be serialized or pushed to the hub?
How do I get verbalized prompts into my existing HuggingFace training loop without writing a converter?
Can I use Unitxt-prepared prompts in my current HuggingFace training and inference code as-is?
The Unitxt data-preparation pipeline emits a HuggingFace dataset that can be saved or pushed to the hub, so Unitxt drops into existing HuggingFace-based codebases without rewriting downstream code. Integrating it into LM-eval-harness took about 30 lines of code, with one line changed per recipe in a yaml.
Holds for: The data-preparation pipeline output format; raw data and metrics arrive through Unitxt resource APIs covering HuggingFace Hub, local files and cloud storage.
Can the same prompt wording be reused across several different datasets?
Are Unitxt templates decoupled from datasets and tasks, unlike PromptSource's per-dataset prompt binding?
How do I write one prompt template once and apply it to every dataset of the same task?
If I author 20 prompt templates, will I have to duplicate them per dataset?
In Unitxt, templates, datasets and tasks are not exclusively tied: one task can use many templates and one template can serve many datasets. In PromptSource, by contrast, each prompt is bound to a single dataset.
Holds for: Unitxt tasks whose interface fixes input and output field names and types, so any metric accepting those fields applies; templates and datasets drawn from the open catalog.
How does free-text model output get turned back into something a metric can actually score?
Where does de-verbalization and type casting of generative predictions happen in a Unitxt pipeline?
How do I post-process a model's answer like "two and a half" into a number my similarity metric can use?
Do I have to write my own output parser for each task, or does Unitxt handle prediction post-processing?
Unitxt templates own de-verbalization as well as verbalization, first standardizing model output text by taking the first non-empty line, lowercasing and stripping whitespace. The output is then cast to the task's type, turning "2.43" or "two and a half" into a float for sentence similarity.
Holds for: Generative model predictions evaluated through the Unitxt evaluation pipeline; the task-specific cast depends on the verbalization the template defines and on its in-context demonstrations.
Do the evaluation scores come with any measure of uncertainty?
Does Unitxt report bootstrap confidence intervals alongside metric scores?
How do I get error bars on a benchmark score so I know whether two models really differ?
Can I report confidence intervals with my Unitxt evaluation numbers without coding a bootstrap myself?
Unitxt metrics report confidence intervals alongside scores using a built-in statistical bootstrap mechanism.
Holds for: Metrics run through the Unitxt evaluation pipeline; the authors state that coverage of generative-task metrics still needs improvement.
Can I deliberately add noise or typos to prompts to test how sensitive a model is?
Can Unitxt Extensions inject augmentations such as spelling noise, synonym replacement or demonstration label noising at arbitrary points in the pipeline?
How do I perturb prompts and corrupt in-context example labels to probe model robustness?
Do I need custom code to augment or label-noise my prompt data, or is it built in?
Unitxt Extensions insert augmentations at any point between two operators in the data-preparation pipeline, including random whitespace, spelling mistakes, synonym replacement, and label-noising that randomizes demonstration labels.
Holds for: Extensions are independent units reusable across datasets, tasks, templates and formats, and custom ones can be added to the Catalog; back-translation for multilinguality is named as not yet supported.
Is there a way to see what a prompt will look like before running a whole evaluation?
Does Unitxt ship an exploration UI that previews verbalized prompts and exports the equivalent recipe code?
How do I preview a prompt with a chosen template and shot count, and try it on a model, before writing code?
Can I browse and test prompt configurations in a UI first and only then copy out the code?
The Unitxt exploration UI walks a user from task to dataset to template and previews the generated prompt with optional system prompt, response schema and number of shots. It also runs the example on a preset model such as flan-t5-base and exports the equivalent code.
Holds for: Preview and single-example execution on pre-set models; the UI targets recipe creation and catalog exploration rather than full-scale benchmark runs.
Is the shared library for preparing prompts and evaluating language models actually used by people, or just described in a paper?
What evidence of production adoption exists for Unitxt across LLM evaluation and training workloads?
How do I tell whether Unitxt is mature enough to depend on for both training and evaluation work?
Is Unitxt used in real industrial LLM pipelines, and for which kinds of tasks?
Unitxt was already in use as a core LLM utility by multiple IBM teams for both evaluation and training, across classification, extraction, summarization, generation, question answering, code and bias tasks.
Holds for: Internal adoption reported by the authors as of January 2024, without usage counts or per-team detail; open-source community adoption was at an early stage.
Does using Unitxt mean giving up the evaluation harness I already run?
Is Unitxt positioned as a standalone data-preparation and evaluation layer rather than a replacement evaluation harness?
How do I add modular prompt preparation to my existing evaluation framework instead of switching frameworks?
Should I replace my current LLM evaluation harness with Unitxt, or embed Unitxt inside it?
Unitxt is usable as a standalone data-preparation and evaluation layer rather than a full evaluation harness, which is what lets it be embedded inside other frameworks including LM-eval-harness instead of replacing them.
Holds for: Contrast drawn with evaluation frameworks whose pipelines are coupled to their inference engine; reflects the design as described at NAACL 2024 demo track.
Integrating Unitxt into LM-eval-harness required no code changes on the data side and about 30 lines of code to register Unitxt metrics. A Unitxt recipe becomes an LM-eval-harness task through a one-line change in the task YAML.
Holds for: LM-eval-harness as of the version used in January 2024; the data-side integration works because Unitxt recipes load as standard HuggingFace datasets, and the existing LM-eval-harness API is preserved.

Claims and scope

Common misreadings

Terminology in this paper

Recipe
A declarative Unitxt specification naming the resources, task, template, format and extensions for a data pipeline, loadable as a dataset in one call.
Data-Task Card
A Unitxt artifact describing where raw data is loaded from and how its fields, values and splits are standardized into a task's input and output interface.
Task (in Unitxt)
An NLP task defined by a fixed interface — named and typed input and output fields — plus the evaluation metrics for it, so that any metric accepting those fields can be applied.
Template (in Unitxt)
A component that verbalizes standardized input and target fields into text and also de-verbalizes model predictions back into the task's expected type.
Format (in Unitxt)
A component holding formatting requirements independent of the data or task: system prompts, special tokens, user/agent prefixes and the placement of in-context demonstrations.
Extensions (in Unitxt)
Optional operators insertable anywhere in the data-preparation pipeline, such as input augmentation or label-noising of in-context demonstrations.
Unitxt Catalog
The shared store of Unitxt artifacts — recipes, data-task cards, templates, operators, formats and metrics — with an open version plus optional private catalogs for proprietary artifacts.

How to cite

@inproceedings{bandel2024unitxt,title = "Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative {AI}",
    author = "Bandel, Elron  and
      Perlitz, Yotam  and
      Venezian, Elad  and
      Friedman, Roni  and
      Arviv, Ofir  and
      Orbach, Matan  and
      Don-Yehiya, Shachar  and
      Sheinwald, Dafna  and
      Gera, Ariel  and
      Choshen, Leshem  and
      Shmueli-Scheuer, Michal  and
      Katz, Yoav",
    editor = "Chang, Kai-Wei  and
      Lee, Annie  and
      Rajani, Nazneen",
    booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations)",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.naacl-demo.21",
    doi = "10.18653/v1/2024.naacl-demo.21",
    pages = "207--215",
    abstract = "In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution.Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt",
}

References

See the full reference list in the paper.