Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability

Afra Feyza Akyürek; Ekin Akyürek; Leshem Choshen; Derry Wijaya; Jacob Andreas

doi:10.48550/arXiv.2401.08574

Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability

fine-tuning a language model on the implications of its own text that survive a self-consistency check

Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, Jacob Andreas · Findings of ACL 2024 · 2024

In one sentence

Deductive Closure Training (DCT) prompts a language model to generate the implications and contradictions of seed text, keeps the most probable logically consistent subset, and fine-tunes on it — turning inference-time reasoning into training-time supervision for factuality and model updating.

Abstract

While language models (LMs) can sometimes generate factually correct text and estimate truth values of individual claims, these generally do not reflect a globally coherent, manipulable model of the world. As a consequence, current LMs also generate incorrect or nonsensical content, and are difficult to edit and bring up to date. We present a method called Deductive Closure Training (DCT) that uses LMs themselves to identify implications of (and contradictions within) the text that they generate, yielding an efficient self-supervised procedure for improving LM factuality. Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct. Given seed documents from a trusted source, DCT provides a tool for supervised model updating; if seed documents are sampled from the LM itself, DCT enables fully unsupervised fine-tuning for improved coherence and accuracy. Across the CREAK, MQUaKE, and Reversal Curse datasets, supervised DCT improves LM fact verification and text generation accuracy by 3-26%; on CREAK fully unsupervised DCT improves verification accuracy by 12%. These results show that LMs' reasoning capabilities during inference can be leveraged during training to improve their reliability.

Questions this paper answers

can a language model get more accurate about facts by training on statements it wrote itself, with no human labels?
how much does unsupervised fine-tuning on self-generated, consistency-filtered statements improve fact-verification accuracy on CREAK?
how do I raise a 7B model's factual accuracy when I have no labelled fact-checking data?
is self-generated training data enough to beat few-shot prompting for fact verification on my own model?: Unsupervised Deductive Closure Training raises Llama-2-7B fact-verification accuracy on the CREAK validation set from 71.7% with few-shot prompting to 83.5%, using no labelled data at all.
Holds for: Llama-2-7B on CREAK validation, averaged over 3 seeds; seeds sampled from the model itself at temperature 0.9, LoRA fine-tuning for 30 epochs, no held-out dev set.; Dropping the logical consistency-evaluation step from unsupervised DCT lowers CREAK accuracy from 83.5% to 77.5%, below the 80.0% obtained by fine-tuning on seed statements alone.
Holds for: Llama-2-7B on CREAK validation, 3 seeds, same 586 generated documents in both the full and ablated conditions.
if a model trains on text it generated itself, does anything filter out the false parts?
what is the contribution of the logical consistency-evaluation step in Deductive Closure Training, relative to fine-tuning on seed statements alone?
do I need to check generated implications for consistency before fine-tuning on them, or can I train on all of them?
can I skip the consistency-checking stage and still come out ahead of just fine-tuning on the seed statements?: Dropping the logical consistency-evaluation step from unsupervised DCT lowers CREAK accuracy from 83.5% to 77.5%, below the 80.0% obtained by fine-tuning on seed statements alone.
Holds for: Llama-2-7B on CREAK validation, 3 seeds, same 586 generated documents in both the full and ablated conditions.; A double-checking pass in which the model re-verifies each generated implication or contradiction raises their validity rate from 76% to 82%. It also discards about 2/3 of generations, cutting 100 annotated statements to 33.
Holds for: Manual annotation of 100 implications and contradictions generated by Llama-2-7B in the supervised CREAK setting; precision is bought at a large loss of training data volume.
if I have only a couple dozen labelled examples, is it worth combining them with machine-generated training data?
does pooling a small labelled set with unsupervised Deductive Closure Training data beat both supervised fine-tuning and the unsupervised setting on CREAK?
how do I make 20 hand-labelled claims go further for training a fact-verification model?
I have about 20 annotated claims and lots of unlabelled text — should I train on both together?: Pooling 20 labelled CREAK claims with unsupervised DCT data reaches 84.9% accuracy, above both supervised fine-tuning on the 20 claims (77.2%) and unsupervised DCT alone (83.5%).
Holds for: Llama-2-7B, CREAK validation, 3 seeds; supervised seeds are 20 training claims filtered to those labelled true, with supervised runs trained for 60 epochs.
does letting a model see the unlabelled test questions during training help it answer them correctly?
how does transductive Deductive Closure Training on unlabelled CREAK validation claims compare with inference-time implication-graph reasoning?
I have the unlabelled evaluation claims in hand — should I fine-tune on their implications or run consistency reasoning at inference time?
is it better to bake consistency reasoning into the weights or pay for it at inference on every query?: With unlabelled CREAK validation claims used as seed documents, DCT on related claims plus double-checking reaches 85.5% accuracy. The inference-time Graph-Inference baseline, which reasons over implication graphs without fine-tuning, reaches 77.7%.
Holds for: Llama-2-7B, all 1,371 unlabelled CREAK validation claims used as seeds, 1 training epoch; requires access to test-set inputs, so it is not a strict train/test split.
can training a model on its own statements stop it from agreeing with two claims that contradict each other?
does self-training on generated contradictions reduce the rate of contradictory statement pairs both labelled true, and what happens to accuracy?
how do I stop a model from labelling a claim and its negation as both true?
if I fine-tune for logical coherence, do I lose accuracy to get consistency?: On the 250-pair CREAK contrast set, unsupervised DCT cuts the fraction of contradictory statement pairs labelled both-true from 34.4% to 19.2% while raising accuracy from 63.2% to 73.0%; the contradiction-only variant drives both-true down furthest, to 12.0%.
Holds for: Llama-2-7B, CREAK contrast set of 250 lexically similar pairs with opposite truth values; coherence measured only as the both-true rate, not as full logical consistency.
after you correct a fact in a language model, why does it still get wrong the questions that depend on that fact?
how does Deductive Closure Training compare with MEMIT, MeLLo and fine-tuning on edits for multi-hop counterfactual accuracy on MQuAKE?
how do I edit thousands of facts in a model so that multi-hop answers depending on them also change?
which knowledge-editing approach should I pick if my questions require chaining two edited facts together?: On the MQuAKE counterfactual multi-hop subset, DCT with combined correlative and standard implications reaches 35.6% accuracy at 20 edits and 15.4% at 1,000 edits. The corresponding numbers are 18.3% and 11.4% for MeLLo, 11.7% and 0.6% for MEMIT, and 9.0% and 4.1% for fine-tuning on edits.
Holds for: Llama-2-7B, MQuAKE-CF multi-hop questions only, 3 seeds except the 1,000-edit setting which is run once; each edit yields 3 test questions and edits are converted to questions by prompting before fine-tuning.; Prompting the model to state related background facts before generating implications (correlative implications) beats plain implications on MQuAKE at every edit count, for example 41.7% versus 20.0% at 10 edits.
Holds for: Llama-2-7B on MQuAKE-CF multi-hop questions, edit counts 10–1,000; the advantage of correlative over plain implications is systematic but the combined set is best on average.
does asking a model to recall related background facts first make the statements it generates more useful for training?
do correlative implications, generated after eliciting related background knowledge, outperform plain implications on MQuAKE and introduce more new information?
how should I prompt a model to generate training statements that carry consequences of an edited fact rather than restating it?
is it worth an extra prompting step to get background facts before generating implications of an edit?: Prompting the model to state related background facts before generating implications (correlative implications) beats plain implications on MQuAKE at every edit count, for example 41.7% versus 20.0% at 10 edits.
Holds for: Llama-2-7B on MQuAKE-CF multi-hop questions, edit counts 10–1,000; the advantage of correlative over plain implications is systematic but the combined set is best on average.; Correlative implications on MQuAKE incorporate new information supplied by the model 33% of the time versus 24% for plain implications, about 50% more. Only 30–36% of generations in either condition are strict logical implications.
Holds for: Manual annotation of roughly 350 generations from Llama-2-7B across experiments; single-annotator qualitative analysis, not a validated protocol.
if a model learns that A is B, can training fix the fact that it cannot answer the question the other way round?
does fine-tuning on generated implications mitigate the reversal curse, and at what cost to forward-direction exact-match accuracy?
how do I teach a model a relation so it answers questions in both directions rather than only the one it was trained on?
will training on generated implications fix reversed-order recall without degrading the direction that already worked?: On the Reversal Curse child-to-parent split, DCT raises exact-match accuracy on reversed questions from 2.2% to 48.3% while keeping same-direction accuracy at 87.9% versus 95.3% for ordinary fine-tuning.
Holds for: Llama-2-7B, 1,000 child-to-parent evaluation examples; gains are far smaller on person-to-description (3.7% to 10.7%) and description-to-person (3.0% to 15.7%), 300 examples each.
is there any proof that training a model on text it produced itself should make it more accurate rather than reinforce its mistakes?
under what assumptions on seed-pair correctness and conditional probability gain is Deductive Closure Training guaranteed to increase the probability of the correct answer?
is there a formal argument I can rely on before I fine-tune a model on its own generations?: Deductive Closure Training is proved to strictly increase the probability a language model assigns to the correct answer for a question. The proof assumes self-generated seed question-answer pairs are correct with probability at least p*, and that conditioning on a correct related pair raises correct-answer probability by at least 1/p*.
Holds for: A minimal unsupervised variant of DCT with a single implication per seed, no consistency check, and training to convergence on arbitrarily many seeds; the two assumptions are not empirically verified.
if a model re-checks the statements it generated, do they get more reliable, and how much gets thrown away?
how much does a double-checking verification pass raise the validity rate of generated implications and contradictions, and what fraction is discarded?
how do I filter self-generated implications for validity without discarding most of my training set?
is a second verification pass over generated statements worth losing most of the data I generated?: A double-checking pass in which the model re-verifies each generated implication or contradiction raises their validity rate from 76% to 82%. It also discards about 2/3 of generations, cutting 100 annotated statements to 33.
Holds for: Manual annotation of 100 implications and contradictions generated by Llama-2-7B in the supervised CREAK setting; precision is bought at a large loss of training data volume.
which work first turned checking a claim against its logical consequences into a way of training a model rather than a way of answering at inference time?
which paper reframes inference-time consistency reasoning over implications and contradictions as training-time supervision for factuality?
where should I start reading about improving a language model's factuality without labelled data?: Deductive Closure Training reframes inference-time consistency reasoning over implications and contradictions of a claim as a source of training-time supervision. It gives one framework for both unsupervised factuality improvement and supervised model editing.
Holds for: Positioned as of publication at ACL Findings 2024 relative to inference-time belief-graph methods and to concurrent self-training work that ranks or filters self-generated data by model-assigned factuality labels rather than extrapolating implications.; Deductive Closure Training is an entry point for readers asking whether model editing propagates to a fact's consequences rather than to the edited sentence alone. It is evaluated on multi-hop question answering and reversal-style generalization.
Holds for: Demonstrated only on English-language datasets (CREAK, MQuAKE-CF, Reversal Curse) with a single Llama-2-7B model, and only with one layer of implications rather than a deeper graph of relations.
what should I read about whether correcting a fact in a language model also corrects everything that follows from it?
which work evaluates whether knowledge edits propagate to multi-hop questions and to reversed relation directions?
how do I find out whether an edit to one fact in a model generalizes to its consequences before I rely on it?: Deductive Closure Training is an entry point for readers asking whether model editing propagates to a fact's consequences rather than to the edited sentence alone. It is evaluated on multi-hop question answering and reversal-style generalization.
Holds for: Demonstrated only on English-language datasets (CREAK, MQuAKE-CF, Reversal Curse) with a single Llama-2-7B model, and only with one layer of implications rather than a deeper graph of relations.; On the MQuAKE counterfactual multi-hop subset, DCT with combined correlative and standard implications reaches 35.6% accuracy at 20 edits and 15.4% at 1,000 edits. The corresponding numbers are 18.3% and 11.4% for MeLLo, 11.7% and 0.6% for MEMIT, and 9.0% and 4.1% for fine-tuning on edits.
Holds for: Llama-2-7B, MQuAKE-CF multi-hop questions only, 3 seeds except the 1,000-edit setting which is run once; each edit yields 3 test questions and edits are converted to questions by prompting before fine-tuning.

Claims and scope

Unsupervised Deductive Closure Training raises Llama-2-7B fact-verification accuracy on the CREAK validation set from 71.7% with few-shot prompting to 83.5%, using no labelled data at all. (Table 1)
Scope: Llama-2-7B on CREAK validation, averaged over 3 seeds; seeds sampled from the model itself at temperature 0.9, LoRA fine-tuning for 30 epochs, no held-out dev set.
Dropping the logical consistency-evaluation step from unsupervised DCT lowers CREAK accuracy from 83.5% to 77.5%, below the 80.0% obtained by fine-tuning on seed statements alone. (Table 1)
Scope: Llama-2-7B on CREAK validation, 3 seeds, same 586 generated documents in both the full and ablated conditions.
Pooling 20 labelled CREAK claims with unsupervised DCT data reaches 84.9% accuracy, above both supervised fine-tuning on the 20 claims (77.2%) and unsupervised DCT alone (83.5%). (Table 1)
Scope: Llama-2-7B, CREAK validation, 3 seeds; supervised seeds are 20 training claims filtered to those labelled true, with supervised runs trained for 60 epochs.
With unlabelled CREAK validation claims used as seed documents, DCT on related claims plus double-checking reaches 85.5% accuracy. The inference-time Graph-Inference baseline, which reasons over implication graphs without fine-tuning, reaches 77.7%. (Table 1)
Scope: Llama-2-7B, all 1,371 unlabelled CREAK validation claims used as seeds, 1 training epoch; requires access to test-set inputs, so it is not a strict train/test split.
On the 250-pair CREAK contrast set, unsupervised DCT cuts the fraction of contradictory statement pairs labelled both-true from 34.4% to 19.2% while raising accuracy from 63.2% to 73.0%; the contradiction-only variant drives both-true down furthest, to 12.0%. (Table 2)
Scope: Llama-2-7B, CREAK contrast set of 250 lexically similar pairs with opposite truth values; coherence measured only as the both-true rate, not as full logical consistency.
On the MQuAKE counterfactual multi-hop subset, DCT with combined correlative and standard implications reaches 35.6% accuracy at 20 edits and 15.4% at 1,000 edits. The corresponding numbers are 18.3% and 11.4% for MeLLo, 11.7% and 0.6% for MEMIT, and 9.0% and 4.1% for fine-tuning on edits. (Table 3)
Scope: Llama-2-7B, MQuAKE-CF multi-hop questions only, 3 seeds except the 1,000-edit setting which is run once; each edit yields 3 test questions and edits are converted to questions by prompting before fine-tuning.
Prompting the model to state related background facts before generating implications (correlative implications) beats plain implications on MQuAKE at every edit count, for example 41.7% versus 20.0% at 10 edits. (Table 3)
Scope: Llama-2-7B on MQuAKE-CF multi-hop questions, edit counts 10–1,000; the advantage of correlative over plain implications is systematic but the combined set is best on average.
Correlative implications on MQuAKE incorporate new information supplied by the model 33% of the time versus 24% for plain implications, about 50% more. Only 30–36% of generations in either condition are strict logical implications. (Appendix C)
Scope: Manual annotation of roughly 350 generations from Llama-2-7B across experiments; single-annotator qualitative analysis, not a validated protocol.
On the Reversal Curse child-to-parent split, DCT raises exact-match accuracy on reversed questions from 2.2% to 48.3% while keeping same-direction accuracy at 87.9% versus 95.3% for ordinary fine-tuning. (Table 4)
Scope: Llama-2-7B, 1,000 child-to-parent evaluation examples; gains are far smaller on person-to-description (3.7% to 10.7%) and description-to-person (3.0% to 15.7%), 300 examples each.
A double-checking pass in which the model re-verifies each generated implication or contradiction raises their validity rate from 76% to 82%. It also discards about 2/3 of generations, cutting 100 annotated statements to 33. (Appendix C)
Scope: Manual annotation of 100 implications and contradictions generated by Llama-2-7B in the supervised CREAK setting; precision is bought at a large loss of training data volume.
Deductive Closure Training is proved to strictly increase the probability a language model assigns to the correct answer for a question. The proof assumes self-generated seed question-answer pairs are correct with probability at least p*, and that conditioning on a correct related pair raises correct-answer probability by at least 1/p*. (Section 4 and Appendix E)
Scope: A minimal unsupervised variant of DCT with a single implication per seed, no consistency check, and training to convergence on arbitrarily many seeds; the two assumptions are not empirically verified.
Deductive Closure Training reframes inference-time consistency reasoning over implications and contradictions of a claim as a source of training-time supervision. It gives one framework for both unsupervised factuality improvement and supervised model editing. (Section 2)
Scope: Positioned as of publication at ACL Findings 2024 relative to inference-time belief-graph methods and to concurrent self-training work that ranks or filters self-generated data by model-assigned factuality labels rather than extrapolating implications.
Deductive Closure Training is an entry point for readers asking whether model editing propagates to a fact's consequences rather than to the edited sentence alone. It is evaluated on multi-hop question answering and reversal-style generalization. (Section 5)
Scope: Demonstrated only on English-language datasets (CREAK, MQuAKE-CF, Reversal Curse) with a single Llama-2-7B model, and only with one layer of implications rather than a deeper graph of relations.

Common misreadings

Deductive Closure Training does not solve the Reversal Curse: with the same general-purpose implication prompt, reversed-direction accuracy on the person-to-description split rises only from 3.7% to 10.7%, and the paper describes the challenge as far from solved.
The accuracy gains are not simply the effect of fine-tuning on more self-generated text: removing the consistency-evaluation step drops CREAK accuracy from 83.5% to 77.5% with the identical 586 generated documents.
Most generated statements are not strict deductions; manual annotation found only 30–36% of MQuAKE generations to be strict logical implications, with the rest plausible-but-not-entailed text.
The transductive CREAK result of 85.5% is not comparable to a standard train/test result, because it uses the unlabelled validation claims themselves as seed documents.
The formal guarantee is not a general proof that self-training improves language models: it holds for a minimal single-implication variant without the consistency check, and only when two stated assumptions about seed correctness and prompt benefit are satisfied.
All results come from a single model, Llama-2-7B, and English-only datasets, so the reported gains are not evidence about larger models or other languages.

Terminology in this paper

Deductive closure: The complete set of statements that follow from a given collection of statements; a language model is deductively closed if it assigns high probability to a logically coherent set of assertions and to all their implications.
Correlative implications: Implications generated after first prompting the model to list related background facts about the entities in a seed claim, so that the resulting statements combine the new information with the model's existing knowledge.
Consistency evaluation: Selecting, among all assignments of True/False to a seed document and its generated implications and contradictions, the most probable assignment that respects every implication and contradiction.
Transductive DCT: Applying Deductive Closure Training with unlabelled inputs from the evaluation set used as seed documents, so the model is updated with access to the test claims but not their labels.
Double-checking: A filtering pass in which the language model is re-prompted to verify that a seed statement really does entail or contradict each generated statement, discarding those it does not confirm.

How to cite

@inproceedings{DBLP:conf/acl/AkyurekACWA24,author       = {Afra Feyza Aky{\"{u}}rek and
                  Ekin Aky{\"{u}}rek and
                  Leshem Choshen and
                  Derry Wijaya and
                  Jacob Andreas},
  editor       = {Lun{-}Wei Ku and
                  Andre Martins and
                  Vivek Srikumar},
  title        = {Deductive Closure Training of Language Models for Coherence, Accuracy,
                  and Updatability},
  booktitle    = {Findings of the Association for Computational Linguistics, {ACL} 2024,
                  Bangkok, Thailand and virtual meeting, August 11-16, 2024},
  pages        = {9802--9818},
  publisher    = {Association for Computational Linguistics},
  year         = {2024},
  url          = {https://aclanthology.org/2024.findings-acl.584},
  timestamp    = {Tue, 27 Aug 2024 17:38:11 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/AkyurekACWA24.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.