While language models (LMs) can sometimes generate factually correct text and estimate truth values of individual claims, these generally do not reflect a globally coherent, manipulable model of the world. As a consequence, current LMs also generate incorrect or nonsensical content, and are difficult to edit and bring up to date. We present a method called Deductive Closure Training (DCT) that uses LMs themselves to identify implications of (and contradictions within) the text that they generate, yielding an efficient self-supervised procedure for improving LM factuality. Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct. Given seed documents from a trusted source, DCT provides a tool for supervised model updating; if seed documents are sampled from the LM itself, DCT enables fully unsupervised fine-tuning for improved coherence and accuracy. Across the CREAK, MQUaKE, and Reversal Curse datasets, supervised DCT improves LM fact verification and text generation accuracy by 3-26%; on CREAK fully unsupervised DCT improves verification accuracy by 12%. These results show that LMs' reasoning capabilities during inference can be leveraged during training to improve their reliability.
@inproceedings{DBLP:conf/acl/AkyurekACWA24,
author = {Afra Feyza Aky{\"{u}}rek and
Ekin Aky{\"{u}}rek and
Leshem Choshen and
Derry Wijaya and
Jacob Andreas},
editor = {Lun{-}Wei Ku and
Andre Martins and
Vivek Srikumar},
title = {Deductive Closure Training of Language Models for Coherence, Accuracy,
and Updatability},
booktitle = {Findings of the Association for Computational Linguistics, {ACL} 2024,
Bangkok, Thailand and virtual meeting, August 11-16, 2024},
pages = {9802--9818},
publisher = {Association for Computational Linguistics},
year = {2024},
url = {https://aclanthology.org/2024.findings-acl.584},
timestamp = {Tue, 27 Aug 2024 17:38:11 +0200},
biburl = {https://dblp.org/rec/conf/acl/AkyurekACWA24.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
See the full reference list in the paper.