# The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models

Authors: Adithya Bhaskar, Dan Friedman, Danqi Chen
Venue: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2024)

## Abstract
Prior work has found that pretrained language models (LMs) fine-tuned with different random seeds can achieve similar in-domain performance but generalize differently on tests of syntactic generalization. In this work, we show that, even within a single model, we can find multiple subnetworks that perform similarly in-domain, but generalize vastly differently. To better understand these phenomena, we investigate if they can be understood in terms of {``}competing subnetworks{\''}: the model initially represents a variety of distinct algorithms, corresponding to different subnetworks, and generalization occurs when it ultimately converges to one. This explanation has been used to account for generalization in simple algorithmic tasks ({``}grokking{\''}). Instead of finding competing subnetworks, we find that all subnetworks{---}whether they generalize or not{---}share a set of attention heads, which we refer to as the {\_}heuristic core{\_}. Further analysis suggests that these attention heads emerge early in training and compute shallow, non-generalizing features. The model learns to generalize by incorporating additional attention heads, which depend on the outputs of the {``}heuristic{\''} heads to compute higher-level features. Overall, our results offer a more detailed picture of the mechanisms for syntactic generalization in pre-trained LMs.

## Links
- DOI: https://doi.org/10.18653/v1/2024.acl-long.774
- Publisher: https://aclanthology.org/2024.acl-long.774

## How to cite
@inproceedings{bhaskar-etal-2024-heuristic,
    title = "The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models",
    author = "Bhaskar, Adithya  and
      Friedman, Dan  and
      Chen, Danqi",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.774",
    doi = "10.18653/v1/2024.acl-long.774",
    pages = "14351--14368",
    abstract = "Prior work has found that pretrained language models (LMs) fine-tuned with different random seeds can achieve similar in-domain performance but generalize differently on tests of syntactic generalization. In this work, we show that, even within a single model, we can find multiple subnetworks that perform similarly in-domain, but generalize vastly differently. To better understand these phenomena, we investigate if they can be understood in terms of {``}competing subnetworks{\''}: the model initially represents a variety of distinct algorithms, corresponding to different subnetworks, and generalization occurs when it ultimately converges to one. This explanation has been used to account for generalization in simple algorithmic tasks ({``}grokking{\''}). Instead of finding competing subnetworks, we find that all subnetworks{---}whether they generalize or not{---}share a set of attention heads, which we refer to as the {\_}heuristic core{\_}. Further analysis suggests that these attention heads emerge early in training and compute shallow, non-generalizing features. The model learns to generalize by incorporating additional attention heads, which depend on the outputs of the {``}heuristic{\''} heads to compute higher-level features. Overall, our results offer a more detailed picture of the mechanisms for syntactic generalization in pre-trained LMs.",
}