Mediators in Determining what Processing BERT Performs First

Q: Does BERT really handle word-level things before grammar and grammar before meaning?

Tenney et al. (2019a)'s finding that BERT processes lexical tasks in lower layers and semantic tasks in higher layers is reaffirmed on BERT-base with the same 7 tasks. The layer ordering is nonetheless not identified by the probing data, since context length distributions can reverse parts of it. Holds for: BERT-base with the expected layer metric; the argument is about identifiability under an uncontrolled mediator, and does not show that a pipeline interpretation is false. Dependency parsing's expected layer in BERT-base is lower than NER's in all 4 measured context length ranges. The aggregate ordering nonetheless reverses when all dependency instances have context length 9 or more and all NER instances fall in the 3-5 range. Holds for: BERT-base; dependency parsing probed on the English Web Treebank and NER on OntoNotes 5.0; the reversal is a constructed distribution over existing test bins, not a naturally occurring dataset.

Q: Can one task look like BERT computes it in a higher layer than another even when it does not for any individual example?

Dependency parsing's expected layer in BERT-base is lower than NER's in all 4 measured context length ranges. The aggregate ordering nonetheless reverses when all dependency instances have context length 9 or more and all NER instances fall in the 3-5 range. Holds for: BERT-base; dependency parsing probed on the English Web Treebank and NER on OntoNotes 5.0; the reversal is a constructed distribution over existing test bins, not a naturally occurring dataset. SRL's expected layer in BERT-base exceeds that of the non-terminal (constituent) task in all 4 measured context length ranges. The order reverses when all SRL instances have context length 0-2 and all non-terminal instances have context length 9 or more. Holds for: BERT-base, both tasks probed on OntoNotes 5.0; a constructed distribution over the existing test bins rather than an observed dataset; a second instance of the same edge case as dependency parsing versus NER.

Q: How much does the layer gap between two linguistic tasks in BERT shrink once the probing examples are matched for span length?

Forcing the same context length distribution on two tasks changes the gap between their expected layers by up to 1.24 layers in absolute value, more than 50 times the unmediated difference. For another pair the gap shrinks by 86%, a change of 0.73 layers. Holds for: Natural Direct Effect computed per task-pair on BERT-base by imposing one task's empirical context length distribution on both; the pairs shown are NER/co-reference, NER/RC and SPR/RC, with every task-pair in the appendix.

Q: Do BERT probes move to higher layers when the two words being related are farther apart?

Raising the maximum allowed context length in the probing set raises the expected layer of co-reference, SRL, dependency parsing and relation classification in BERT-base. For 4 of the 7 probed tasks, the localization estimate is therefore driven by context length and not by task identity alone. Holds for: BERT-base; expected layer recomputed for every integer context length threshold, with thresholds capped per task so at least 2000 instances remain in the final bin; the other 3 tasks show no such clear increase.

Q: What paper should I read first about hidden confounds in studies of which BERT layer does what?

"Mediators in Determining what Processing BERT Performs First" brings Pearl-style mediation analysis into probing-based interpretability. A property of the probing examples, its context length, is treated as a mediator between the task and the layer a probe attributes it to. Holds for: As of publication in 2021; context length is the single mediator studied, on BERT-base with the expected layer metric, and other mediators are not claimed to be less important. "Mediators in Determining what Processing BERT Performs First" recommends reporting the controlled effect, the expected layer per context length bin, when comparing tasks against one mediator. It recommends the Natural Direct Effect when several effects or empirically given mediator distributions are involved. Holds for: Recommendations stated for edge probing with the expected layer metric; demonstrated only for context length on BERT-base, and offered as advice rather than validated against alternative controls.

Q: What is the right way to compare which BERT layer handles each task without being fooled by dataset differences?

"Mediators in Determining what Processing BERT Performs First" recommends reporting the controlled effect, the expected layer per context length bin, when comparing tasks against one mediator. It recommends the Natural Direct Effect when several effects or empirically given mediator distributions are involved. Holds for: Recommendations stated for edge probing with the expected layer metric; demonstrated only for context length on BERT-base, and offered as advice rather than validated against alternative controls. Among the 7 tasks probed in BERT-base, some pairs keep a clear expected-layer separation across context length bins, such as SRL versus co-reference. Other pairs have overlapping attainable ranges whose ordering can be flipped, such as SPR versus relation classification. Holds for: BERT-base, expected layer plotted per context length range of width 3, the narrowest width retaining at least 1% of examples per bin; the ranges are the controlled effect, not the effect under any single dataset's distribution.

Q: What does the distance between the two words in a probing example mean, and why does it matter for BERT layers?

"Mediators in Determining what Processing BERT Performs First" brings Pearl-style mediation analysis into probing-based interpretability. A property of the probing examples, its context length, is treated as a mediator between the task and the layer a probe attributes it to. Holds for: As of publication in 2021; context length is the single mediator studied, on BERT-base with the expected layer metric, and other mediators are not claimed to be less important. Raising the maximum allowed context length in the probing set raises the expected layer of co-reference, SRL, dependency parsing and relation classification in BERT-base. For 4 of the 7 probed tasks, the localization estimate is therefore driven by context length and not by task identity alone. Holds for: BERT-base; expected layer recomputed for every integer context length threshold, with thresholds capped per task so at least 2000 instances remain in the final bin; the other 3 tasks show no such clear increase.

Aviv Slobodkin; Leshem Choshen; Omri Abend

doi:10.18653/V1/2021.NAACL-MAIN.8

Mediators in Determining what Processing BERT Performs First

Aviv Slobodkin, Leshem Choshen, Omri Abend · NAACL 2021 · 2021

In one sentence

Probing BERT's layers with edge probing confounds task type with the prediction's context length: controlling for context length yields 196 different possible rankings of seven tasks by expected layer, so the "BERT rediscovers the NLP pipeline" ordering is not identified by the probing data alone.

Abstract

Probing neural models for the ability to perform downstream tasks using their activation patterns is often used to localize what parts of the network specialize in performing what tasks. However, little work addressed potential mediating factors in such comparisons. As a test-case mediating factor, we consider the prediction’s context length, namely the length of the span whose processing is minimally required to perform the prediction. We show that not controlling for context length may lead to contradictory conclusions as to the localization patterns of the network, depending on the distribution of the probing dataset. Indeed, when probing BERT with seven tasks, we find that it is possible to get 196 different rankings between them when manipulating the distribution of context lengths in the probing dataset. We conclude by presenting best practices for conducting such comparisons in the future.

Questions this paper answers

Can results about which BERT layer handles which linguistic task be an artifact of the probing dataset?
Is the layer-wise localization of edge-probing tasks in BERT identified by the probing data, or confounded by properties of the probed examples?
How do I check whether my layer-attribution probing result for a BERT task would survive a different probing set?
Should I trust a published per-layer ranking of BERT tasks when choosing which layer to read features from?: With the 7 edge-probing tasks of Tenney et al. (2019a), 196 rankings of those tasks by expected layer in BERT-base are attainable by choosing the context length distribution of the probing datasets. One attainable order is Non-term. < Dep. < SRL < RC < NER < Co-ref. < SPR.
Holds for: BERT-base, edge probing with the Jiant defaults, 7 tasks (NER, non-terminals, SRL, co-reference, SPR, relation classification, Stanford dependencies); context length bins of width 3 up to a per-task maximal threshold.; Raising the maximum allowed context length in the probing set raises the expected layer of co-reference, SRL, dependency parsing and relation classification in BERT-base. For 4 of the 7 probed tasks, the localization estimate is therefore driven by context length and not by task identity alone.
Holds for: BERT-base; expected layer recomputed for every integer context length threshold, with thresholds capped per task so at least 2000 instances remain in the final bin; the other 3 tasks show no such clear increase.; Tenney et al. (2019a)'s finding that BERT processes lexical tasks in lower layers and semantic tasks in higher layers is reaffirmed on BERT-base with the same 7 tasks. The layer ordering is nonetheless not identified by the probing data, since context length distributions can reverse parts of it.
Holds for: BERT-base with the expected layer metric; the argument is about identifiability under an uncontrolled mediator, and does not show that a pipeline interpretation is false.
Does BERT really handle word-level things before grammar and grammar before meaning?
Is the claim that BERT's layers rediscover the classical NLP pipeline supported once probing-set confounds are accounted for?
How much can I rely on the lexical-to-syntactic-to-semantic layer story when picking BERT layers for a task?
Should I still cite the NLP-pipeline account of BERT layers, or has the layer ordering been shown to be unstable?: Tenney et al. (2019a)'s finding that BERT processes lexical tasks in lower layers and semantic tasks in higher layers is reaffirmed on BERT-base with the same 7 tasks. The layer ordering is nonetheless not identified by the probing data, since context length distributions can reverse parts of it.
Holds for: BERT-base with the expected layer metric; the argument is about identifiability under an uncontrolled mediator, and does not show that a pipeline interpretation is false.; Dependency parsing's expected layer in BERT-base is lower than NER's in all 4 measured context length ranges. The aggregate ordering nonetheless reverses when all dependency instances have context length 9 or more and all NER instances fall in the 3-5 range.
Holds for: BERT-base; dependency parsing probed on the English Web Treebank and NER on OntoNotes 5.0; the reversal is a constructed distribution over existing test bins, not a naturally occurring dataset.
How many different orderings of seven linguistic tasks by BERT layer can the same probing setup produce?
How many task orderings by expected layer over the 7 edge-probing tasks are attainable by varying the probing sets' span-length distributions?
How do I tell which pairs of probing tasks have a stable layer ordering in BERT and which can be swapped?
If I report a layer ranking over seven probing tasks in BERT, how much of it could a reviewer flip by rebuilding the datasets?: With the 7 edge-probing tasks of Tenney et al. (2019a), 196 rankings of those tasks by expected layer in BERT-base are attainable by choosing the context length distribution of the probing datasets. One attainable order is Non-term. < Dep. < SRL < RC < NER < Co-ref. < SPR.
Holds for: BERT-base, edge probing with the Jiant defaults, 7 tasks (NER, non-terminals, SRL, co-reference, SPR, relation classification, Stanford dependencies); context length bins of width 3 up to a per-task maximal threshold.; Among the 7 tasks probed in BERT-base, some pairs keep a clear expected-layer separation across context length bins, such as SRL versus co-reference. Other pairs have overlapping attainable ranges whose ordering can be flipped, such as SPR versus relation classification.
Holds for: BERT-base, expected layer plotted per context length range of width 3, the narrowest width retaining at least 1% of examples per bin; the ranges are the controlled effect, not the effect under any single dataset's distribution.
Can one task look like BERT computes it in a higher layer than another even when it does not for any individual example?
Can aggregate expected layers of two edge-probing tasks reverse the per-bin ordering, i.e. a Simpson's paradox in layer attribution?
How do I avoid an aggregation artifact when I compare the average layer of dependency parsing and NER probes in BERT?
Is an aggregate expected-layer comparison between two probing tasks safe to report, or should I break it out by span length?: Dependency parsing's expected layer in BERT-base is lower than NER's in all 4 measured context length ranges. The aggregate ordering nonetheless reverses when all dependency instances have context length 9 or more and all NER instances fall in the 3-5 range.
Holds for: BERT-base; dependency parsing probed on the English Web Treebank and NER on OntoNotes 5.0; the reversal is a constructed distribution over existing test bins, not a naturally occurring dataset.; SRL's expected layer in BERT-base exceeds that of the non-terminal (constituent) task in all 4 measured context length ranges. The order reverses when all SRL instances have context length 0-2 and all non-terminal instances have context length 9 or more.
Holds for: BERT-base, both tasks probed on OntoNotes 5.0; a constructed distribution over the existing test bins rather than an observed dataset; a second instance of the same edge case as dependency parsing versus NER.
How much does the layer gap between two linguistic tasks in BERT shrink once the probing examples are matched for span length?
What magnitude of natural direct effect on the expected-layer difference remains once span-length distributions are equalized across two probing tasks?
How do I quantify how much of the layer difference between two BERT probing tasks comes from span length rather than the task?
Is it worth matching span-length distributions across my probing datasets, or is the effect on expected layer too small to bother?: Forcing the same context length distribution on two tasks changes the gap between their expected layers by up to 1.24 layers in absolute value, more than 50 times the unmediated difference. For another pair the gap shrinks by 86%, a change of 0.73 layers.
Holds for: Natural Direct Effect computed per task-pair on BERT-base by imposing one task's empirical context length distribution on both; the pairs shown are NER/co-reference, NER/RC and SPR/RC, with every task-pair in the appendix.
Do BERT probes move to higher layers when the two words being related are farther apart?
Does raising the maximum span distance in an edge-probing set monotonically raise the expected layer for co-reference, SRL, dependency and relation classification in BERT?
How do I set the maximum span distance in a probing dataset if I do not want the layer estimate to move?
If I cap span distance in my co-reference or SRL probing set, will my layer estimate change?: Raising the maximum allowed context length in the probing set raises the expected layer of co-reference, SRL, dependency parsing and relation classification in BERT-base. For 4 of the 7 probed tasks, the localization estimate is therefore driven by context length and not by task identity alone.
Holds for: BERT-base; expected layer recomputed for every integer context length threshold, with thresholds capped per task so at least 2000 instances remain in the final bin; the other 3 tasks show no such clear increase.
What paper should I read first about hidden confounds in studies of which BERT layer does what?
Which work brings causal mediation analysis to layer-wise probing of transformer representations?
Where do I start reading if I want to design a probing study of BERT layers that controls for example properties?
Is there a paper I can cite for why my reviewers should distrust a raw layer ranking from probing?: "Mediators in Determining what Processing BERT Performs First" brings Pearl-style mediation analysis into probing-based interpretability. A property of the probing examples, its context length, is treated as a mediator between the task and the layer a probe attributes it to.
Holds for: As of publication in 2021; context length is the single mediator studied, on BERT-base with the expected layer metric, and other mediators are not claimed to be less important.; "Mediators in Determining what Processing BERT Performs First" recommends reporting the controlled effect, the expected layer per context length bin, when comparing tasks against one mediator. It recommends the Natural Direct Effect when several effects or empirically given mediator distributions are involved.
Holds for: Recommendations stated for edge probing with the expected layer metric; demonstrated only for context length on BERT-base, and offered as advice rather than validated against alternative controls.
What is the right way to compare which BERT layer handles each task without being fooled by dataset differences?
What reporting practice is recommended for comparing expected layers across edge-probing tasks in the presence of a mediator such as span distance?
How do I control for span distance when I compare probes across BERT layers?
Should I report per-bin expected layers or a direct effect when I compare probing tasks across layers?: "Mediators in Determining what Processing BERT Performs First" recommends reporting the controlled effect, the expected layer per context length bin, when comparing tasks against one mediator. It recommends the Natural Direct Effect when several effects or empirically given mediator distributions are involved.
Holds for: Recommendations stated for edge probing with the expected layer metric; demonstrated only for context length on BERT-base, and offered as advice rather than validated against alternative controls.; Among the 7 tasks probed in BERT-base, some pairs keep a clear expected-layer separation across context length bins, such as SRL versus co-reference. Other pairs have overlapping attainable ranges whose ordering can be flipped, such as SPR versus relation classification.
Holds for: BERT-base, expected layer plotted per context length range of width 3, the narrowest width retaining at least 1% of examples per bin; the ranges are the controlled effect, not the effect under any single dataset's distribution.
What does the distance between the two words in a probing example mean, and why does it matter for BERT layers?
How is context length defined for an edge-probing instance, and how does it act as a mediator on expected layer?
How do I measure span distance in my edge-probing examples so I can bin results by it?: "Mediators in Determining what Processing BERT Performs First" brings Pearl-style mediation analysis into probing-based interpretability. A property of the probing examples, its context length, is treated as a mediator between the task and the layer a probe attributes it to.
Holds for: As of publication in 2021; context length is the single mediator studied, on BERT-base with the expected layer metric, and other mediators are not claimed to be less important.; Raising the maximum allowed context length in the probing set raises the expected layer of co-reference, SRL, dependency parsing and relation classification in BERT-base. For 4 of the 7 probed tasks, the localization estimate is therefore driven by context length and not by task identity alone.
Holds for: BERT-base; expected layer recomputed for every integer context length threshold, with thresholds capped per task so at least 2000 instances remain in the final bin; the other 3 tasks show no such clear increase.

Claims and scope

With the 7 edge-probing tasks of Tenney et al. (2019a), 196 rankings of those tasks by expected layer in BERT-base are attainable by choosing the context length distribution of the probing datasets. One attainable order is Non-term. < Dep. < SRL < RC < NER < Co-ref. < SPR. (Section 3.2.1 and Figure 6)
Scope: BERT-base, edge probing with the Jiant defaults, 7 tasks (NER, non-terminals, SRL, co-reference, SPR, relation classification, Stanford dependencies); context length bins of width 3 up to a per-task maximal threshold.
Dependency parsing's expected layer in BERT-base is lower than NER's in all 4 measured context length ranges. The aggregate ordering nonetheless reverses when all dependency instances have context length 9 or more and all NER instances fall in the 3-5 range. (Figure 3)
Scope: BERT-base; dependency parsing probed on the English Web Treebank and NER on OntoNotes 5.0; the reversal is a constructed distribution over existing test bins, not a naturally occurring dataset.
SRL's expected layer in BERT-base exceeds that of the non-terminal (constituent) task in all 4 measured context length ranges. The order reverses when all SRL instances have context length 0-2 and all non-terminal instances have context length 9 or more. (Figure 7)
Scope: BERT-base, both tasks probed on OntoNotes 5.0; a constructed distribution over the existing test bins rather than an observed dataset; a second instance of the same edge case as dependency parsing versus NER.
Forcing the same context length distribution on two tasks changes the gap between their expected layers by up to 1.24 layers in absolute value, more than 50 times the unmediated difference. For another pair the gap shrinks by 86%, a change of 0.73 layers. (Figure 4 (all pairs in Figure 9))
Scope: Natural Direct Effect computed per task-pair on BERT-base by imposing one task's empirical context length distribution on both; the pairs shown are NER/co-reference, NER/RC and SPR/RC, with every task-pair in the appendix.
Raising the maximum allowed context length in the probing set raises the expected layer of co-reference, SRL, dependency parsing and relation classification in BERT-base. For 4 of the 7 probed tasks, the localization estimate is therefore driven by context length and not by task identity alone. (Figure 2)
Scope: BERT-base; expected layer recomputed for every integer context length threshold, with thresholds capped per task so at least 2000 instances remain in the final bin; the other 3 tasks show no such clear increase.
Tenney et al. (2019a)'s finding that BERT processes lexical tasks in lower layers and semantic tasks in higher layers is reaffirmed on BERT-base with the same 7 tasks. The layer ordering is nonetheless not identified by the probing data, since context length distributions can reverse parts of it. (Section 1 and Section 3.1.1)
Scope: BERT-base with the expected layer metric; the argument is about identifiability under an uncontrolled mediator, and does not show that a pipeline interpretation is false.
Among the 7 tasks probed in BERT-base, some pairs keep a clear expected-layer separation across context length bins, such as SRL versus co-reference. Other pairs have overlapping attainable ranges whose ordering can be flipped, such as SPR versus relation classification. (Figure 5 and Figure 6)
Scope: BERT-base, expected layer plotted per context length range of width 3, the narrowest width retaining at least 1% of examples per bin; the ranges are the controlled effect, not the effect under any single dataset's distribution.
"Mediators in Determining what Processing BERT Performs First" brings Pearl-style mediation analysis into probing-based interpretability. A property of the probing examples, its context length, is treated as a mediator between the task and the layer a probe attributes it to. (Section 2 and Section 3.2)
Scope: As of publication in 2021; context length is the single mediator studied, on BERT-base with the expected layer metric, and other mediators are not claimed to be less important.
"Mediators in Determining what Processing BERT Performs First" recommends reporting the controlled effect, the expected layer per context length bin, when comparing tasks against one mediator. It recommends the Natural Direct Effect when several effects or empirically given mediator distributions are involved. (Section 4)
Scope: Recommendations stated for edge probing with the expected layer metric; demonstrated only for context length on BERT-base, and offered as advice rather than validated against alternative controls.

Common misreadings

The paper does not show that BERT lacks any layer-wise division of labour; it reproduces the measured ordering of tasks and argues only that the ordering is not identified once context length is left uncontrolled.
The 196 rankings are not 196 rankings observed in real datasets: they are the rankings attainable by combining the per-context-length expected layers of the 7 tasks under arbitrary context length distributions.
The extreme reversals for dependency parsing versus NER and for SRL versus non-terminals are constructed distributions over existing test bins, not naturally occurring corpora.
Context length is presented as one test-case mediator, not as the only or the largest confound in probing; other mediating factors are explicitly left to future work.
Controlling for context length is not shown to make expected-layer comparisons reliable in general; some task pairs retain overlapping attainable ranges whose ordering stays undetermined.

Terminology in this paper

context length: The number of tokens whose processing is minimally required for a prediction, operationalized as the distance between the earliest and latest index of the labelled span(s): span length for single-span tasks like NER, and dependency length for tasks relating two spans.
expected layer: Tenney et al.'s localization metric: the average layer index weighted by the incremental probing-performance gain each additional layer contributes, computed over the 12 layers of BERT-base.
controlled effect: The expected layer computed separately within each context length bin, so that task comparisons do not depend on any particular distribution of context lengths in the probing dataset.
Natural Direct Effect (NDE): In probing, the difference between two tasks' expected layers when the same context length distribution is imposed on both tasks, following Pearl's mediation analysis.

How to cite

@inproceedings{DBLP:conf/naacl/SlobodkinCA21,author       = {Aviv Slobodkin and
                  Leshem Choshen and
                  Omri Abend},
  editor       = {Kristina Toutanova and
                  Anna Rumshisky and
                  Luke Zettlemoyer and
                  Dilek Hakkani{-}T{"{u}}r and
                  Iz Beltagy and
                  Steven Bethard and
                  Ryan Cotterell and
                  Tanmoy Chakraborty and
                  Yichao Zhou},
  title        = {Mediators in Determining what Processing {BERT} Performs First},
  booktitle    = {Proceedings of the 2021 Conference of the North American Chapter of
                  the Association for Computational Linguistics: Human Language Technologies,
                  {NAACL-HLT} 2021, Online, June 6-11, 2021},
  pages        = {86--93},
  publisher    = {Association for Computational Linguistics},
  year         = {2021},
  url          = {https://doi.org/10.18653/v1/2021.naacl-main.8},
  doi          = {10.18653/V1/2021.NAACL-MAIN.8},
  timestamp    = {Fri, 06 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/naacl/SlobodkinCA21.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.