Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets

true-positive agreement: the fraction of independently trained networks that classify a given example correctly

Guy Hacohen, Leshem Choshen, Daphna Weinshall · ICML 2020 · 2020

In one sentence

"Let's Agree to Agree" shows that independently trained neural networks learn benchmark datasets in nearly the same example-by-example order — measured by TP-agreement, the fraction of models classifying an example correctly — and that this order breaks down on synthetic data, on shuffled labels, and for AdaBoost.

Abstract

We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries -- models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Thus, when fixing the architecture, we show synthetic datasets where this pattern ceases to exist. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.

Questions this paper answers

If two neural networks with the same architecture are trained separately, do they get the same pictures right and the same pictures wrong?
Do independently initialized networks of identical architecture exhibit correlated per-example correctness over the course of training, and does the effect survive changes to learning rate, optimizer and batch size?
How do I tell whether the examples my model gets wrong are specific to my training run or common to any run of that architecture?
If I retrain my classifier with a different seed and different hyperparameters, should I expect the same examples to fail?
Independently trained neural networks of the same architecture produce a bi-modal distribution of per-example TP-agreement throughout training on every natural benchmark tested, including ImageNet with 27 ResNet-50 models. Most examples are classified correctly by all the models or by none of them.
Holds for: 27 ResNet-50 models on ImageNet, train and validation sets, epochs 1 to 100; each epoch measured by training a fresh set of networks from scratch. Random classification vectors with matching accuracy give a uni-modal Gaussian instead.
The shared classification order of same-architecture networks survives changes to learning rate (10^-4 to 1), optimizer (SGD, Adam, AdaDelta), batch size (1 to 2500) and dropout (0 to 0.5). It also survives changes of initialization (Xavier, He normal, LeCun normal, truncated normal) and activation (ReLU, ELU, tanh, linear).
Holds for: Direct hyper-parameter sweeps were run on st-VGG trained on the small-mammals dataset with 100 instances each; other ranges came indirectly from varying architectures. Qualitative bi-modality is what is preserved, not exact accuracies.
Has the finding that separately trained networks learn examples in the same order been checked on more than one dataset?
On which benchmarks has consistent per-example learning order across networks been demonstrated, and does it extend beyond vision to text classification?
How do I know whether shared example learning order will show up on my dataset, whether it is images or text?
My data is text, not ImageNet photos — does the shared learning order result apply to me?
The shared learning order of neural networks holds across MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, tiny ImageNet, VGGFace2, ImageNet and a 20-class Stack Overflow text classification task, for both train and test sets.
Holds for: Small handcrafted CNNs, st-VGG, VGG-19, AlexNet, DenseNet, ResNet-50 and an attention-based BiLSTM with GloVe embeddings; 20 to 100 models per setting. The text case is one dataset of 39K train and 1K test questions.
Independently trained neural networks of the same architecture produce a bi-modal distribution of per-example TP-agreement throughout training on every natural benchmark tested, including ImageNet with 27 ResNet-50 models. Most examples are classified correctly by all the models or by none of them.
Holds for: 27 ResNet-50 models on ImageNet, train and validation sets, epochs 1 to 100; each epoch measured by training a fresh set of networks from scratch. Random classification vectors with matching accuracy give a uni-modal Gaussian instead.
Do two very different network designs learn the images of a dataset in roughly the same order?
How strongly do per-example accessibility scores correlate across ResNet-50, AlexNet and DenseNet collections trained on ImageNet?
How do I check whether an ordering of examples by difficulty computed from one architecture transfers to another?
Can I reuse example-difficulty scores computed with ResNet-50 if my model is a different convolutional architecture?
Accessibility scores on the ImageNet train set correlate at r=0.87 between ResNet-50 and AlexNet collections and r=0.97 between ResNet-50 and DenseNet, against r=0.99 between two disjoint collections of ResNet-50.
Holds for: 27 ResNet-50, 22 AlexNet and 6 DenseNet models on ImageNet, all p ≤ 10^-50; Top-1 error differs widely (AlexNet 0.45, ResNet-50 0.24, DenseNet 0.27).
ResNet-50 on ImageNet first learns the examples AlexNet gets right and only then learns examples AlexNet never learns. The count of examples that only one of the two architectures classifies correctly stays constant and low as accuracy improves.
Holds for: Majority-vote ensembles of 27 ResNet-50 and 22 AlexNet models on the ImageNet validation set, compared at epochs of matched accuracy (±1% tolerance); ResNet-50 reaches 80% and AlexNet 60%, so past 60% the comparison is against converged AlexNet.
When one network is more accurate than another, does the better one get right everything the weaker one gets right plus more?
Is the set of examples classified correctly by a higher-capacity architecture approximately a superset of the set learned by a weaker or purely linear model?
How do I predict which extra examples I gain by moving from a weak model to a stronger one?
If I upgrade from a small or linear model to a deeper one, will the examples my old model already handled stay correct?
ResNet-50 on ImageNet first learns the examples AlexNet gets right and only then learns examples AlexNet never learns. The count of examples that only one of the two architectures classifies correctly stays constant and low as accuracy improves.
Holds for: Majority-vote ensembles of 27 ResNet-50 and 22 AlexNet models on the ImageNet validation set, compared at epochs of matched accuracy (±1% tolerance); ResNet-50 reaches 80% and AlexNet 60%, so past 60% the comparison is against converged AlexNet.
Linear st-VGG networks on the small-mammals dataset reach 0.43 average accuracy versus 0.56 for non-linear st-VGG, yet still show bi-modal TP-agreement (maximum Pearson bi-modality 0.06 versus 0.22). The non-linear networks first learn the examples the linear ones learn.
Holds for: 100 linear st-VGG models on the small-mammals dataset (2500 train, 500 test images, 5 classes); linear networks converge in a few epochs, too fast for a meaningful accessibility score, so the nesting is read off matched-accuracy counts.
Are there datasets where separately trained networks pick up examples in completely different orders?
Does the bi-modal per-example agreement distribution vanish on synthetic data such as Gabor patches and overlapping Gaussians, or under label randomization?
How do I test whether the shared learning order I see is a property of my data rather than of stochastic gradient descent?
If my benchmark is synthetically generated, should I still expect networks to agree on which examples are easy?
The shared learning order disappears on synthetic data. 100 st-VGG models on a 12-class Gabor-patch dataset and 100 fully connected models on a 2-class overlapping-Gaussian task show approximately normal, not bi-modal, TP-agreement distributions.
Holds for: Both datasets are hand-crafted and are learned successfully by the networks; the Gabor case partially regains bi-modal character on the test set at convergence. Gaussian samples are 3072-dimensional, means 0 and 0.1, identity covariance.
With randomly shuffled labels on the small-mammals dataset, 100 st-VGG models memorize the training set to 100% accuracy but do so in different orders. TP-agreement stays Gaussian, with minimum Pearson bi-modality 1.07 on train and 1.35 on test.
Holds for: 100 st-VGG instances with dropout layers removed to enable fitting random labels; test accuracy stays at chance. Shows association between bi-modality and generalization, not a causal mechanism.
If the labels of a dataset are scrambled at random, do networks still memorize the examples in the same order?
Under randomly permuted labels, does per-example agreement across a collection of networks remain bi-modal or become Gaussian?
How do I separate agreement that comes from genuine data structure from agreement that comes from mere memorization capacity?
My labels are very noisy — can I still trust example-difficulty scores derived from agreement across training runs?
With randomly shuffled labels on the small-mammals dataset, 100 st-VGG models memorize the training set to 100% accuracy but do so in different orders. TP-agreement stays Gaussian, with minimum Pearson bi-modality 1.07 on train and 1.35 on test.
Holds for: 100 st-VGG instances with dropout layers removed to enable fitting random labels; test accuracy stays at chance. Shows association between bi-modality and generalization, not a causal mechanism.
Do classifiers that are not neural networks find the same images easy and hard as neural networks do?
How correlated are accessibility scores from boosted linear weak learners with those from convolutional networks on CIFAR-10, on raw pixels and on Inception-V3 features?
How do I find out whether an example-difficulty ranking is a property of the dataset or of the specific learning algorithm that produced it?
Can I use a difficulty ordering derived from a boosted-tree or boosted-linear model to curate data for a CNN?
AdaBoost with up to 100 weak linear classifiers learns CIFAR-10 in an order only weakly related to that of neural networks. Accessibility scores correlate at r=0.35 on raw pixels and r=0.05 when AdaBoost uses Inception-V3 penultimate-layer features.
Holds for: AdaBoost versus 100 st-VGG models on CIFAR-10, both p ≤ 10^-20; replicated on small-mammals, fish, insect, cats-and-dogs and ImageNet-cats subsets. CIFAR-100 and full ImageNet excluded as AdaBoost accuracy is too low there.
The shared learning order disappears on synthetic data. 100 st-VGG models on a 12-class Gabor-patch dataset and 100 fully connected models on a 2-class overlapping-Gaussian task show approximately normal, not bi-modal, TP-agreement distributions.
Holds for: Both datasets are hand-crafted and are learned successfully by the networks; the Gabor case partially regains bi-modal character on the test set at convergence. Gaussian samples are 3072-dimensional, means 0 and 0.1, identity covariance.
If two networks never see the same training examples, do they still agree on which unseen examples are easy?
Does consistent test-set learning order persist when networks are trained on disjoint samples drawn from the same distribution?
How do I check that an example-difficulty score I computed is not an artifact of the particular training split I used?
I only have one training split — will difficulty scores estimated from it hold for models trained on other samples of the same data?
Neural networks trained on disjoint partitions of the same training distribution learn a common test set in nearly the same order, with almost perfect correlation between accessibility scores computed from different partitions.
Holds for: Fashion-MNIST train set split into 60 parts of 1000 images, 100 st-VGG instances per part, epochs 0 to 40; the shared order concerns the common unmodified test set.
Once a network starts getting a particular image right, does it later start getting it wrong again?
Is per-example correctness stable across epochs once an example reaches high agreement across a collection of trained networks, or is forgetting common?
How do I know whether I can stop tracking an example once most of my training runs classify it correctly?
If I checkpoint mid-training, will the examples already classified correctly still be correct at the end?
Once an example is classified correctly by most networks of a collection it is rarely misclassified later, with per-example TP-agreement staying pinned at 0 or 1 for most of training. The number of examples that rise near 1 and then fall to 0 is negligible.
Holds for: 5 tracked example images from st-VGG trained on the small-mammals dataset, with the pattern reported for the majority of examples in that setting; this property is what licenses defining a per-example learning epoch.
Do convolutional networks with no nonlinearity pick up examples in the same order as ordinary ones?
Does a purely linear convolutional network reproduce bi-modal per-example agreement, and is its learned subset nested inside that of the non-linear counterpart?
How do I test whether nonlinear activations are what causes networks to agree on example ordering?
Is the agreement in learning order something I would still see if I stripped the activations out of my convolutional model?
Linear st-VGG networks on the small-mammals dataset reach 0.43 average accuracy versus 0.56 for non-linear st-VGG, yet still show bi-modal TP-agreement (maximum Pearson bi-modality 0.06 versus 0.22). The non-linear networks first learn the examples the linear ones learn.
Holds for: 100 linear st-VGG models on the small-mammals dataset (2500 train, 500 test images, 5 classes); linear networks converge in a few epochs, too fast for a meaningful accessibility score, so the nesting is read off matched-accuracy counts.
Is there a way to compare two trained networks by looking at which examples they get right instead of at their internal features?
What measure quantifies per-example prediction agreement across a collection of networks as an alternative to representation-similarity analyses such as SVCCA?
How do I measure similarity between two trained models without probing or aligning their hidden representations?
Should I compare my two models with a representation-similarity method or just with their per-example predictions?
"Let's Agree to Agree" compares trained neural networks by their per-example predictions rather than by their internal representations. It offers an alternative to representation-similarity methods such as SVCCA for asking whether two networks are alike.
Holds for: As of the ICML 2020 publication; the comparison is behavioural and needs a labelled dataset and multiple trained instances, and it says nothing about which internal features a network uses.
Independently trained neural networks of the same architecture produce a bi-modal distribution of per-example TP-agreement throughout training on every natural benchmark tested, including ImageNet with 27 ResNet-50 models. Most examples are classified correctly by all the models or by none of them.
Holds for: 27 ResNet-50 models on ImageNet, train and validation sets, epochs 1 to 100; each epoch measured by training a fresh set of networks from scratch. Random classification vectors with matching accuracy give a uni-modal Gaussian instead.
What should I read first about neural networks learning examples in a consistent order?
Which work established consistent per-example learning order across independently trained networks as an empirical phenomenon on natural datasets?
Where do I start reading if I want to build example-difficulty or data-pruning scores from training dynamics?
Which paper should I cite as the origin of shared example learning order in deep networks?
"Let's Agree to Agree" established shared example learning order as an empirical phenomenon of neural networks on natural datasets, a starting point for later work on example difficulty, learning-order and data-pruning scores.
Holds for: Evidence is empirical across image benchmarks and one text benchmark, with no theoretical account of why particular examples are easy; the paper reports that using the discovered order as a curriculum did not improve learning.
"Let's Agree to Agree" compares trained neural networks by their per-example predictions rather than by their internal representations. It offers an alternative to representation-similarity methods such as SVCCA for asking whether two networks are alike.
Holds for: As of the ICML 2020 publication; the comparison is behavioural and needs a labelled dataset and multiple trained instances, and it says nothing about which internal features a network uses.
Do networks learn whole categories in a consistent order, or only individual pictures?
Does consistent classification order across networks extend from individual examples to class labels, inducing a hierarchy over classes?
How do I find out which classes in my dataset my models pick up first?
Can I use the order in which classes are learned to decide which categories need more data?
Classes, not just examples, are learned in a consistent order. Early in training only images from 2 of the 5 small-mammals classes reach TP-agreement 1, and further class labels emerge gradually as learning proceeds.
Holds for: 100 st-VGG instances on the 5-class small-mammals dataset, epochs 1, 2, 30 and 140; each image coloured by its most frequent predicted label across the collection.
Can knowing which examples a network learns first be used to train faster or design a curriculum?
Does consistent per-example learning order support downstream applications such as curriculum construction or dataset pruning?
How do I turn an ordering of examples by when networks learn them into a training curriculum or a pruned dataset?
Should I order or prune my training data by the order networks tend to learn examples?
"Let's Agree to Agree" established shared example learning order as an empirical phenomenon of neural networks on natural datasets, a starting point for later work on example difficulty, learning-order and data-pruning scores.
Holds for: Evidence is empirical across image benchmarks and one text benchmark, with no theoretical account of why particular examples are easy; the paper reports that using the discovered order as a curriculum did not improve learning.

Claims and scope

Common misreadings

Terminology in this paper

TP-agreement
For a single labelled example, the fraction of a collection of independently trained networks that classify it correctly after the same number of training epochs; contrasted with accuracy, which averages over examples rather than over models.
Agreement score
For a single example, the largest fraction of a collection of networks that predict the same label, whether or not that label is correct; complements TP-agreement by capturing consensus among mistakes.
accessibility score
An example's TP-agreement averaged over all training epochs measured, used as a per-example measure of how early and how robustly a given architecture learns it.
Pearson bi-modality
The score kurtosis(X) − skewness²(X) − 1 of a distribution, where lower values indicate a more bi-modal distribution; used to quantify how sharply per-example agreement splits between 0 and 1.
st-VGG
A stripped 8-convolutional-layer version of VGG with 32 to 256 filters per layer, max-pooling and dropout, used as a fast stand-in for larger CNNs on CIFAR-scale data.
small-mammals dataset
The 5-class small-mammals super-class of CIFAR-100, with 2500 train and 500 test images of size 32×32×3.

How to cite

@inproceedings{DBLP:conf/icml/HacohenCW20,author       = {Guy Hacohen and
                  Leshem Choshen and
                  Daphna Weinshall},
  title        = {Let's Agree to Agree: Neural Networks Share Classification Order on
                  Real Datasets},
  booktitle    = {Proceedings of the 37th International Conference on Machine Learning,
                  {ICML} 2020, 13-18 July 2020, Virtual Event},
  series       = {Proceedings of Machine Learning Research},
  volume       = {119},
  pages        = {3950--3960},
  publisher    = {{PMLR}},
  year         = {2020},
  url          = {http://proceedings.mlr.press/v119/hacohen20a.html},
  timestamp    = {Tue, 15 Dec 2020 00:00:00 +0100},
  biburl       = {https://dblp.org/rec/conf/icml/HacohenCW20.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.