ComSum: Commit Messages Summarization and Meaning Preservation

a summarization dataset built from Git commit messages and their one-line subjects

Leshem Choshen, Idan Amit · arXiv · 2021

• arXiv
• PDF
• HTML
• Hugging Face
• alphaXiv
• Semantic Scholar
• Code
• dataset
• zenodo

In one sentence

ComSum turns 7.5 million Git commits into a text summarization dataset by pairing each commit message with its one-line subject, and adds a meaning-preservation evaluation based on the corrective/adaptive/refactor commit taxonomy instead of Rouge alone.

Abstract

We present ComSum, a data set of 7 million commit messages for text summarization. When documenting commits, software code changes, both a message and its summary are posted. We gather and filter those to curate developers' work summarization data set. Along with its growing size, practicality and challenging language domain, the data set benefits from the living field of empirical software engineering. As commits follow a typology, we propose to not only evaluate outputs by Rouge, but by their meaning preservation.

Questions this paper answers

is there a big collection of software commit messages paired with their one-line subjects for training summarizers?

what scale and provenance does the ComSum commit-message summarization corpus have across GitHub projects and authors?

where do I get a large-scale training set for generating commit subject lines from commit message bodies?

is there enough commit-message data out there for me to train a summarization model on, or do I need to scrape GitHub myself?

ComSum contains 7,540,026 commit-message/subject pairs drawn from 19,720 GitHub projects and 317,423 authors. The average message has 494 characters, the average subject 53, an average compression ratio of 11.08.
Holds for: Commits pushed to GitHub before 2021 and indexed in the BigQuery GitHub schema; projects required at least 50 commits during 2020, and each message must be at least 100 characters longer than its subject.

ComSum introduces software commit messages as a summarization domain and argues the pairing is natural because Git and GitHub already present the one-line subject as a summary of the longer message.
Holds for: English-language commits, which are about 99% of the sampled commits; the domain is specific to software development, so gains on ComSum need not transfer to news or literary summarization.

do commit message summaries reuse the words of the original text less than news headlines do?

how does ComSum compare with XSum and BOOKSUM on extractive coverage, density and vocabulary size?

which summarization corpus should I pick if I need genuinely abstractive reference summaries and a large vocabulary?

if my model keeps copying spans instead of abstracting, would commit-message data push it harder than news data?

ComSum is the most abstractive of the summarization datasets it is compared with, at a coverage of 0.27 and a density of 0.89. XSum scores 0.66 and 1.09, and BOOKSUM Paragraph 0.5 and 0.92.
Holds for: Coverage and density as defined by Grusky et al. (2018); comparison set is Arxiv/PubMed, BigPatent, CNN/DM, Newsroom, XSum and BOOKSUM Paragraph.

ComSum commit messages have a vocabulary of over 2M types in the validation set alone and 19M overall, against the 1.4M reported for the NYTimes dataset. The summary vocabulary is 0.5M and 3.9M against NYTimes' 0.3M.
Holds for: Validation-set figures are size-matched for fairness against news datasets; vocabulary counted as unfiltered surface types, which in the commit domain includes identifiers, version numbers and hashes.

how well does an off-the-shelf summarization model do at writing commit subject lines?

what RougeL does BART reach on ComSum zero-shot versus fine-tuned, and how does that compare to CNN/Daily Mail and XSum?

do I need to fine-tune a pretrained summarizer on commit messages, or will it work out of the box?

is commit-message summarization already solved by fine-tuning BART, or is there headroom left for my model?

Fine-tuned BART reaches RougeL 27.2 on the ComSum test set against 14.9 for zero-shot BART. The gain suggests the dataset is large enough to partly overcome the domain shift from BART's Wikipedia and Books pretraining.
Holds for: BART fine-tuned with max source length 512, target length 128, learning rate 1e-4, batch size 256, trained for 1 week on 4 Nvidia M60 GPUs; heuristic baselines computed on 10k samples.

ComSum is harder for BART than news summarization benchmarks are: BART reaches RougeL 44.2 on CNN/Daily Mail and 27.2 on XSum. On ComSum it reaches 33.2 on the train set and 27.2 on the test set.
Holds for: Single BART configuration, no error bars because training was repeated only once; CNN/DM and XSum figures are the ones reported by Lewis et al. (2020) rather than re-run.

can you get a decent commit subject line just by copying part of the message?

how do lead/random-sentence extraction and same-author related-commit subjects score in RougeL on ComSum?

what trivial baselines should I beat before claiming a commit summarizer works?

would a simple copy-the-first-sentence heuristic be good enough for generating commit subjects in my tooling?

Copying the commit message instead of summarizing it scores only RougeL 12.0 on ComSum, and picking a random sentence from the message scores 13.6. Extraction from the input is far from a good commit summary.
Holds for: Heuristic baselines on 10k training-set samples; the Subject-and-Message upper reference reaches 29.5 RougeL but is not usable for prediction because it includes the gold summary.

A subject borrowed from a related commit by the same author in the same project within a week scores RougeL 14.6 on ComSum, rising to 15.5 when both commits are bug fixes. Topic and author style are therefore not substitutes for the summary.
Holds for: Related-commit and Related-Fix baselines computed on the training split, 10k samples; pairing requires same author, same project and a one-week window.

does it matter whether a commit dataset is split by project or just shuffled by commit?

how much RougeL does a repository-level split of ComSum cost fine-tuned BART relative to a commit-level split?

how should I split a commit-message dataset so the test set actually measures generalization to new projects?

if I train a commit summarizer on my own repos, should I expect the score to drop on a codebase it has never seen?

Splitting ComSum by repository rather than by commit costs fine-tuned BART about 6 RougeL points, while the split-by-commit drop is about 1 point. The gap is therefore domain shift between repositories rather than memorization.
Holds for: Comparison between the main repository-level split and the minor commit-level split; the test set is also more abstractive, with coverage 0.25 versus 0.31 and density 0.86 versus 0.99.

how often does a summarization model quietly change what the original text was saying?

what is BART's meaning-preservation precision on ComSum for corrective, refactor and adaptive commit categories with distractor terms?

how do I test whether my summarizer keeps the intent of the input rather than just matching words?

can I trust a fine-tuned summarizer to keep a bug fix labeled as a bug fix rather than as a refactor?

BART changes a ComSum commit's meaning in 10% of corrective cases, 16% of refactor cases and 35% of adaptive cases. Those rates are measured on messages that carry a distractor core term but are not of the matching commit type.
Holds for: Meaning judged by the commit classifiers of Amit and Feitelson (93% accuracy for corrective and refactoring, 65% for adaptive), applied to both message and summary.

BART's highest meaning-preservation precision on any commit concept tested on ComSum is 75%. A quarter or more of summaries therefore drop the concept expressed in the message even when Rouge scores are high.
Holds for: Precision-like metric P(concept(model(message))) given P(concept(message)) over corrective, adaptive and refactor concepts, estimated with automatic commit classifiers rather than human labels.

which research suggests judging a summarizer by whether it keeps the original meaning rather than by word overlap?

what work proposes domain-grounded meaning-preservation evaluation for summarization as an alternative to Rouge?

where do I start reading if I want an evaluation for my summarizer that goes beyond Rouge?

my Rouge scores look fine, so what should I read to find out whether my summaries actually preserve meaning?

ComSum proposes evaluating summarizers by whether the output preserves the commit's type in Swanson's corrective/adaptive/perfective taxonomy, a domain-grounded alternative to word-overlap scores like Rouge.
Holds for: Applicable where an automatic, high-agreement classifier for the meaning aspect exists; for commits, human agreement on bug classification is 95%.

BART's highest meaning-preservation precision on any commit concept tested on ComSum is 75%. A quarter or more of summaries therefore drop the concept expressed in the message even when Rouge scores are high.
Holds for: Precision-like metric P(concept(model(message))) given P(concept(message)) over corrective, adaptive and refactor concepts, estimated with automatic commit classifiers rather than human labels.

is the short first line of a git commit really a summary of the rest of the message?

what proportion of commit subjects in ComSum were manually validated as faithful summaries of the message body, and how does filtering merge and administrative commits change it?

can I use commit subject lines as reference summaries without hand-annotating them?

how much label noise am I accepting if I treat the commit subject as ground-truth summary in my training data?

In manual labeling of 100 ComSum samples, 80% of commit subjects were proper summaries of their message. Filtering merge commits and administrative messages raises that to about 90% on the labeled sample.
Holds for: Two authors labeled independently with 82% initial agreement, rising to 99% after protocol tuning; the administrative-message heuristic has 98.9% precision and about 75% recall.

are the results the same for bug-fix commits as for cleanup or feature-adaptation commits?

do RougeL trends on the corrective, refactor and adaptive ComSum test subsets track the full test set for zero-shot and fine-tuned BART?

do I need separate models or separate evaluations for different kinds of commits?

if most of my commits are bug fixes, will a commit summarizer behave differently on them than the headline numbers suggest?

Rouge trends on the corrective, refactor and adaptive ComSum test subsets match the general test set, with fine-tuned BART at RougeL 26.2 to 26.9 against 14.3 to 16.0 for zero-shot BART.
Holds for: Typed test subsets built from the commit-type classifiers; results are for BART, zero-shot BART and the Random Message Sentence heuristic only.

can a commit-message dataset be rebuilt from newer commits as projects keep growing?

does ComSum release the extraction pipeline and SQL queries alongside a frozen static split so the corpus can be regenerated?

how do I regenerate a larger commit-summarization dataset with my own filtering rules?

if I need more data or different filters than the released commit-summarization split gives me, can I rebuild it myself?

ComSum ships both a frozen static release and the extraction code and SQL queries, so the same pipeline can rebuild a larger dataset from later commits or with different filtering choices.
Holds for: As of the 2021 release; regeneration depends on the BigQuery GitHub schema, from which projects can disappear, so a regenerated dataset will not match the frozen one.

which paper first treated software commit messages as a text summarization task?

what work introduced commit-message-to-subject pairs as a summarization domain with a software-grounded evaluation?

what should I read to find NLP datasets and tasks drawn from software engineering artifacts?

I work on developer tools and want a summarization benchmark from real code history, so which paper should I cite as the starting point?

ComSum introduces software commit messages as a summarization domain and argues the pairing is natural because Git and GitHub already present the one-line subject as a summary of the longer message.
Holds for: English-language commits, which are about 99% of the sampled commits; the domain is specific to software development, so gains on ComSum need not transfer to news or literary summarization.

ComSum proposes evaluating summarizers by whether the output preserves the commit's type in Swanson's corrective/adaptive/perfective taxonomy, a domain-grounded alternative to word-overlap scores like Rouge.
Holds for: Applicable where an automatic, high-agreement classifier for the meaning aspect exists; for commits, human agreement on bug classification is 95%.

Claims and scope

• ComSum contains 7,540,026 commit-message/subject pairs drawn from 19,720 GitHub projects and 317,423 authors. The average message has 494 characters, the average subject 53, an average compression ratio of 11.08. (Section 4)
  Scope: Commits pushed to GitHub before 2021 and indexed in the BigQuery GitHub schema; projects required at least 50 commits during 2020, and each message must be at least 100 characters longer than its subject.
• ComSum is the most abstractive of the summarization datasets it is compared with, at a coverage of 0.27 and a density of 0.89. XSum scores 0.66 and 1.09, and BOOKSUM Paragraph 0.5 and 0.92. (Table 1)
  Scope: Coverage and density as defined by Grusky et al. (2018); comparison set is Arxiv/PubMed, BigPatent, CNN/DM, Newsroom, XSum and BOOKSUM Paragraph.
• ComSum commit messages have a vocabulary of over 2M types in the validation set alone and 19M overall, against the 1.4M reported for the NYTimes dataset. The summary vocabulary is 0.5M and 3.9M against NYTimes' 0.3M. (Section 2)
  Scope: Validation-set figures are size-matched for fairness against news datasets; vocabulary counted as unfiltered surface types, which in the commit domain includes identifiers, version numbers and hashes.
• Fine-tuned BART reaches RougeL 27.2 on the ComSum test set against 14.9 for zero-shot BART. The gain suggests the dataset is large enough to partly overcome the domain shift from BART's Wikipedia and Books pretraining. (Table 2)
  Scope: BART fine-tuned with max source length 512, target length 128, learning rate 1e-4, batch size 256, trained for 1 week on 4 Nvidia M60 GPUs; heuristic baselines computed on 10k samples.
• ComSum is harder for BART than news summarization benchmarks are: BART reaches RougeL 44.2 on CNN/Daily Mail and 27.2 on XSum. On ComSum it reaches 33.2 on the train set and 27.2 on the test set. (Table 2)
  Scope: Single BART configuration, no error bars because training was repeated only once; CNN/DM and XSum figures are the ones reported by Lewis et al. (2020) rather than re-run.
• Copying the commit message instead of summarizing it scores only RougeL 12.0 on ComSum, and picking a random sentence from the message scores 13.6. Extraction from the input is far from a good commit summary. (Table 2)
  Scope: Heuristic baselines on 10k training-set samples; the Subject-and-Message upper reference reaches 29.5 RougeL but is not usable for prediction because it includes the gold summary.
• A subject borrowed from a related commit by the same author in the same project within a week scores RougeL 14.6 on ComSum, rising to 15.5 when both commits are bug fixes. Topic and author style are therefore not substitutes for the summary. (Table 2)
  Scope: Related-commit and Related-Fix baselines computed on the training split, 10k samples; pairing requires same author, same project and a one-week window.
• Splitting ComSum by repository rather than by commit costs fine-tuned BART about 6 RougeL points, while the split-by-commit drop is about 1 point. The gap is therefore domain shift between repositories rather than memorization. (Section 5)
  Scope: Comparison between the main repository-level split and the minor commit-level split; the test set is also more abstractive, with coverage 0.25 versus 0.31 and density 0.86 versus 0.99.
• BART changes a ComSum commit's meaning in 10% of corrective cases, 16% of refactor cases and 35% of adaptive cases. Those rates are measured on messages that carry a distractor core term but are not of the matching commit type. (Table 4)
  Scope: Meaning judged by the commit classifiers of Amit and Feitelson (93% accuracy for corrective and refactoring, 65% for adaptive), applied to both message and summary.
• BART's highest meaning-preservation precision on any commit concept tested on ComSum is 75%. A quarter or more of summaries therefore drop the concept expressed in the message even when Rouge scores are high. (Section 6)
  Scope: Precision-like metric P(concept(model(message))) given P(concept(message)) over corrective, adaptive and refactor concepts, estimated with automatic commit classifiers rather than human labels.
• In manual labeling of 100 ComSum samples, 80% of commit subjects were proper summaries of their message. Filtering merge commits and administrative messages raises that to about 90% on the labeled sample. (Section 3.2)
  Scope: Two authors labeled independently with 82% initial agreement, rising to 99% after protocol tuning; the administrative-message heuristic has 98.9% precision and about 75% recall.
• Rouge trends on the corrective, refactor and adaptive ComSum test subsets match the general test set, with fine-tuned BART at RougeL 26.2 to 26.9 against 14.3 to 16.0 for zero-shot BART. (Table 3)
  Scope: Typed test subsets built from the commit-type classifiers; results are for BART, zero-shot BART and the Random Message Sentence heuristic only.
• ComSum introduces software commit messages as a summarization domain and argues the pairing is natural because Git and GitHub already present the one-line subject as a summary of the longer message. (Appendix B)
  Scope: English-language commits, which are about 99% of the sampled commits; the domain is specific to software development, so gains on ComSum need not transfer to news or literary summarization.
• ComSum proposes evaluating summarizers by whether the output preserves the commit's type in Swanson's corrective/adaptive/perfective taxonomy, a domain-grounded alternative to word-overlap scores like Rouge. (Section 6)
  Scope: Applicable where an automatic, high-agreement classifier for the meaning aspect exists; for commits, human agreement on bug classification is 95%.
• ComSum ships both a frozen static release and the extraction code and SQL queries, so the same pipeline can rebuild a larger dataset from later commits or with different filtering choices.
  Scope: As of the 2021 release; regeneration depends on the BigQuery GitHub schema, from which projects can disappear, so a regenerated dataset will not match the frozen one.

Common misreadings

• ComSum's high Rouge numbers for fine-tuned BART do not mean the summaries are factually correct: manual inspection found hallucinated terms, names and version numbers, such as "Merge pull request #14" for #1110.
• The meaning-preservation figures in ComSum are estimated with automatic commit classifiers, not human judgments, so cross-concept comparisons must account for classifier accuracy — 93% for corrective and refactoring but 65% for adaptive.
• ComSum is not a filtered-clean corpus: merge commits, administrative messages and generic subjects are deliberately left in, with 429K merge commits listed and filtering code provided, so users must apply the optional filters themselves.
• ComSum's release is not a moving target by accident: a static frozen version limited to pre-2021 commits is published for reproducibility alongside the extraction code, and a regenerated version will differ because projects can be deleted from the index.
• Improvements on ComSum should not be assumed to transfer to other summarization domains, since bugs and refactoring have no analogue outside software development.

Terminology in this paper

Core term

A word whose appearance in a commit message is indicative of a commit concept, such as 'bug', 'bugfix', 'error', 'fail' or 'fix' for the corrective concept.

Meaning preservation (commit summarization)

The requirement that a generated commit summary carry the same commit type as the source message, measured as the probability that the summary is classified into a concept given that the message is.

Corrective / adaptive / perfective

Swanson's 1976 commit taxonomy, in which a change is a bug fix (corrective), an added feature (adaptive), or refactoring and documentation improvement (perfective).

Corrective Commit Probability (CCP)

The estimated share of a project's commits that fix bugs; a negative estimate is treated as a sign that a repository is not a software project.

Not Preserved

The share of generated summaries whose commit meaning changed, computed as the sum of the Core-and-Concept and Not-Core-and-Concept cases on messages that carry a distractor core term without belonging to that concept.

How to cite

@article{DBLP:journals/corr/abs-2108-10763,author       = {Leshem Choshen and
                  Idan Amit},
  title        = {ComSum: Commit Messages Summarization and Meaning Preservation},
  journal      = {CoRR},
  volume       = {abs/2108.10763},
  year         = {2021},
  url          = {https://arxiv.org/abs/2108.10763},
  eprinttype    = {arXiv},
  eprint       = {2108.10763},
  timestamp    = {Fri, 27 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2108-10763.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.