Public Datasets for Code Understanding Tasks August, 2026

Labeled data is typically scarce for code-understanding tasks, both for training and evaluation purposes. In this post, I share a number of datasets that ML/SE researchers have collected and (kindly) shared. I will keep this post up-to-date as I find new datasets. Please email me if some dataset is missing.

I first put this list up in December 2021, when it fit in two short sections. Almost everything below arrived after that, and the shape of the field changed along with it. The datasets worth using now are mostly repository-level, executable, and multi-language, so I have grouped them by task instead of by language. Last updated August 2026.

Issue resolving and agentic tasks

  • SWE-bench: real GitHub issues paired with the repository's own test suite. A patch counts only if those tests pass. site, code, paper
  • SWE-bench Verified: the human-filtered subset, and the split most papers report on. Worth understanding how it was filtered before comparing numbers against the full set. dataset
  • Multi-SWE-bench: the same task in seven languages besides Python, which is useful when you suspect a result is really a result about Python. code, paper
  • SWE-Gym: executable environments meant for training issue-resolving agents rather than for scoring them. code, paper
  • SWE-smith: synthesized task instances at scale, again on the training side. site, paper
  • SWT-bench: the same repositories with the task inverted. The agent writes the test that reproduces the bug instead of the patch that fixes it. code, paper
  • Commit0: build a library from scratch against its specification and test suite. site, paper

Bugs and program repair

  • QuixBugs: 40 small programs with single-line bugs, in both Python and Java. Still the fastest way to sanity-check a repair tool. site, paper
  • BugSwarm: reproducible fail-pass pairs mined from CI logs. dataset, paper
  • BugsInPy: real bugs from 17 Python projects with tests, in the spirit of Defects4J. code, paper
  • refactory: student Python assignments with reference fixes. code, paper
  • XCorpus: Java programs with executable harnesses. code, paper
  • GitBug-Java: recent Java bugs that still build, which is rarer than it sounds. code, paper
  • RunBugRun: repair tasks in several languages, executable, so candidate fixes are checked by running them rather than by matching a reference patch. code, paper
  • Defectors: Python defect prediction at file and line granularity. dataset, paper
  • DeepBugs for Python: a Kaggle mirror of name-based bug detection data. dataset

Generation and execution

  • EvalPlus (HumanEval+, MBPP+): the original benchmarks with far more tests. Several published pass rates drop once you run these instead, which says something about the original suites. code, paper
  • BigCodeBench: function calls across many libraries, with instructions more involved than a docstring. site, paper
  • LiveCodeBench: contest problems tagged with release dates, so evaluation can be restricted to problems published after a model's training cutoff. The most practical answer to contamination that I know of. site, paper
  • CRUXEval: predict a function's output from its input, and its input from its output. Execution reasoning rather than generation. site, paper
  • DS-1000: data science problems over pandas, numpy and friends, drawn from real StackOverflow questions. site, paper
  • MultiPL-E: HumanEval and MBPP translated into more than a dozen languages. site, paper
  • ClassEval: class-level generation instead of standalone functions. code, paper
  • CodeContests: competitive programming problems with many tests each, from the AlphaCode paper. code, paper
  • HumanEvalPack: HumanEval extended to synthesis, explanation, and repair across six languages. dataset, paper

Repository context and completion

  • RepoBench: completion where the useful context lives in other files. code, paper
  • CrossCodeEval: cross-file completion in Python, Java, TypeScript, and C#. site, paper
  • Long Code Arena: a set of long-context tasks, including commit message generation, bug localization, and module summarization. site, paper
  • DevEval: manually annotated against real repository structure and dependencies. code, paper

Vulnerabilities

  • DiverseVul: vulnerable functions from a wide set of projects. The label-quality analysis is arguably more useful than the data. code, paper
  • PrimeVul: cleaner labels and a paired setting. Detectors that looked strong on older datasets do considerably worse here, which is the point of the paper. code, paper
  • ARVO: reproducible OSS-Fuzz vulnerabilities, each with a crashing input and the fixing commit. code, paper
  • CVEfixes: CVEs joined to their fixing commits across many projects. code, paper
  • CyberSecEval: insecure code generation and cyberattack helpfulness, from Meta's Purple Llama. code, paper

Pretraining corpora

  • The Stack v2: the permissively licensed corpus behind StarCoder2. dataset, paper
  • CommitPack: 4TB of git commits across 350 languages, which is the closest thing available to a corpus of code edits rather than code snapshots. dataset, paper

One habit I would recommend to anyone picking from this list: write down the construction date of the dataset next to the training cutoff of whatever model you plan to evaluate, and do it before you run anything. Most of the entries above predate the models people now test on them. The ones that handle this explicitly, like LiveCodeBench and GitBug-Java, are the exception rather than the rule.