genomic-benchmarks-qc
Automated quality control for genomic ML datasets. Scores the biases, duplicates and train/test leakage a classifier could exploit, and gives you a standalone HTML report plus a CSV you can put in CI.
Genomic Benchmarks
We provide
Quality‑curated genomic benchmarks for fair and reproducible ML evaluation.
The problem
A high score can mean the model learned biology — or that it found a shortcut.
If the negative sequences in a dataset are shorter than the positives, a model that measures length alone will beat a model that reads the sequence. If the classes differ in GC content, or every sequence in one class begins with the same adapter, or the test set quietly repeats the training set, the reported accuracy is measuring the dataset, not the method.
These flaws are easy to introduce and hard to spot by eye. We build the tools that find them, publish what those tools measured on the benchmarks already in common use, and standardise the datasets so that a reported number can be read against what its data makes possible.
Projects
Three pieces, one ecosystem
Measure your own dataset, get an audited one in a standard format, compare methods on it.
genomic-benchmarks-data
Standardized, metadata-rich access to 93 curated DNA and RNA datasets from seven collections. One layout, fixed splits and stable sequence IDs, with each dataset's QC results returned from the same package as the data.
Coming soon
Leaderboard
A public evaluation hub with reproducible baselines and community submissions, so that two reported numbers can actually be compared.
Coming soon
A manuscript describing the full ecosystem is in preparation.
The audit
Ten checks, 93 published benchmarks, and four that nothing was flagged on.
We pointed the tool at the benchmarks people already train on. Seven collections, curated into one format and put through the same ten checks — so these are measurements on published data, browsable per dataset, with the report behind every number.
93
Datasets audited, across seven collections
57
Flagged Fail on at least one of the ten checks
0.67
Median AU-ROC recovered by one hand-crafted feature on its own
6
Datasets where half or more of the held-out split is also in training
Four of the 92 datasets that define a training split had nothing flagged on any check. That is not a verdict on the other 88, and a Pass is not a clean bill of health: it establishes that these ten checks do not separate the classes, so every number above is a lower bound on what a learned model could recover.
The aggregate is not the useful part. What tells you what to control for is the name of the feature that separated your benchmark and the distribution it was measured on — so the audit is published per dataset, not as a headline.
Look up a dataset
- All 93, filtered and grouped by collection, task, molecule and species
- The flag on each of the ten checks, and the strongest single feature by name
- 423 standalone HTML reports, one per comparison, linked from the row
- Archived at Zenodo with a DOI, so a flag stays citable
Catalogue coming soon
See what a report looks like →
Reading it in a browser is one of two routes to the same measurements. The other is genomic-benchmarks-data, which ships these datasets and returns their QC results to your code.
Score your own data
Check a dataset in two commands
Point gb-qc at your
sequences and give it somewhere to write. Requires Python 3.12 or newer.
terminal
# install pip install genomic-benchmarks-qc # compare your classes gb-qc evaluate-classes \ --input train.csv \ --out-folder qc-out # check test-set leakage gb-qc evaluate-splits \ --train-input train.csv \ --test-input test.csv \ --out-folder qc-out
What it looks for
- Sequence length distributions that differ between classes
- GC content, nucleotide and dinucleotide composition
- Per-position give-aways, such as a shared adapter or a class-specific base
- Duplicate sequences within and between classes
- Near-duplicate overlap between train and test, via MMseqs2 similarity search
How results are flagged
Each check is scored by the AU-ROC of a classifier that sees only that one feature.
| Flag | AU-ROC | Meaning |
|---|---|---|
| Pass | ≤ 0.6 | Classes not separable by this feature |
| Warning | ≤ 0.7 | A model could get some traction here |
| Fail | > 0.7 | Significant bias detected |
Use the audited corpus
Or start from a dataset that has already been checked
The same 93 datasets, in code: one format, fixed splits, stable sequence IDs, and the audit's results returned alongside the data rather than looked up somewhere else.
python
# pip install genomic-benchmarks-data import genomic_benchmarks_data as gbd # a split, checked against its own sha256 train = gbd.load_dataset( "genomic-benchmarks", "human_ocr_ensembl", "train") # what the audit found, per dataset and check qc = gbd.load_qc_results("class") # only where every scored check returned Pass gbd.filter_datasets_by_qc("class", flag="Pass")
What comes with it
- 93 datasets from seven collections, in one layout with one metadata record each
- Fixed splits and stable sequence IDs, so two papers can report on the same rows
- The audit’s flags and probe scores for every dataset, from the same call
- Republished only where the licence allows it; where the licensor declined, a hash of each split and a pointer upstream
Selecting on a flag is a choice, not a recommendation. A Pass says these ten checks did not separate the classes — it does not say nothing else will.
Coming soon
Get involved
Bug reports, feature requests and pull requests are all welcome. Open an issue on the relevant repository, or write to us — especially if you have a dataset you would like to see checked or included.