The audit
Ten checks, 93 published benchmarks, and four that nothing was flagged on.
We pointed
genomic-benchmarks-qc
at the benchmarks people already train on.
Seven collections, curated into one format and put through the
same ten checks — so these are measurements on published
data, browsable per dataset, with the report behind every number.
- 93 Datasets audited, across seven collections Curated from 147; 54 could not be put through the same ten checks
- 57 Flagged Fail on at least one of the ten checks 31 rolled up to Warning, 5 to Pass
- 0.67 Median AUROC recovered by one hand-crafted feature on its own 34 of the 93 probed datasets reach 0.70 or more from one feature
- 0.80 Median AUROC when those same features are read together rather than one at a time 46 of 92 reach 0.80 or more, including six no check flagged Fail
The corpus
All 93 datasets
One row per dataset and one column per check, with everything else the run measured available beside it. Tick a column to bring it in; sort any column; filter by any facet; open a row for what the checks returned on it. Whatever you filter, sort and put on screen is in this page's address, so a view of these rows is something you can send someone.
These rows
What the checks returned, and how far one feature got
Both figures describe the rows the table above is showing rather than the whole corpus, so every filter narrows them with it. Narrow to one collection and the shape of that collection is on screen; narrow to one species, or to the datasets one feature already separates, and so is that.
The first is also a control. Selecting a segment filters the table to the datasets it counts — the same filter the checks page links in with — and selecting it again takes the filter off.
Every mark is a measurement on that dataset’s reference split. An AUROC is what one hand-crafted feature reached on its own, which is a lower bound on what is recoverable from the sequences and never an upper one: a row that sits to the left is a feature that did not separate these classes, not a set of datasets a model would do badly on. How to read a probe AUROC.
Reading the grid
How results are flagged
Six of the ten checks ask one narrow question: could a classifier that saw only this one feature tell the two classes apart? Nothing is trained. The feature is the score, so its AUROC answers directly — 0.5 is a feature that says nothing about the classes, 1.0 is one that settles them on its own — and a flag is a boundary drawn on that number.
| Flag | AUROC | Read it as |
|---|---|---|
| Pass | ≤ 0.60 | The classes are not separable along this feature |
| Warning | 0.60 – 0.70 | A model could get some traction here |
| Fail | > 0.70 | Separable from this one feature alone |
| Unknown | — | The comparison was not made, which is not a Pass |
Two guards sit in front of the scoring, and a check that trips one returns Unknown rather than a number: nothing is scored on fewer than 250 sequences per class, and a per-position comparison stops where fewer than a quarter of the class is long enough to reach that position — past there the sequences being compared are a sample of the long ones rather than of the class. What a check reports is the worst of its sub-checks and not their average, so a Fail on per-position nucleotide content can be one base at one position out of hundreds, which is what the report behind it is for.
Neither the three integrity checks nor the leakage scan is scored that way. Unique bases asks whether both classes use the same alphabet; the two duplicate checks are thresholds on how much of a class survives deduplication, and on whether any sequence appears under both labels; and the leakage scan counts the sequences of one split that are also present in another.
That is the short version. How a flag is decided gives the rules in full, including the simulations the 0.60 boundary comes from. What each check returned across this corpus, and which split every number here was measured on, is on the checks page.
Go deeper
-
The ten checks
What each check measures, what it returned across the corpus, the whole 93-by-10 grid — and the boundaries, which you can move to see which flags depend on them.
-
The Core Set
Five criteria and one selection, the 10 datasets that came through — and the six no check flagged that the same features recover together.
What identifies this run, and what a citation of it has to name