Genomic Benchmarks audit
GitHub

Screened on their inputs

The Core Set

Five criteria applied in order to the 93 measured datasets, then one selection among the 25 that met all five. Ten came out. A dataset that did not is not thereby unusable — the criteria say which checks returned Fail on it, and what the same checks’ features recovered together, and nothing more.

These 10 in the catalogue, where they sort against the other 93 · What the joint reading is · What each of the ten checks returned

The funnel

What each step removed

Core Set selectionA funnel of 6 gates narrowing 93 datasets to 10, with the count removed at each gate hatched.datasets remaining025507593included in the audit931. evaluablereference split = 'train', leakage scanpresent92 (−1)2. integrityflag != Fail in any split65 (−27)3. leakageflag != Fail in any split pair (< 2% ofheld-out sequences)54 (−11)4. separabilitystrongest-probe AUROC <= 0.7 in every split31 (−23)5. jointjoint QC-feature AUROC < 0.8 on every held-outsplit25 (−6)6. diversity selectiontarget size 1010 (−15)removed by this gate

The 6 gates that narrow the 93 included datasets to the 10 of the Core Set. Each gate is applied to what the one above it left.

The gates are selection criteria, not a ranking. A dataset removed at a gate is one this audit measured a flag on, and nothing more than that.

Table view
The Core Set gates in order.
GateRequirementThresholdInRemovedOut
1. evaluableDefines the reference split and was scanned for leakagereference split = 'train', leakage scan present93192
2. integrityNo Fail on unique bases, duplicate sequences between or within labelsflag != Fail in any split922765
3. leakageNo Fail on the train-test leakage scanflag != Fail in any split pair (< 2% of held-out sequences)651154
4. separabilityNo Fail on any single-feature probestrongest-probe AUROC <= 0.7 in every split542331
5. jointThe QC features jointly do not recover the labeljoint QC-feature AUROC < 0.8 on every held-out split31625
6. diversity selectiongreedy facet coverage, at most 2 per collectiontarget size 10251510
The six steps in the order applied, with the threshold each one used. The first five are criteria a dataset either meets or does not. The last is not: it picks among the 25 that met all five, so no dataset is excluded by it for anything it was measured on.
Step Criterion Requirement Threshold In Removed Out
1 evaluable Defines the reference split and was scanned for leakage reference split = 'train', leakage scan present 93 1 92
2 integrity No Fail on unique bases, duplicate sequences between or within labels flag != Fail in any split 92 27 65
3 leakage No Fail on the train-test leakage scan flag != Fail in any split pair (< 2% of held-out sequences) 65 11 54
4 separability No Fail on any single-feature probe strongest-probe AUROC <= 0.7 in every split 54 23 31
5 joint The QC features jointly do not recover the label joint QC-feature AUROC < 0.8 on every held-out split 31 6 25
9 of the 25 admitted joint values are lower bounds (position window capped or training pool subsampled), which for admission is the anticonservative direction; 21 of the 92 modelled datasets carry one.
6 diversity selection greedy facet coverage, at most 2 per collection target size 10 25 15 10

The fifth criterion

The checks read together

Nine of the ten checks reduce a sequence to one feature and ask whether that feature alone separates the classes. The fifth criterion asks something the ten cannot answer between them: hand a model all of those same features at once — length, GC, mononucleotide and dinucleotide content, and the two per-position profiles — and score it on a held-out split. Nothing is added that the checks do not already look at. The only change is that they are read jointly rather than one at a time.

Over the 92 datasets a held-out split could be scored on, the median is 0.80, and 46 reach 0.80 or more. The criterion cuts at 0.80, which is that median.

Two differences from every other number on this site are worth holding on to. It is measured on a held-out split, not on the reference split everything else is measured on, because a model fitted on the training pool and scored there would report how well it memorised. And it is still a lower bound, for the same reason a probe AUROC is: these are six fixed feature blocks, and a learned model has access to feature classes none of them spans.

21 of the 92 joint figures are lower bounds, not point values: the position window had to be capped, or the training pool subsampled. Both can only move the number down, so a dataset admitted for sitting under the ceiling may in fact sit above it. Every joint figure on this site that is a lower bound says so where it is shown.

The six datasets no check returned Fail on, which the same checks’ features recovered when read together. These are the datasets step 5 removed and no earlier step did.
Dataset Task category Strongest single feature All features together Scored on
nucleotide_benchmarks:splice_sites_donors RNA Processing Prediction 0.685 0.997 test
GUE_v2:transcription_factor_prediction_mouse_1 Molecular Interaction Prediction 0.530 0.935 test
GUE_v2:transcription_factor_prediction_mouse_2 Molecular Interaction Prediction 0.533 0.849 test
BEACON:MeanRibosomeLoading Translation Efficiency Prediction 0.639 0.844 test
OmniGenBench:terminator_strength_leaf Sequence Annotation 0.638 0.813 test
GUE_v2:epigenetic_marks_prediction_H3K79me3 Epigenomic Prediction 0.685 0.803 test

This is the clearest thing this run has to say about what a Pass is worth. On every dataset in that table each of the ten checks returned Pass or Warning and none returned Fail; the features those checks are built from still recover the label. A row where the left figure is near 0.50 and the right is near 1.00 is not a check that misfired — each check answered the narrow question it was asked. It is the whole reason a flag bounds separability from below and never from above.

One consequence is recorded rather than smoothed over. RNA Processing Prediction has no eligible dataset anywhere in this corpus: one of its 11 audited datasets had no Fail on any check, and it does not clear the joint ceiling. So the Core Set spans six task categories and not seven, and the absent one is a finding about the corpus rather than a gap in the selection.

The members

The 10 datasets

Ranked as the selection ranked them. Every per-check figure was measured on the dataset’s reference split; the joint figure was measured on a held-out split, which is what the criterion is defined over.
Rank Dataset Task type Species Rows Worst flag Strongest probe AUROC Joint AUROC Worst split overlap
1 GUE_v2:transcription_factor_prediction_mouse_4 Binary classification Mus musculus 18,830 Pass 0.517 0.564 0.00%
2 mRNABench:rnahl-human Regression Homo sapiens 12,939 Warning 0.626 0.683≥ a lower bound 0.10%
3 genomic-benchmarks:human_ocr_ensembl Binary classification Homo sapiens 174,756 Warning 0.575 0.773≥ a lower bound 1.16%
4 BEACON:CRISPROnTarget Regression Homo sapiens 2,076 Warning 0.602 0.653 0.00%
5 GUE_v2:epigenetic_marks_prediction_H3K4me3 Binary classification Saccharomyces cerevisiae 36,799 Pass 0.586 0.626 0.00%
6 mRNABench:translation-efficiency-mouse Regression Mus musculus 10,853 Warning 0.644 0.745≥ a lower bound 0.00%
7 OmniGenBench:pro_seq_m_esculenta Binary classification Manihot esculenta 8,812 Warning 0.637 0.701 0.00%
8 miRBench:AGO2_eCLIP_Manakov2022 Binary classification Homo sapiens 2,839,760 Warning 0.533 0.737≥ a lower bound 1.30%
9 nucleotide_benchmarks:H3K9me3 Binary classification Homo sapiens 28,288 Warning 0.543 0.703 1.06%
10 OmniGenBench:promoter_strength_protoplast Regression Zea mays 9,000 Warning 0.598 0.735 0.70%

A Core Set member can still carry a Warning: the criteria screen for Fail. The strongest probe on a member still reaches an AUROC worth reading, and the joint column still reaches higher than it — both are there so they can be read.

Where to go next

What identifies this run, and what a citation of it has to name

↑ Top