Train/test leakage

evaluate-classes asks whether your two classes are distinguishable. evaluate-splits asks a different question entirely:

Is your test set already in your training set?

A model that has seen a test sequence during training scores on it for free. If enough of the test set leaks, a benchmark stops measuring generalisation and starts measuring memorisation — and it will rank a memorising model above a generalising one.

gb-qc evaluate-splits \
  --train-input examples/enhancers/data/enhancers_train.csv \
  --test-input examples/enhancers/data/enhancers_test.csv \
  --sequence-column sequence \
  --out-folder qc-out

Unlike evaluate-classes, this one needs MMseqs2 on your PATH — how to install it.

How similarity is measured

Every test sequence is searched against the whole training set with mmseqs easy-search, restricted to nucleotide search on the forward strand. Each hit gets a similarity score:

similarity = min(query_coverage, target_coverage) × percent_identity

Both halves matter, and the min is the important part:

• Percent identity alone is not enough. A 20-nucleotide stretch matching perfectly inside a 500-nucleotide sequence is 100% identical over the aligned region and tells you nothing about leakage.
• Coverage of the shorter sequence is what makes the score symmetric and honest. A short sequence fully contained in a long one is leakage — the model has seen all of it — and taking the minimum coverage catches that, where averaging the two would dilute it away.

A test sequence counts as leaked when its best hit exceeds --similarity-threshold, 90% by default. So the reported percentage is a count of test sequences with at least one near-duplicate in training, not a count of hits.

The flag is a boundary on that percentage: Pass at 0%, Warning below 2%, and Fail at 2% or more. There is no clean band above zero — any leakage at all is worth knowing about, which is why the Warning band starts at the first leaked sequence.

Queries and targets are not the same number

The report gives two percentages, and they answer different questions:

• Leaked queries — what fraction of your test set has a near-duplicate in training. This is the number that matters for your benchmark, because it bounds how much of your test score could come from memorisation.
• Leaked targets — what fraction of your training set is implicated. Almost always the smaller number, because one popular training sequence can be the match for many test sequences.

In variable-length the split is 0.80% of queries against 0.12% of targets: a small number of training sequences accounting for most of the leakage. That shape is worth noticing — it usually means a repeated family rather than diffuse overlap.

Reading the report

Three things, in the order they are useful:

1. The two percentages and the flag. Start here and decide whether to keep reading.
2. The similarity histogram. The shape tells you what kind of leakage you have. A spike at 100% is exact duplicates — a plain mistake, easily fixed. A broad hump in the 90s is a repeated sequence family, which is harder, because removing it may remove a real biological class.
3. The leaked-pair panel. Up to the first 100 pairs, each expanding to its rendered alignment, so you can see what is actually shared. Every flagged pair, including those past 100, is exported to mmseqs/mmseqs2_search_result.tsv beside the report, and the panel's count is of all of them, not of the ones listed.

Start with a small example. enhancers has four leaked pairs — you can expand every one and understand the whole finding in a minute. composition-bias at 6.04% has far too many to read individually, which is the point at which the histogram becomes more useful than the list.

What the numbers look like in practice

Measured on the bundled examples, which between them span the range:

Example	Leaked queries	Flag	What it is
hidden-motif	0.00%	Pass	Genuinely clean — worth seeing once, so you know the check can come back empty
clean-dataset	0.07%	Warning	2 sequences in 3,000. Real, and negligible
enhancers	0.67%	Warning	4 pairs. The right size to learn the report on
variable-length	0.80%	Warning	Concentrated in few training sequences
composition-bias	6.04%	Fail	One test sequence in sixteen. This one changes conclusions

Two things to take from that table. First, a small non-zero number is normal — real genomic data contains repeated families, and 0.1% is not worth acting on. Second, the interesting threshold is not the flag boundary but whether the leaked fraction is large relative to the differences between models you are comparing. If two architectures are half a point apart and 6% of the test set is leaked, the comparison is noise.

What to do about it

Exact duplicates: remove them from the test set. Uncontroversial.

Near-duplicates: the fix is to split on something other than sequence identity. Cluster the sequences first — MMseqs2 will do it — and assign whole clusters to train or test. This is standard practice in protein benchmarking and underused in genomics.

Positional overlap: if train and test were drawn from overlapping genomic windows, split by chromosome or by locus rather than by window. This is the most common cause of the broad-hump histogram.

Nothing: a legitimate answer when the leaked fraction is small and you say so in the paper. The failure mode to avoid is not knowing.

Practical notes

• Multiple sequence columns are concatenated and searched together, unlike evaluate-classes which analyses each column separately. There is no per-column split report.
• Memory. MMseqs2's prefilter structures can be large on big datasets. --split-memory-limit 10G caps them; without it there is no limit and a big run can exhaust the machine.
• Threads. --threads is passed straight through; the default is whatever MMseqs2 picks. See the note below before raising it on a conda-installed MMseqs2.
• Temporary files go into a gb-qc-mmseqs-*/ directory inside the comparison directory and are removed at the end. --keep-tmp-files keeps them and logs each path. Each run gets its own, so several runs can share one --out-folder concurrently.

Backwards alignments on conda-installed MMseqs2

If a run logs

MMSeqs2 returned 33 of 1107 alignments with their coordinates running backwards,
although the search asked for the forward strand only ...

then MMseqs2 has reported hits on the reverse strand, which evaluate-splits never asks for — it searches the forward strand only. Such a row still counts as the leak it is; where one reaches the report's listing it is shown with its scores, and the page says what happened where the alignment would have been.

Your numbers are fine. The leaked percentages, the histogram and the flag are computed from pident, qcov and tcov. On affected rows those three match a known-good build exactly; what is corrupted is the alignment — the coordinates and the aligned strings. Nothing is dropped, so a backwards hit still counts as the leak it is.

The cause is the MMseqs2 build, not your data. Conda/bioconda builds produce these intermittently when running on more than one thread — a different subset each run, and often none at all. Every bioconda release from 14 to 18 does it; the upstream precompiled binaries do not.

Two ways out, whichever suits you:

• Re-run with --threads 1. In testing this gave byte-identical output to the upstream binary across all five bundled examples — same hits, same scores, same alignments — and was deterministic run to run.
• Install MMseqs2 from the upstream precompiled release rather than from conda, and keep your threads.

The trade is yours: single-threaded search costs whatever the rest of the machine would have been worth, which on a many-core node is a lot and on a small dataset is close to nothing.