evaluate_classes

The class-comparison evaluation. gb-qc evaluate-classes is a thin wrapper over run, so anything the CLI can do is callable directly.

evaluate_classes

Compare sequence characteristics between the classes of one dataset.

Reads the input as a set of classes - one FASTA file per class, or one CSV/TSV label column - computes per-class statistics, and reports on every pair of classes: a feature that separates two classes tells a model which class a sequence belongs to without it having to learn anything about the biology.

run is the entry point; the CLI is a thin wrapper around it. The report layout is defined in genomic_benchmarks_qc.utils.naming.

run

run(
    input,
    format,
    out_folder='.',
    sequence_column=None,
    label_column='label',
    label_list=None,
    regression=False,
    report_types=None,
    end_position=None,
    min_coverage=DEFAULT_MIN_COVERAGE,
    plot_type='boxen',
    log_level='INFO',
    log_file=None,
)

Compare every pair of classes in a dataset and write the reports.

Reports go to '/class//vs/', one directory per compared pair. FASTA inputs have no sequence column and use 'sequence', so the layout is the same for every input format. Any grouping above that - collection, dataset, split - is left to the caller, who expresses it through out_folder.

Parameters:

Name	Type	Description	Default
input	list[str]	Paths to the input files. For FASTA, one file per class. For CSV/TSV, the files are read together and the classes come from label_column.	required
format	str	Format of the input files (fasta, csv, csv.gz, tsv, tsv.gz).	required
out_folder	str	Path to the output folder; reports go into '/class/'. Default: '.'.	'.'
sequence_column	list[str] | None	Columns holding the sequences, for CSV/TSV input. Each is compared separately, and all of them together in an extra 'merged' report. Default: ['sequence'].	None
label_column	str	Column holding the class of each row, for CSV/TSV input. Default: 'label'.	'label'
label_list	list[str] | None	Classes to compare, for CSV/TSV input, or "infer" to take them from label_column. Inference stops at MAX_INFERRED_LABELS classes, past which the column is taken to be something other than a label column; an explicit list is not capped. Default: ['infer'].	None
regression	bool | None	Treat label_column as a continuous target and split it at the median into two classes. Raises ValueError if that does not produce two non-empty classes. Default: False.	False
report_types	list[str] | None	Types of reports to generate, from REPORT_TYPES. Default: ['html', 'simple'].	None
end_position	int | None	Last position the per-position checks report on, 1-based and inclusive. Defaults to the last position at least MIN_SEQUENCES_PER_REPORTED_POSITION sequences of each class reach, whichever class runs out first. It cannot widen what gets flagged, so an explicit value only ever trims. Default: None.	None
min_coverage	float	Fraction of each class that must reach a position before it may be flagged, on top of the MIN_SEQUENCES_PER_CLASS sequences every compared position needs. This is also the window the per-position figures draw; positions past it are reported as Unknown. Default: 0.25 (DEFAULT_MIN_COVERAGE).	DEFAULT_MIN_COVERAGE
plot_type	str | None	Distribution plot for the length and content figures, 'boxen' or 'violin'. 'boxen' reads better on large classes. Default: 'boxen'.	'boxen'
log_level	str | None	Logging level. Default: 'INFO'.	'INFO'
log_file	str | None	Path to a log file. Logs go to the console either way.	None

Source code in src/genomic_benchmarks_qc/evaluate_classes.py

def run(input: list[str],
        format: str,
        out_folder: str = '.',
        sequence_column: list[str] | None = None,
        label_column: str = 'label',
        label_list: list[str] | None = None,
        regression: bool | None = False,
        report_types: list[str] | None = None,
        end_position: int | None = None,
        min_coverage: float = DEFAULT_MIN_COVERAGE,
        plot_type: str | None = 'boxen',
        log_level: str | None = 'INFO',
        log_file: str | None = None
    ):
    """Compare every pair of classes in a dataset and write the reports.

    Reports go to '<out_folder>/class/<column>/<classA>_vs_<classB>/', one
    directory per compared pair. FASTA inputs have no sequence column and use
    'sequence', so the layout is the same for every input format. Any grouping
    above that - collection, dataset, split - is left to the caller, who
    expresses it through `out_folder`.

    Args:
        input: Paths to the input files. For FASTA, one file per class. For CSV/TSV, the
            files are read together and the classes come from `label_column`.
        format: Format of the input files (fasta, csv, csv.gz, tsv, tsv.gz).
        out_folder: Path to the output folder; reports go into '<out_folder>/class/'.
            Default: `'.'`.
        sequence_column: Columns holding the sequences, for CSV/TSV input. Each is
            compared separately, and all of them together in an extra 'merged' report.
            Default: `['sequence']`.
        label_column: Column holding the class of each row, for CSV/TSV input.
            Default: `'label'`.
        label_list: Classes to compare, for CSV/TSV input, or "infer" to take them from
            `label_column`. Inference stops at
            [MAX_INFERRED_LABELS][genomic_benchmarks_qc.evaluate_classes.MAX_INFERRED_LABELS]
            classes, past which the column is taken to be something other than a label
            column; an explicit list is not capped. Default: `['infer']`.
        regression: Treat `label_column` as a continuous target and split it at the
            median into two classes. Raises ValueError if that does not produce two
            non-empty classes. Default: `False`.
        report_types: Types of reports to generate, from
            [REPORT_TYPES][genomic_benchmarks_qc.evaluate_classes.REPORT_TYPES].
            Default: `['html', 'simple']`.
        end_position: Last position the per-position checks report on, 1-based and
            inclusive. Defaults to the last position at least
            [MIN_SEQUENCES_PER_REPORTED_POSITION][genomic_benchmarks_qc.utils.seq_stats.MIN_SEQUENCES_PER_REPORTED_POSITION]
            sequences of each class reach, whichever class runs out first. It cannot
            widen what gets flagged, so an explicit value only ever trims.
            Default: `None`.
        min_coverage: Fraction of each class that must reach a position before it may be
            flagged, on top of the
            [MIN_SEQUENCES_PER_CLASS][genomic_benchmarks_qc.utils.testing.MIN_SEQUENCES_PER_CLASS]
            sequences every compared position needs. This is also the window the
            per-position figures draw; positions past it are reported as Unknown.
            Default: `0.25`
            ([DEFAULT_MIN_COVERAGE][genomic_benchmarks_qc.utils.seq_stats.DEFAULT_MIN_COVERAGE]).
        plot_type: Distribution plot for the length and content figures, 'boxen' or
            'violin'. 'boxen' reads better on large classes. Default: `'boxen'`.
        log_level: Logging level. Default: `'INFO'`.
        log_file: Path to a log file. Logs go to the console either way.
    """

    if sequence_column is None:
        sequence_column = ['sequence']
    if report_types is None:
        report_types = ['html', 'simple']
    if label_list is None:
        label_list = ['infer']
    validate_report_types(report_types, REPORT_TYPES, 'evaluate-classes')

    setup_logger(log_level, log_file)
    logger.info("Starting classes evaluation.")

    ensure_directory(out_folder)

    with log_failures("Classes evaluation"):
        # we have multiple fasta files with one label each
        if format.startswith('fa'):
            _evaluate_fasta_classes(input, out_folder, report_types, plot_type, end_position,
                                    min_coverage)
        # we have CSV/TSV
        else:
            _evaluate_table_classes(input, format, out_folder, sequence_column, label_column,
                                    label_list, regression, report_types, plot_type, end_position,
                                    min_coverage)

    logger.info("Classes evaluation successfully completed.")

run_analysis

run_analysis(
    input_statistics, report_dir, report_types, plot_type
)

Analyse each class, then every pair of classes, writing reports under report_dir.

Layout produced, one directory per comparison so that every report type has a fixed, predictable name inside it:

<report_dir>/
    per-class/<class>.json
    <classA>_vs_<classB>/
        gb-qc-report.csv
        gb-qc-report.html
        gb-qc-duplicates.txt
        plots/

Classes are compared in the order they arrive, which run has already sorted by path name.

Source code in src/genomic_benchmarks_qc/evaluate_classes.py

def run_analysis(input_statistics, report_dir, report_types, plot_type):
    """Analyse each class, then every pair of classes, writing reports under `report_dir`.

    Layout produced, one directory per comparison so that every report type has
    a fixed, predictable name inside it:

        <report_dir>/
            per-class/<class>.json
            <classA>_vs_<classB>/
                gb-qc-report.csv
                gb-qc-report.html
                gb-qc-duplicates.txt
                plots/

    Classes are compared in the order they arrive, which `run` has already
    sorted by path name.
    """
    if report_types is None:
        report_types = ['html', 'simple']

    report_dir = Path(report_dir)

    # run individual analysis
    for s in input_statistics:
        stats, _ = s.compute()

        if "json" in report_types:
            per_class_dir = report_dir / PER_CLASS_DIR
            per_class_dir.mkdir(parents=True, exist_ok=True)
            generate_json_report(stats, per_class_dir / f'{s.slug}.json')

    if len(input_statistics) < 2:
        return

    # run pair comparison analysis with all combinations
    for stat1, stat2 in combinations(input_statistics, 2):
        comparison_dir = report_dir / comparison_dirname(stat1.slug, stat2.slug)

        if stat1.seq_column is not None:
            logger.info(f"Comparing classes for sequence column: {stat1.seq_column}")
        logger.info(f"Comparing classes: {stat1.label} vs {stat2.label}")

        logger.debug(f"Running significant differences analysis for {comparison_dir}.")
        results, failed_by_feature = flag_significant_differences(
            stat1, stat2
        )

        # Only create the directory for report types that were actually asked for,
        # so a json-only run does not leave empty comparison directories behind.
        if 'simple' in report_types or 'html' in report_types:
            comparison_dir.mkdir(parents=True, exist_ok=True)

        if 'simple' in report_types:
            generate_simple_report(results, comparison_dir / SIMPLE_REPORT_FILE)

        if 'html' in report_types:
            # Convert results dict to DataFrame using all available result fields
            results_df = pd.DataFrame.from_dict(results, orient='index')
            generate_dataset_html_report(
                stat1, stat2,
                comparison_dir / HTML_REPORT_FILE,
                plots_path=comparison_dir / PLOTS_DIR,
                plot_type=plot_type,
                results=results_df,
                failed_by_feature=failed_by_feature
            )

REPORT_TYPES module-attribute

REPORT_TYPES = ('json', 'html', 'simple')

MAX_INFERRED_LABELS module-attribute

MAX_INFERRED_LABELS = 50