Variant-Effect ACMG/AMP Evidence Benchmark

Challenge

Genomic foundation models (AlphaGenome, Evo 2) and alignment-likelihood models (GPN-MSA) are usually benchmarked against classical variant-effect predictors using bare correlation or AUROC leaderboards. That doesn't tell a clinical geneticist whether a score is strong enough to use as PP3/BP4 evidence under the ACMG/AMP variant-classification framework. AlphaMissense and CADD already have published, ClinGen-vetted evidence thresholds (Bergquist et al. 2025, Pejaver et al. 2022); the three foundation models don't.

Approach

Curated ~10,000 ClinVar SNVs (≥1-star review status, temporal split: calibration snapshot clinvar_20250715, held-out snapshot clinvar_20260720), split into missense and non-coding tracks using ClinVar's own molecular-consequence field. Scored all five tools. Applied published ClinGen thresholds directly to AlphaMissense and CADD. For the three foundation models, independently derived thresholds using Pejaver et al. 2022's local posterior-probability calibration method, ported from their MATLAB reference implementation, using a gene-region-restricted gnomAD v4.1 reference sample to estimate each track's disease-variant prior. Every derived threshold was validated on the held-out set only — never reusing calibration data.

What we found

• Missense track: AlphaMissense's specialization holds, but narrowly — AUROC 0.962 vs. CADD's 0.945 and GPN-MSA's 0.941. CADD actually edges out AlphaMissense on the fraction of pathogenic variants reaching real PP3 evidence (70.0% vs. 69.1%).
• Non-coding track: contrary to the usual framing in the literature, AlphaGenome and Evo 2 do not outperform CADD here. CADD's AUROC (0.983) is the best of the four tools tested; AlphaGenome, the model most often positioned as the non-coding/regulatory leader, has the lowest (0.909).
• GPN-MSA calibrated the most robustly of the three foundation models: its non-coding Moderate PP3 tier held on held-out data with an observed likelihood ratio of 47.0 against a required 6.36, and its Strong-evidence bin captured only 0.0056% of an independent gnomAD reference sample versus a 3.38% estimated population prior.
• Evo 2 and AlphaGenome reliably reached only Moderate/Supporting evidence tiers on held-out data; some Strong-tier point estimates existed in calibration but their bootstrap-discounted values were unstable (NaN), reported as such rather than smoothed over.

Limits of this demonstration

This is a reproduction of Pejaver et al.'s calibration methodology on our own curated data, not new ClinGen guidance — not benchmarked against a client baseline yet, and the derived thresholds should never be presented as official ACMG/AMP recommendations. PP3/BP4 was formally validated around missense variant prediction; applying it to non-coding/regulatory/splice variants, as the entire non-coding track here does, is closer to the research frontier than settled clinical practice. CADD's non-coding thresholds are extrapolated from a missense-only ClinGen calibration (no published calibration of CADD on non-coding variants exists) — its strong non-coding AUROC is a real, honestly-reported finding, but the evidence-tier percentages it's scored against are out-of-scope thresholds, not an independently validated non-coding calibration. AlphaGenome's score is a median(|raw_score|) aggregation over gene-restricted RNA-seq/splicing tracks, not a hand-picked gene+tissue combination the way DeepMind's own published examples use, which likely understates its real ceiling. Two known simplifications versus Pejaver et al.'s exact method: no gene-panel restriction (they required ≥1 pathogenic variant per gene) and no explicit tool-training-set exclusion.