Single-Cell Perturbation Benchmark: Foundation Models vs. Simple Baselines

Challenge

Two 2025 papers disagree about whether deep-learning and foundation models actually beat simple linear baselines at predicting single-cell gene expression after a genetic perturbation. Ahlmann-Eltze, Huber & Anders (Nature Methods, 2025) say no; a follow-up bioRxiv preprint (Oct. 2025) says the "no" result is an artifact of uncalibrated metrics. Reproducing both sides independently, on the same real data, required training/fine-tuning three different model families (a specialized GNN, a fine-tuned transformer foundation model, and a zero-shot foundation model) against three baselines, under two different metric regimes.

Approach

Both datasets were loaded through GEARS's own PertData class using its documented unseen-perturbation train/val/test split, not a custom split, to preserve comparability with published numbers. Three baselines (no-change, mean, ridge-on-gene-identity), GEARS (official default hyperparameters), and scGPT (fine-tuned from its pretrained "whole-human" checkpoint) were evaluated under two conventional metrics — MSE on the top-20 differentially expressed genes and Pearson correlation of the expression change from control. Geneformer ran zero-shot in silico perturbation, which produces a different kind of output (a per-cell embedding cosine-shift, not a predicted expression profile) and is reported separately rather than forced into the same columns. A calibration layer on top of the conventional metrics compared each model against the gap between a no-signal negative control and an "interpolated duplicate" positive control built from random split-half pseudo-replicates of each condition's own held-out cells — a scoped-down version of the rebuttal paper's calibration idea, not its full 13-metric framework.

What we found

The two datasets told different stories. On Adamson (single-gene perturbations), scGPT clearly won on both metrics. On Norman (combinatorial perturbations), there was no single winner: GEARS had the best (lowest) MSE@20DEG of all five models, but ridge — the simplest baseline — had the best Pearson correlation, beating both foundation/specialized models on that metric. Recalibrating against the positive/negative control gap did not change either dataset's ranking — the calibration framework mostly confirmed the raw-metric story rather than overturning it, a real, honest finding rather than a confirmation of the rebuttal paper's headline argument.

Limits of this demonstration

Not benchmarked against a client baseline or production system — this is an independent reproduction on public data, not a client engagement. The calibration framework's positive control uses random split-half pseudo-replicates because the source data has no true biological-replicate metadata, a real, documented narrowing of the rebuttal paper's original method. Geneformer's output isn't directly comparable to the other four models under the same metrics, by design of its own public API, not a limitation this project introduced. scGPT is evaluated on a slightly different real test-condition split than the other four models due to a cell-gears version incompatibility between environments — each model is scored on its own correct split, not a shared canonical subset.