The problem. If Geneformer showed pretraining can work for single cells, the next question is how, what objective, what tokenization, and does the design choice change which tasks benefit?

The idea. scGPT pretrains a generative transformer on 33 million+ human cells (from CELLxGENE). Where Geneformer ranks genes, scGPT bins expression values into tokens and trains with a generative, masked-attention objective adapted for the fact that genes aren’t a sequence, the model predicts held-out expression from the rest of the cell. One backbone then serves cell-type annotation, batch integration, multi-omic integration (RNA + ATAC + protein), perturbation-response prediction, and gene-network inference.

Why it matters. Read back-to-back with Geneformer, this is the clean comparison: rank encoding + masked-rank objective vs. value binning + generative objective, both at ~30M cells. That contrast is the useful part, it isolates what the pretraining recipe buys you, separate from just “more data.” For someone who works with both LLMs and single-cell data, it’s the clearest bridge between the two: the same generative-pretraining idea, ported to a genes-per-cell matrix.

Verdict. Impressive scope, and I’ll believe the multi-omic and perturbation results are the most genuinely new. But the skeptic’s read still applies: independent benchmarks have found single-cell FMs matching but not clearly beating scVI-class methods on core tasks, and results can be sensitive to fine-tuning and preprocessing. My takeaway isn’t “which model wins” but “which tasks actually need a foundation model”, and that answer is still being written.