If you are evaluating a computational protein design tool, whether a generative model, a fitness predictor, or a scoring function, the most honest check you can run is a retrospective validation on held-out data from your own protein system. Not on benchmark datasets from the model developer. Not on published DMS results from proteins you do not work with. On your own assay data, for your own target, held back from the model before you generate predictions.
The output of that validation is typically a rank correlation metric, Spearman's rho being the standard choice for this application. And the number you get requires careful interpretation. A Spearman rho of 0.45 on your held-out data is not straightforwardly "good" or "bad" without context. This article is about how to read that context correctly.
Why Rank Correlation and Not RMSE
First, a quick note on why Spearman's rho is the right metric for this problem. You are not trying to build a model that predicts absolute activity values. You are trying to use computational predictions to rank candidate sequences, so that the ones you prioritize for assay are more likely to be functional. The question is: does the model's rank order match the experimental rank order?
Root-mean-square error (RMSE) between predicted and observed fitness values would measure something different: how well the model predicts absolute values on whatever scale you measured. This is less useful because the scale of your assay readout (HTRF counts, kcat/Km in per-mM-per-second, IC50 in nM) is arbitrary relative to the model's internal score. A model that perfectly ranks all sequences but assigns them scores on a completely different scale would have terrible RMSE and perfect Spearman rho. For prioritization, Spearman rho is what you want to maximize.
What the Rho Value Is Actually Measuring
Spearman's rho on a held-out set tells you specifically: for the variants in your held-out set, which were sampled from whatever experimental design you ran, does the model rank the better variants higher than the worse ones? This is highly dependent on the specific variant set you used for validation.
If your held-out set was a random sample from a combinatorial library covering many positions, the model is being evaluated on its ability to distinguish across a broad, diverse set of variants. If your held-out set is a collection of closely related single-mutants near a known good sequence, the model is being evaluated on a narrow local patch of sequence space. The same underlying model can show rho of 0.65 on the first type of dataset and 0.30 on the second, not because it is inconsistent, but because the two datasets test different aspects of the model's behavior.
This is why we always ask to see the experimental design of the validation set before interpreting a reported rho. A rho from a narrow local dataset mostly tells you the model can distinguish the best-of-the-best from similar sequences, which is useful but limited. A rho from a diverse dataset is a stronger signal of general utility.
The Baseline Question
Before you interpret your rho as meaningful, you need a baseline. Two baselines are worth computing.
The first is a null model: what rho do you get from random ranking? For random rankings, the expected Spearman rho is zero, but there is variance that depends on the size of the held-out set. With 50 data points, the 95th percentile of random ranking rho is around 0.23. A model reporting rho of 0.25 on a 50-variant held-out set is essentially indistinguishable from random. The same rho on 500 variants is a meaningful signal.
The second baseline is simpler heuristics: does the model outperform a simple conservation score? If you compute a per-position conservation score from your MSA and rank variants by how far each mutation deviates from the consensus, that gives you a naive baseline that costs nothing to compute. A model that cannot beat this baseline, after accounting for variance due to dataset size, is not adding value beyond what you already have.
The Coverage Problem
A well-calibrated model on your held-out set tells you something useful, but it tells you about the past. Your held-out data is drawn from the same sequence distribution as your training data. What you actually care about is the model's predictive performance on variants you have not seen yet, specifically the candidates it proposes for your next round of synthesis.
This is the coverage problem. Generative models are designed to propose sequences that are not in your training data. In fact, that is their entire value proposition. But retrospective validation on held-out historical data does not tell you whether the model extrapolates well into regions of sequence space it has not seen. The two can diverge substantially.
We are not saying retrospective validation is useless because of this. We are saying it has a specific scope: it tells you whether the model has learned the fitness patterns within the region of sequence space you have already explored. Whether it reliably extrapolates beyond that region requires prospective validation, which means commissioning a round of model-proposed variants, running them through assay, and comparing the predicted and observed ranks. That is more expensive but it is the only way to assess extrapolation quality.
Interpreting Failure Modes
When retrospective validation returns a low or near-zero rho, there are a few distinct reasons that are worth distinguishing before drawing conclusions about the model.
One is genuine model failure: the model's fitness scores do not track your assay readout because it has not learned the right features for this protein family. This is fixable by fine-tuning on your assay data if you have enough of it, or by switching to a model with better coverage of your protein family.
A second is assay noise. If your experimental replicate CVs are above 20 to 30 percent, even a perfect model would show low rank correlation because the assay itself is not reliably ordering the variants. Before concluding the model is bad, check your assay reproducibility. A scatter plot of replicate measurements against each other will tell you quickly whether the signal is there to be learned from.
A third is range restriction. If your held-out set was collected from a narrow fitness range, most variants cluster within measurement noise of each other, and ranking within a narrow range is inherently unreliable. The model may perform well on the full range but look poor on a range-restricted subset.
A fourth is a distributional mismatch between your held-out set and the kinds of variants the model is designed to score. If the model was developed for single-residue substitutions and your held-out set contains many double and triple mutants, the model may not generalize well, not because it is bad, but because it is being evaluated outside its intended scope.
What a Good Validation Protocol Looks Like
For a protein engineering team thinking about adopting a computational design tool, here is the protocol we recommend before committing to a campaign.
First, set aside at least 30 to 50 variants from your most recent screen as a held-out set before running predictions. Do not show these to the model during any fine-tuning or calibration steps. If you do not have recent assay data, use older campaign data that predates any computational involvement.
Second, run predictions on the held-out set and compute Spearman rho. Compute the null baseline and the conservation-score baseline. If the model does not clear both baselines at p less than 0.05, pause before running a full campaign.
Third, look at the scatter plot, not just the summary statistic. Does the rank correlation break down in a particular region of fitness space? Models often perform better at the extremes (very good and very bad variants) than in the middle. If your campaign is about finding the top 5 percent, strong rank correlation at the high end matters more than global rho.
Fourth, run a small prospective validation: propose 20 to 30 model-recommended variants for your next round, run them, and compare to model predictions. This is the only way to see how the model performs on extrapolation. A one-round prospective validation before committing to a full 300-variant campaign saves real money.
Rank correlation on held-out data is a necessary check, not a sufficient one. It confirms the model has learned from your past data. Whether it will improve your future screens is a question only prospective data can answer.