Post

1/6 What does MSA Pairformer learns about evolutionary relationships? @yoakiyama.bsky.social et al achieved impressive contact & variant-fitness predictions w a fraction of ESM2's parameters; we wanted to better understand the mechanism arcadia-science.github.io/2025-phyloge... 🧪🧬 🖥️

2/6 Their key innovation is query-biased attention, where the model weights MSA sequences by relevance to the query. We tested whether "relevance" maps onto phylogenetic distance by correlating sequence weights with evolutionary distances across thousands of protein families.

3/6 The model consistently upweights evolutionarily close sequences and downweights distant ones, despite never being trained on this. There's also a division of labor across layers: layer 11 acts as a strong phylogenetic filter, while others contribute more modestly.

4/6 But surprisingly, encoding phylogeny ≠ leveraging it. Comparing learned weights against uniform averaging for contact prediction, the aggregate improvement was marginal. Per-MSA, uniform averaging outperformed learned weights about half the time, sometimes by a large margin.

5/6 We traced this to layers 5-11, which integrate sequence weights blindly, regardless of whether they help or hurt. When they hurt prediction, we could progressively rescue performance by shuffling weights at earlier layers, saving the model from its own bad intuition.