Post

2/6 Their key innovation is query-biased attention, where the model weights MSA sequences by relevance to the query. We tested whether "relevance" maps onto phylogenetic distance by correlating sequence weights with evolutionary distances across thousands of protein families.

3/6 The model consistently upweights evolutionarily close sequences and downweights distant ones, despite never being trained on this. There's also a division of labor across layers: layer 11 acts as a strong phylogenetic filter, while others contribute more modestly.

4/6 But surprisingly, encoding phylogeny ≠ leveraging it. Comparing learned weights against uniform averaging for contact prediction, the aggregate improvement was marginal. Per-MSA, uniform averaging outperformed learned weights about half the time, sometimes by a large margin.