Structure targets for small protein language models
Eleven 7.4M-parameter protein encoders, identical except for the pretraining target, compared on secondary structure and fold retrieval. Regressing frozen ESM-IF1 structure embeddings with a linear head beat masked language modelling; predicting the same targets through a JEPA-style predictor did not.