[LAMPS] Attention-based PCA

Résumé

We study attention as an unsupervised method for PCA. For Gaussian data, we show that both softmax and linear attention learn the leading principal direction of the covariance matrix. This holds in both infinite- and finite-prompt regimes, up to sampling effects, and also extends to an in-context setting with spiked covariances. Overall, the work gives a theoretical explanation for how attention can perform PCA-like representation learning.

Intervenant

Rodrigo Maulen-Soto Postdoctoral researcher, Université de Toulouse Paul Sabatier – IMT
https://rodrigomaulen.github.io/