Polynomial Autoencoder Beats PCA on Transformer Embeddings

Ever tried to squeeze a high-dimensional BERT embedding into a 2D plane for a pretty plot? If so, you know PCA is like a Swiss Army knife: works, but not always perfectly. And now researchers have come up with a new trick — a polynomial autoencoder.

The idea is simple: instead of a linear transformation (like PCA), use polynomial layers in an autoencoder. This captures nonlinear dependencies in the data. The result is more accurate compression of transformer embeddings, especially when the original dimensionality is high (and it is, oh boy, it is).

Of course, there's a catch: the polynomial autoencoder takes longer to train than PCA, which just computes eigenvectors. But on test tasks (e.g., visualization or clustering), it shows better metrics. So if you have time to wait, you can get a plot that makes your colleagues' jaws drop.

METABYTE studio comment: We also like when a model works better, but if your autoencoder trains longer than a sprint lasts, maybe you should give us a call. We know how to speed up neural nets without losing quality (and without polynomial shamanism).