KEP-SVGP

(ICML 2024) PyTorch implementation of KEP-SVGP attention mechanism available on OpenReview.

Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes

by Yingyi Chen, Qinghua Tao, Francesco Tonin, Johan A.K. Suykens

[arXiv] [PDF] [Video] [Poster] [Project Page]

Figure 1. An illustration of canonical self-attention and our KEP-SVGP attention in one layer.

If our project is helpful for your research, please consider citing:

@inproceedings{chen2024self,
  title={Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes},
  author={Chen, Yingyi and Tao, Qinghua and Tonin, Francesco and Suykens, Johan A.K.},
  booktitle={International Conference on Machine Learning},
  year={2024}
}

Table of Content

Please refer to different folders for detailed experiment instructions. Note that we specified different environments for different tasks.

Please feel free to contact [email protected] for any discussion.

• 1. CIFAR
• 2. CoLA
• 3. IMDB

Acknowledgement

This repository is based on the official codes of CIFAR: SGPA, OpenMix, ViT-CIFAR, CoLA: SGPA, huggingface, IMDB: pytorch-sentiment-analysis, text.