New paper! Sinkformers: Transformers with Doubly Stochastic Attention
We use Sinkhorn instead of SoftMax to make attention doubly stochastic. It promotes a democratic principle.
With @PierreAblin, @mblondel_ml & @gabrielpeyre
Paper (AISTATS 🥳): arxiv.org/abs/2110.11773
1/8
Very proud of this paper accepted #NeurIPS2022 🥳
"Do ResNets discretize Neural ODEs?"
arxiv.org/abs/2205.14612
w. @PierreAblin@gabrielpeyre 🙏
We study the convergence of ResNets to Neural ODEs and train ResNets with a discrete adjoint method.
1/9
Space complexity for training Transformers becomes huge as the sequence/batch size increases.
These memory requirements can be significantly reduced with a Momentum ResNet version of any Transformer.
With momentumnet: github.com/michaelsdr/mom…
Tuto: colab.research.google.com/drive/1zAyNz2m…
1/4
🥳🥳 New work: arxiv.org/abs/2309.01213
Implicit Regularization of ResNets towards Neural ODEs
w. @PierreMari0n, Yu-Han Wu and @gerardbiau
We show: ResNet initialized as discretization of a neural ODE -> such a discretization holds throughout training.
Tomorrow (Wednesday) at #NeurIPS2022 we'll be presenting our paper "Do ResNets discretize Neural ODEs?", a joint work with @PierreAblin and @gabrielpeyre.
Come to Hall J, #642 from 11am to 1pm if you want to know more about this work 😺
Paper: arxiv.org/abs/2205.14612
Very proud of this paper accepted #NeurIPS2022 🥳
"Do ResNets discretize Neural ODEs?"
arxiv.org/abs/2205.14612
w. @PierreAblin@gabrielpeyre 🙏
We study the convergence of ResNets to Neural ODEs and train ResNets with a discrete adjoint method.
1/9
👋👋🇦🇹🇦🇹 Tomorrow we present our poster at ICLR on Implicit Regularization of Deep ResNets towards Neural ODEs.
👉4:30 pm spot #210 👈
w. @PierreMari0n, YuHan Wu, @gerardbiaux.com/m_e_sander/sta…
🥳🥳 New work: arxiv.org/abs/2309.01213
Implicit Regularization of ResNets towards Neural ODEs
w. @PierreMari0n, Yu-Han Wu and @gerardbiau
We show: ResNet initialized as discretization of a neural ODE -> such a discretization holds throughout training.
