Last week’s Post-Transformer debate post raised one question: Can long term memory become part of the architecture?
It points to one promising mathematical idea behind Post Transformer AI: Linear attention in high dimension with persistent state.
In a standard Transformer,
