Article Source

• Reducing the Dimension of Language; A Spectral Perspective on Transformers

Reducing the Dimension of Language; A Spectral Perspective on Transformers

Abstract

Can we build neural architectures that go beyond Transformers by leveraging principles from dynamical systems? In this talk, we’ll introduce a novel approach to sequence modeling that draws inspiration from the paradigm of online control of dynamical systems to achieve long-range memory, fast inference, and provable robustness.