Beyond Transformers: State Space Models as the Next Paradigm in AI - Badri Patro

This talk delves into state space models (SSMs), which are emerging as effective alternatives to transformer-based architectures for modeling long-range dependencies in sequential data. The speaker explains how SSMs leverage linear recurrence relations and convolutional structures, leading to increased efficiency and scalability compared to the quadratic scaling of transformer attention mechanisms. Attendees will gain insights into innovations such as Structured State Space Sequence models (S4), Mamba, and Hyena, which enhance high-performance learning across various modalities including natural language, vision, and time series. This session highlights the potential of state space models to redefine AI paradigms, offering a balanced approach to accuracy, efficiency, and scalability beyond traditional transformer models.