Lightning Talk: FlexAttention + FlashAttention-4: Fast and Flexible - Driss Guessous, Meta

This talk features Driss Guessous from Meta, who discusses the integration of FlexAttention with FlashAttention-4 to enhance performance in attention research. FlexAttention has allowed researchers to prototype custom attention variants in PyTorch but has faced throughput challenges compared to FlashAttention-3. The speaker details the innovative use of CuTeDSL to optimize performance on Blackwell GPUs, resulting in significant speed improvements across various workloads. Key topics include how PyTorch's Inductor generates modifications for FlashAttention-4 and practical advice for users looking to adopt this new backend.