Designing High-Performance GPU Kernels with TileLang: Tensor-Core GEMM, Fused Softmax, FlashAttention, and Autotuning

A tutorial shows how to use TileLang through TVM to build and benchmark GPU kernels for vector add, tensor-core GEMM, fused epilogues, row-wise softmax, and FlashAttention.

It compares the results with PyTorch and cuBLAS, demonstrating that a high-level DSL can still produce strong CUDA kernel performance.

The tutorial also tunes kernel schedules for different GPUs, reducing manual low-level work while optimizing for speed.