SGLang refactors CUDA Graph support, with Breakable CUDA Graph becoming the default prefill solution
Blog Advanced CUDA Graph Techniques in Inference CUDA Graphs promise to remove kernel-launch overhead, but getting close to that benefit in a real inference engine requires graphing as much of the workload as possible without sacrificing compatibili... SGLang Team
AI BriefingSGLang refactored its CUDA Graph support by splitting it through runner/backend interfaces, allowing different capture strategies to be reused. The community-pioneered Breakable CUDA Graph (BCG) is now the default for prefill. Its codebase is only about one-quarter the size of the torch.compile solution (521 lines versus 1,771 lines), and it builds 3.8–5.2 times faster.
Source: LMSYS Blog · lmsys.org