Together Compiler Beats Megakernels by 28%

Together Compiler Beats Megakernels by 28%
Tuesday, November 17, 2026
  • •Together Compiler reaches 2,118 tokens/s on Llama 3.2 1B with B200, 28% above hand-written megakernels
  • •CUDA Graphs and PDL recovered 98% of megakernel performance on default Torch Inductor kernels
  • •Inductor’s 607 tuning flags and isolated benchmarks produced workload-specific slowdowns and kernel-selection errors

Together AI says Together Compiler uses AI agents to rewrite and fuse GPU kernels in a model’s forward pass, reaching 2,118 tokens/s on Llama 3.2 1B with a B200 GPU. The company reports that this is 28% faster than its hand-written megakernels, which reached 1,649 tokens/s. The compiler grew out of experiments aimed at making inference optimization work across changing model architectures, batch sizes, sequence lengths and hardware.

Last year, Together helped release megakernels: single, highly fused GPU kernels that ran an entire model forward pass. They reached record speeds, but production work yielded one model, one batch size and one customer after months of effort. The team then found that CUDA Graphs (a way to reduce GPU kernel launch overhead) and PDL, layered on default Torch Inductor kernels, recovered 98% of megakernel performance without writing a kernel. Adding multi-kernels and agent loops beat the previous megakernels by up to 1.7x, exceeding 1,100 TPS for Llama 1B on H100.

The development followed an investigation into Torch Compile, PyTorch’s just-in-time compiler, and its Inductor component. On the team’s Llama 1B workload, Inductor delivered 669 TPS. The researchers found that its tuning choices depend heavily on workload: enabling weight freezing improved throughput by 10–26% at larger batch sizes but caused a 49% slowdown at batch size 1. As of Q3 2026, Inductor exposed 607 tuning flags, including 198 added in the previous year; combinations could change results, with aggressive fusion alone costing 6 TPS but improving throughput by 3 TPS when paired with three other flags.

The team also found that isolated kernel benchmarks did not reliably predict inference performance. Inductor’s benchmarks measured some kernels 39–58% slower than during a live forward pass, while zero-filled token IDs made an embedding lookup 2.1× faster than random IDs for a 64K-token sequence. Benchmarking overhead added about 3.6 µs in some measurements. On Llama 1B, Inductor chose a Triton output-projection kernel that took 7.4 µs, versus 6.4 µs for cuBLAS in live inference, a 15% slowdown. On GLM 5.3, random inputs made MoE grouped-GEMM timings 4–21% slower than tests with realistic inputs.

Inductor’s greedy, pairwise fusion also missed opportunities that require considering GPU utilization. In one Llama 1B example, a six-CTA activation kernel left most of the GPU’s 132 streaming multiprocessors idle before a separate indexing kernel ran; the compiler rejected combining them because they shared no memory accesses. Together’s stated takeaway was that fusion should account for utilization, while benchmarking should use real input data inside a forward pass. The article’s opening describes Together Compiler as the resulting framework for continual inference optimization, built after two years of work on GPU programming and optimization systems.

Subscribe to the AIB Newsletter

The fastest, most reliable AI news, straight to your inbox

Companies related to this article