1. Introduction
Was there an AI compiler in 2017 that could discover FlashAttention?
Transformers were invented that year. FlashAttention arrived in 2022. For five years the most valuable optimization in deep learning sat in plain sight and every AI compiler in the world walked straight past it. Not because the math was hidden: computing softmax against a running normalizer was published in 2018, and by the end of 2021 someone had already used exactly that to tile attention and cut its memory cost, in public, with code. The ingredients were on the shelf.
What was missing was the part no compiler does. Someone had to look at the memory hierarchy, decide that traffic between HBM and SRAM was bottlenecking the performance, and restructure the computation so the attention matrix never touches HBM at all. That is not a schedule, a tiling, or a loop order. It is a different kernel, and Tri Dao had to write it.
Hand that kernel to the compiler and it will gladly do the rest: tile the loops, tune the launch configuration, enumerate the variants, measure which one wins. That is what compilers are for. What none of them could do was arrive at the kernel.
So where does the next novel optimization come from?
FlashAttention is not a special case. Many of the optimizations that matter most in inference today arrived the same way. Collapsing a transformer layer into a single kernel. Storing the KV cache in fixed-size blocks so memory doesn’t fragment. Restructuring an expert-routing step so it stops serializing. These are engineering judgements, and a human is the one to make them. The state of things was taken for granted, but very recently the laws have changed.
Getting a model to run at all is easy now. Making it fast costs engineer-days per kernel, and that bill arrives again for every model architecture and again for every hardware generation. AI compilers can convert human-discovered optimizations to an explicit rule that expands the search space. They fuse elementwise chains, plan memory, choose layouts, explore the schedules they were taught to explore, and then they stop. Everything past that point waits on judgement nobody has encoded. The engineers who can supply it are among the scarcest in the industry, which means most models in production run well below hardware limits.
Hoid is a bet that this has stopped being a human bottleneck. It is a compiler whose optimizer is an agent: it reads a profile of a model graph, forms a hypothesis about what is slow and why, proposes an optimization and confirms it. A proposed rewrite has to agree numerically with the graph it replaces, and it has to make the whole graph measurably faster. What survives both is kept, what doesn’t is thrown away. The loop moves on and tries something else, unattended, for hours.
In this blog we will introduce the philosophy of designing AI compiler in the age of agents.
2. Hoid’s design
Hoid consists of two pieces with a deliberately narrow interface between them, AI compiler infra and an agentic harness. AI infra builds model graphs, turns them into GPU kernels, executes them, and serializes them to a file you can read. Agentic harness drives an agent that rewrites that file. Everything the agent does (profile this graph, replace this region, prove the replacement is correct and faster) happens through that boundary, which means the optimizer and the verifier never share assumptions.
2.1 An IR built for agents
Hoid represents a model as a directed acyclic graph (DAG) of tensor operators, each one a self-contained function with declared inputs, outputs, and shapes. An agent reading a graph of named operations can see dependencies, shapes, and layouts at a glance, and a rewrite becomes a local substitution: replace this region with this kernel, leave everything outside it untouched.
One level down, where those operators actually become kernels, the conventional answer is to call into a vendor library, and libraries lag on two axes at once. New model architectures arrive faster than kernels get written for them, and new hardware arrives faster than existing kernels get tuned for it. The result is that the fast path available to you is usually the one someone tuned for last year’s architecture on last year’s hardware. The gap widens exactly when a generation is new, which is precisely when the performance matters most.
If the agent is proposing optimizations instead of a human, we should rethink the intermediate representation (IR) from first principles. We can do more than the standard graph rewrites of the IR. An IR can be an object that agents can read, rewrite and extend. Following are the first principles in our design.
The graph is a document. A model is a complete, static description of the computation, with every operator, dependency, and shape written down, and it serializes to a file. Nothing has to run for that description to exist. That file is everything the agent touches: it reads the model, rewrites a region, and hands it back, so every change arrives as a diff you can read before it reaches a GPU.
A rewrite has a declared interface. Rewrite defines what region to override and how. Deterministic checks for correctness and speed are automatically triggered when rewrite is proposed. This is the design decision that significantly reduces the number of reward hacking instances.
Custom kernels are first-class citizens. Built-in nodes cover the usual arithmetic, comparisons, reductions, matmul, and gather/scatter, plus the operations inference actually leans on: embeddings, RMSNorm, RoPE, softmax. The more important property is how new ones arrive. A specialized kernel enters through the same rewrite mechanism the agent uses for everything else, so the vendor-library lag above stops being something you wait out.
Together these turn the graph into something more than a description of a model. It becomes an optimization surface: inspect it, find the bottleneck, rewrite the region, compile it, measure it on real hardware, keep the result if it won. And because a node can map to a vendor library, a generated kernel, or something written by hand for one specific shape on one specific card, none of that workflow changes as the implementations underneath it grow more specialized.
2.2 Where Hoid sits relative to existing compilers
Why couldn’t an existing compiler have found FlashAttention back in 2017? XLA, TVM, and Inductor are each very good at the class of transformation they were designed to automate, and each structurally unable to reach the class FlashAttention belongs to. All three now get attention performance by calling a kernel a human wrote.
These systems differ enormously in mechanism, and they agree on one thing: who does the optimizing. In every case it is the compiler itself. XLA’s rewrite passes hold that role. Halide and TVM hand it to a schedule: the loop order, tile sizes, and memory placement for a computation whose math is already fixed. Written by hand or found by search, a schedule rearranges the work but computation remains fixed. Polyhedral compilers give it to an affine solver that will apply nothing it cannot first prove. MLIR-based compilers and Tinygrad rely on graph rewrites as a primitive for optimization. Luminal builds and searches through the search space using e-graphs.
They also share a consequence. If the optimizer is the compiler, then a human’s kernel
is the one part of the program the compiler has nothing to say about. The graph can hold
it, as a custom_call in XLA or a Triton kernel in torch.compile, and the
compiler can schedule around it, hand it inputs, even glue a small operation onto its
edge. What it cannot do is discover a fully novel, model and hardware specific rewrite for that exact scenario.
Traditional compilers generate candidate optimizations based on the prewritten rules. Benchmarking all of them is impossible, so a heuristic or a trained cost model predicts which ones will win. Rewrites nobody carved out will never be considered. The search space is large but shallow.
Hoid solves this by using an agent to navigate the search space. Each proposed candidate is reasoned about and much more likely to be the best solution, and therefore it is worth benchmarking each one. With an agent as the main driver, you get the high quality candidates from the search space by design.
Additionally, because a rewritable region has a well-defined interface, any proposed change can be checked against the graph it replaced under identical conditions, in isolation, before it is accepted. A compiler normally earns the right to transform your program by only ever attempting transformations it can prove are safe, which keeps it honest and also keeps it inside whatever its proof system can describe. Hoid takes the opposite position: any implementation is allowed to be proposed, and none is allowed in without being checked. Correctness stops being a property of the transformation language and becomes a property of the loop the harness runs.
3. Results
For each model, Hoid’s loop ran unattended against the graph and kept only the
rewrites that passed the numerical check and made the whole forward faster. The
kernels that survived (CUDA, Triton and CuTe DSL) were then exported out of Hoid as
plain torch custom ops and DSL functions, and dropped into an otherwise
unmodified PyTorch model. Every number below comes from that pure-PyTorch model; no
Hoid runtime is involved at measurement time. The models, kernels and benchmark
scripts are published in
hoid-ai/hoid-optimized-models,
one pinned uv environment per model, so the comparison can be re-run with two
commands.
The baseline is not eager PyTorch. For each model a best_torch.py script sweeps a
lattice of stock configurations (eager SDPA, torch.compile default,
reduce-overhead, max-autotune, static KV caches, CUDA graphs where they apply)
and the Hoid stack is compared against the best stock result per metric.
Everything was measured on one NVIDIA B200
(CUDA 13.0, torch 2.13, bf16); the kernels target Blackwell (sm100).
3.1 Serving LLMs
Three decoder-only models were benchmarked on single-user serving scenario: a small dense model (Qwen3-4B), a mid-size dense model (Llama-3.1-8B) and a mixture-of-experts model with 3B active parameters (Qwen3-30B-A3B).
3.2 Beyond language models
The same loop was pointed at three workloads with nothing in common with an LLM decoder: the UNet and VAE decoder of SDXL, the encoder and decoder of Whisper-large-v3, and the BGE-M3 embedding model at full occupancy.
4. Where this leaves us
Every one of these designs was built under the constraint that a compiler could only automate what someone could first reduce to a rule, a search space, or a proof. Everything else, the judgement calls, the algorithmic insight, the willingness to read a profile and rethink an operation from the memory hierarchy up, had to come from a human expert. This was a real limit software for compiler design.
Hoid just moved that limit. Agents can now do the work that used to sit on the human side of it, and they can do it at a scale no team can match!