Together AI Launches Together Compiler: AI Agents Now Retune Inference Overnight

Together AI launched Together Compiler on November 4, 2026, an agent-driven system that continuously retunes inference for new open-source models, claiming 1.3 to 3 times the tokens per second of optimized baselines within 24 hours. It is the first project of the company's Polymorphic Inference initiative, is already running on many of its serverless endpoints, and is available to other customers only through a limited beta for now.

WHAT HAPPENEDWhat Happened

Together AI announced Together Compiler on November 4, 2026, a system that uses teams of AI agents to rewrite the GPU code behind the open-source models it serves. New open-source models now get faster overnight instead of over weeks, the company said, with no additional work from customers. Together Compiler is the first project of a larger initiative the company calls Polymorphic Inference, a form of recursive self-improvement for inference workflows.

The problem it targets is the pace of change in inference itself: Together AI said 80% of OpenRouter traffic goes to models released in the last three months, and each of the ten most popular open models on OpenRouter uses a different architecture. Its answer is continuous compilation: agents take the traffic shape, model and hardware as inputs, generate and measure candidate optimizations, and hot-reload the winning forward passes without restarting the full engine. The company claims the design runs 252 times more experiments than existing systems that use agents for inference optimization, and that it reaches state-of-the-art optimizations within 24 hours.

In Together AI's reported tests, the compiler reached 1.5 times and 1.4 times higher peak throughput than the best result on SemiAnalysis's InferenceX benchmark for Kimi K3 and GLM 5.3 respectively. On Llama 1B it passed 2,100 tokens per second and ran up to 1.7 times faster than hand-written megakernels, including Together AI's own, on NVIDIA H100 and B200 GPUs. MiniMax M3.1 Flash reached 1,111 tokens per second on B200 hardware, 1.8 times its baseline, and DeepSeek v4.1 Flash reached 1,205 tokens per second on GB300 hardware, 1.3 times its baseline. Unite.AI's same-day coverage independently summarized the headline claim as 1.3 to 3 times the tokens per second of highly optimized open-source baselines in under 24 hours.

WHY IT MATTERSThe bottleneck moved from the model to the serving stack

When a new open model lands every few weeks with a different architecture, hand-tuning the serving code for each one cannot keep up. Together AI's bet is that the tuning itself becomes an agent workload: measure real traffic, search for faster kernels, and ship the winner while the model is still current.

If that loop works in production, the competitive edge in inference shifts from who writes the best kernel once to who runs the fastest search continuously. The 24-hour claim is the one to watch, because it reframes optimization from a launch event into an ongoing service.

WHY IT MATTERSVendor numbers set the claim, the beta sets the proof

Every speed figure in this announcement comes from Together AI's own tests, and the baselines are the company's chosen comparisons. The Phonic customer quote, a nearly 50% cut in median time to first speech overnight, is the only outside voice in the release, and it describes one deployment, not a general result.

The limited beta for dedicated deployments is therefore the real test. Whether customers reproduce anything near 1.3 to 3 times throughput on their own models, traffic and hardware, while passing the same quality checks, will decide whether Polymorphic Inference is a product category or a launch narrative.

SIGNAL JUDGMENT — ANALYSISInference optimization becomes a service, not a launch task

Tokenta Signal's judgment: Together Compiler matters less as a single speed claim and more as a shift in where inference work happens. Model releases now outpace manual tuning, so a system that searches for faster kernels against live traffic, every day, attacks the real bottleneck. The claim to discount is the headline multiplier, which is vendor-measured; the claim to test is the loop, and the limited beta will supply that test. If it holds on customer hardware, every inference provider will need a continuous-compilation answer.

WHAT REMAINS UNCERTAINWhat Remains Uncertain

EVIDENCEEvidence

Together AI's November 4, 2026 BusinessWire announcement was opened and verified directly: Together Compiler uses teams of AI agents to rewrite GPU code, runs 252 times more experiments than existing agentic inference-optimization systems, and is the first project of the Polymorphic Inference initiative. Unite.AI's same-day coverage independently confirms the announcement and its headline 1.3x to 3x tokens-per-second claim.

ORIGINAL: Together AI launches agent-tuned inference compiler ↗
businesswire.com · 2026-11-04 · BusinessWire announcement

PRIMARY SOURCESPrimary Sources

• BusinessWire: Together AI Launches Together Compiler for Continual Optimization of Open Source AI Models
  https://www.businesswire.com/news/home/20261104879153/en/Together-AI-Launches-Together-Compiler-for-Continual-Optimization-of-Open-Source-AI-Models
• Unite.AI: Together AI Brings Continuous Compilation to Production Inference
  https://www.unite.ai/together-ai-brings-continuous-compilation-to-production-inference/

RELATEDRelated Intelligence

• Microsoft Winds Down Its Oracle AI Cloud Rental as Older Nvidia Chips Fall Behind ISSUE 256
• An Unreleased Anthropic Model Proved an 18-Year-Old Quantum Conjecture, Machine-Checked in Lean ISSUE 254
• IBM Unveils Quantum System Two in Chicago, Its Most Advanced Computer Yet, Powered by Nighthawk ISSUE 253
• Vistra Posts $1.14B Q3 Profit as Power EBITDA Jumps Over 50%, Lifts 2026 Outlook ISSUE 251

CORRECTIONS: if a fact on this page is wrong, we keep a correction record on the page. Contact team@tokenta.space