Kinference is a new open source control plane from Buoyant that sends each AI request to the vLLM server that can answer it with the least work. If you run your own models on Kubernetes, it aims to cut wait times without adding GPUs, though every number so far comes from Buoyant’s own tests.
Here are the facts a skimmer needs:
- What it is: an inference control plane for vLLM on Kubernetes
- Who made it: Buoyant, announced on the vLLM blog on November 4, 2026
- License: Apache 2.0, with source on GitHub
- Works with: clients that speak the OpenAI or Anthropic API format
- Catch: the benchmarks are vendor-run and cover one workload type
Those five lines are the whole story in short. The rest of this post explains what the router actually does and whether it’s worth a weekend of your time.
What Kinference does that a plain load balancer can’t
A standard Kubernetes load balancer spreads requests across your model servers without knowing what each one is holding. That works fine for web pages. It works less well for language models.
Here’s why. A vLLM server keeps a KV cache, which is stored work from prompt text it has already processed. If your next request starts with the same text, a server with that cache can skip a lot of effort. A blind load balancer sends it to whichever server is next in line.
Kinference looks at three things before it picks a worker: GPU queue sizes, how each worker has performed historically, and what’s in its KV cache. It then routes the request to the worker that will do the least work. According to the vLLM blog post, it also adds token rate limiting, per-tenant usage and cost tracking, and inference policies.
If you’re already self-hosting open models, like the ones in our looks at ERNIE 5.5 Flash Base or Recursive Flint, this is the layer that sits in front of them once more than a few people start sending requests.
The Kinference numbers, and why to read them carefully
Buoyant compared Kinference with standard Kubernetes load balancing (kube-proxy) and published these results:
| Test | Result | Workload |
|---|---|---|
| Median time to first token | 95% lower | Unloaded, one workload type |
| Median end-to-end latency | 45% lower | Unloaded, one workload type |
| Goodput (requests that meet the latency target) | About 7.4x higher | Shared-prefix prompts |
| Goodput | About 3.5x higher | Multiturn chats |
| Latency target met | At concurrency 32 (shared-prefix) and 8 (multiturn) | kube-proxy never met it |
The big gains show up when prompts share a long opening, such as a fixed system prompt or a growing chat history. That’s exactly where cache-aware routing should help. If every request in your app is unique, expect much less.
Treat these as a reason to test, not a promise. The numbers are vendor-run, and the post doesn’t list any limitations. Run your own traffic through it before you believe them.
Policies you can write in YAML
Kinference’s rules are InferencePolicy resources, so they live in YAML next to the rest of your cluster config. The examples in the announcement cover three jobs:
- Semantic routing: send simple prompts to a cheaper model and hard ones to a bigger one
- Bursting to a cloud provider when your own GPUs are saturated
- Keeping requests flagged as containing personal data on your own hardware
That last one is the interesting policy for teams with privacy rules. It lets you use a cloud fallback for traffic spikes without letting sensitive prompts leave the building.
How to try Kinference on your own cluster
You need a Kubernetes cluster with kubectl and Helm, plus access to the OCI Helm chart registries. The guide uses Gateway API CRDs version 1.5.1. It doesn’t state GPU or Kubernetes version requirements.
The install runs in four steps:
- Install the Gateway API CRDs.
- Install the gateway CRDs and gateway charts with Helm.
- Install the Kinference CRDs and chart, with
extension.enabled=true. - Check that the pods are running with
kubectl get pods.
The guide installs into the default namespace, so override the namespace settings if you want another. Buoyant also warns that these steps will change, so use the docs at kinference.ai as your reference rather than copying commands from this post.
Frequently Asked Questions
Is Kinference free to use?
Yes. Buoyant says it’s fully open source under Apache 2.0, and the code is on GitHub at Kinference-AI/kinference.
Do I need Kubernetes to use Kinference?
Yes. It’s built as a control plane for vLLM on Kubernetes, and the install guide assumes a cluster with Helm and kubectl.
Will Kinference make my chatbot faster?
Possibly. The published gains came from shared-prefix and multiturn workloads on one test setup. Apps with mostly unique prompts may see less.
Can my existing app talk to it?
Clients send OpenAI- or Anthropic-compatible traffic to its HTTP endpoint, so apps that already use those formats shouldn’t need a rewrite.
Is Kinference ready for production?
The announcement doesn’t say, and it warns the install steps will change. Try it in staging first.
Our take on Kinference
If you run vLLM for a team and your prompts share long prefixes, Kinference is worth a staging test this month. Cache-aware routing is a sensible idea, and the Apache license means trying it costs you nothing but time. If you only run a model on one machine, or you use hosted APIs, skip it. Wait for independent benchmarks before you plan hardware around the 7.4x figure.
Leave a Reply