The AI Gateway Landscape: agentgateway, LiteLLM, Kong and Envoy AI Gateway
published at 07-20-2026 by Max Körbächer
Throughout out recent events we have seen that AI Gateways have become the single most important entry point for companies. Because every team building with LLMs eventually hits the same issue: it starts with a single <Your Favorite AI Provider> key for one service and eighteen months later you have a dozen services, five model providers, a pile of MCP tool servers and no single place to answer "who's calling what, at what cost, under what policy."
That's where an AI gateway enters the stage. In this post I will look with you at where agentgateway fits, the cloud-native/AI-native choice and how it stacks up against LiteLLM and the other gateways platform teams are evaluating in 2026.
The two lineages
Todays AI gateways come from one of two starting points:
- LLM-request-first gateways like LiteLLM, Portkey, Bifrost, Cloudflare AI Gateway. These started from the problem of "I need one API in front of 100+ model providers" and added routing, budgets, virtual keys and observability.
- Traffic-infrastructure-first gateways like Kong AI Gateway, Envoy AI Gateway (kgateway), Apigee, F5 and agentgateway. These started from an existing (or newly built) L7 proxy/data-plane and are extending it to understand AI-specific traffic: tokens, prompts, tool calls and now agent protocols.
agentgateway sits squarely in the second camp, but with a twist: instead of bolting AI awareness onto an existing proxy, its creators build a new data plane from scratch in Rust, specifically because they concluded the first approach doesn't hold up once you add MCP and A2A traffic into the mix.
I think of it like the solution for an agentic, massive scaled system on an enterprise level.
What agentgateway actually is
agentgateway is an open-source proxy, now hosted under the Linux Foundation (via the Agentic AI Foundation), that serves primarily four kinds of traffic:
- Service traffic: plain HTTP/gRPC/TCP, the stuff a normal API gateway or Envoy handles: mTLS, OIDC, retries, timeouts.
- LLM traffic: an OpenAI-compatible front door to OpenAI, Anthropic, Bedrock, Gemini, Vertex, Azure OpenAI, and a long list of other providers, with credentialing, failover, budgets and semantic caching.
- MCP traffic: treated as a first-class, stateful protocol rather than a generic HTTP passthrough, with tool discovery, federation across multiple backend MCP servers, RBAC per tool, and audit logging.
- A2A traffic: agent-to-agent invocations across frameworks like LangChain, CrewAI, and Google ADK, with identity and tracing baked in.
Without too much geeking out, the architectural claim worth paying attention to is the MCP one. MCP is JSON-RPC over a long-lived, bidirectional session. So a client opens a connection, the server can push messages back asynchronously and a single "list tools" call might need to fan out to several backend MCP servers and merge the results into one coherent response, scoped per-session to whatever that client is actually allowed to see. Traditional gateways, including Envoy, were built around the assumption of short-lived, stateless request/response cycles, so this pattern doesn't map cleanly onto their data-plane model. The workaround the industry has largely used so far is bolting an ext_proc sidecar (often written in Go) onto Envoy to shim in the missing session logic, which one independent review of agentgateway's design called out as producing "mediocre" results in practice.
agentgateway's answer is a purpose-built Rust data plane that treats sessions as first-class citizens, reusing lessons from Solo.io's ztunnel (the sidecar-less proxy behind Istio Ambient Mesh) rather than trying to graft statefulness onto Envoy's cluster/listener model. It also uses its own xDS-derived control-plane protocol rather than reusing Envoy's Listener/Cluster/Route types, specifically so that a one-field policy change doesn't require re-pushing megabytes of route config to every proxy instance. The later can be a real operational pain point in large Envoy deployments.
Also, I just learned that, Solo.io has already moved its own kgateway project's AI data plane over from Envoy+ext_proc to agentgateway, which tells you where they think the bolt-on approach runs out of road.
What LiteLLM actually is
Now let us have a look at LiteLLM and how it comes to solve the problem from the opposite direction. It's an MIT-licensed Python library and proxy server whose core abstraction is "one OpenAI-compatible endpoint in front of 100+ (I think it’s more than 140+ actually) model providers." You run it as:
- a Python SDK, imported directly into your app, or
- a standalone Proxy Server, deployed behind a load balancer, backed by Postgres (for virtual keys, teams, and spend records) and optionally Redis (for distributed rate limits and routing state).
And it comes with a lot of features: virtual keys, per-team/per-key budgets, load balancing, automatic fallback chains, response caching, an admin UI and export hooks into Langfuse/Prometheus/OpenTelemetry. This way it has made its way up to become a default starting point for a huge number of teams. It has a very large install base and the ecosystem of tutorials, Terraform modules, and integrations around it is correspondingly deep.
But LiteLLM comes also with some tradeoffs:
- It scales as a request-router, not a stateful protocol proxy. Its MCP support exists, but MCP and A2A were added onto a design center that's still "the model completion request," not the long-lived session.
- Python's GIL and GC put a ceiling on raw throughput compared to Rust- or Go-based alternatives. This shows up in independent and vendor benchmarks alike, where LiteLLM consistently comes in behind Kong's Nginx/Lua-based data plane and behind newer Go-based entrants like Bifrost on sustained-load tests.
- Supply-chain exposure is a live operational concern. In March 2026, LiteLLM confirmed that two PyPI releases were compromised via a tainted CI dependency, shipping a credential-exfiltration payload for roughly 40 minutes before detection. It's a useful reminder that "just pip install the proxy" carries the same supply-chain surface as any other fast-moving PyPI package and it's pushed some production deployments toward pinned, signed image tags and more conservative upgrade cadences. (this can happen also with other programming languages and their ecosystem, but scripting languages facing this issue frequently).
Anyhow, I have to say none of this makes LiteLLM a bad choice. It's a mature, extremely widely deployed piece of infrastructure and for most teams whose problem really is "normalize and govern LLM completion calls," it remains one of the fastest ways to get there.
Long story short, LiteLLM and agentgateway have a different architectural bet.
Where the rest of the field lands
A few other names come up in every one of these evaluations, worth placing on the same map:
Gateway | Core abstraction | Language/data plane | MCP maturity | Where it's strongest |
agentgateway | Unified proxy for HTTP/gRPC/LLM/MCP/A2A | Rust, custom data plane (ztunnel lineage) | Native, stateful, session-aware | Kubernetes/Envoy shops that need MCP/A2A as first-class traffic, not a plugin |
LiteLLM | LLM completion request | Python (SDK + proxy), Postgres/Redis | Present, not the design center | Fastest path to multi-provider routing, budgets, and a huge ecosystem |
Kong AI Gateway | API request, AI-aware plugins | Nginx + Lua (Kong core) | Plugin-based | Orgs already standardized on Kong for API management |
Portkey | AI operations workflow | Managed/hosted (Apache-2.0 core) | One of the more mature MCP-specific offerings | Teams that want a productized dashboard: prompts, guardrails, analytics out of the box |
Envoy AI Gateway / kgateway | Kubernetes Gateway API + Envoy | Envoy (Go control plane), agentgateway as the emerging AI data plane | Historically thin; converging with agentgateway | Cloud-native/Istio shops wanting CNCF governance |
Bifrost | LLM completion request | Go | Present | Teams that want LiteLLM-like scope with lower per-request overhead |
A couple of landscape notes worth flagging because they change the perspective now in mid-2026:
- Portkey agreed to be acquired by Palo Alto Networks (announced April 30, 2026), which is likely to fold it into Prisma AIRS over time 👉 worth factoring in if you're betting on Portkey's independent roadmap.
- Envoy AI Gateway and agentgateway are converging, not competing, at least inside the Solo.io/kgateway ecosystem: kgateway (the CNCF-hosted, Envoy-based Kubernetes gateway) is adopting agentgateway as its AI-specific data plane rather than continuing to extend Envoy directly for this purpose. If you're evaluating "Envoy AI Gateway," it's worth checking whether the project you're looking at has already made this move.
- Vendor-run benchmarks (Kong's own comparison against Portkey and LiteLLM, Solo's own comparison against Envoy+ext_proc) should be read as directionally useful, but not neutral. Each was run by the project it favors, on the workload shape that flatters that project's architecture.
Practical guidance
Reach for LiteLLM (or Bifrost) if: your problem is squarely "normalize N providers behind one endpoint with budgets and fallbacks," you want the largest ecosystem and fastest path to a working proxy, and MCP/A2A traffic is a minor part of your footprint today. It’s also what we have seen very often in the wild and lead fast to reliable integrations.
Reach for Kong or Portkey if: you want a productized experience ($$$), dashboards, guardrails, prompt management and either already run Kong for API management or want a managed/hosted control plane rather than another thing to operate yourself.
Reach for agentgateway if: you're already in (or heading toward) a Kubernetes/Envoy/Istio world, MCP tool-serving and agent-to-agent traffic are core to your architecture rather than an afterthought and you want one data plane that doesn’t feel like three stitched-together gateways. For me it is the engineering and scaled choice as it integrates and handling service mesh traffic, LLM routing and agent protocols with consistent policy, identity and observability. Be aware it's genuinely younger: expect faster-moving docs, some feature/doc mismatches typical of an early-stage project and an xDS-based configuration model that has a learning curve if you've never worked with Envoy-style control planes.
Further reading
- agentgateway.dev and its GitHub repo