1) Introduction: Why LLM Serving Breaks Traditional Infrastructure

Legacy ML serving stacks - Flask, TensorFlow Serving, Triton with basic backends - were never architected for generative models. Serve a 70B parameter model on these frameworks and you will hit OOM errors and unusable latency immediately. The problem is not scale alone; it is a fundamentally different inference paradigm with three critical bottlenecks.

GPU Memory Pressure. LLM memory consumption is dynamic. The KV cache scales linearly with sequence length and batch size. A 70B model requires either multi-GPU tensor parallelism or aggressive quantization just to load. At INT4 (~35 GB for model weights alone) it fits on a single 80 GB A100 with roughly 45 GB remaining for KV cache state - workable at moderate concurrency, but a budget that depletes fast as sequence lengths and parallel requests scale up. With context windows reaching hundreds of thousands of tokens, memory contention is the primary production constraint.

Batching Inefficiency. Autoregressive generation is sequential per-request. Naive serving gives good latency but catastrophic throughput. Static batching is equally broken: response lengths vary wildly, so shorter sequences are padded with dummy tokens to match the longest completion in the batch - burning GPU cycles on meaningless computation until the entire batch finishes.

Autoscaling and Cost. Standard signals (CPU utilization, RPS) are meaningless for GPU-bound workloads. Scaling must key off GPU memory utilization and KV cache saturation-metrics most orchestration layers do not natively expose. Get this wrong and you are either dropping requests or burning budget on idle A100s at ~$2-4/hour per GPU.

The industry's response: purpose-built LLM inference engines. TGI entered maintenance mode in late 2025. SGLang emerged for structured outputs and prefix caching. TensorRT-LLM remains unmatched for raw NVIDIA throughput.

But for teams needing the best balance of throughput, hardware support (NVIDIA + AMD), and production-grade memory management, one framework has cemented itself as the default: vLLM.

2) vLLM: The Inference Engine Standard

Core Innovations

vLLM's primary mechanism, PagedAttention, manages the KV cache like virtual memory paging in an OS - near-zero memory waste, enabling 2–4× higher throughput compared to static KV cache allocation. Continuous batching injects new requests the moment execution slots free up, eliminating idle GPU cycles from waiting on full batch completion.

From an infrastructure perspective: hardware support spans NVIDIA and AMD GPUs, it exposes an OpenAI-compatible API (drop-in replacement for existing integrations), and production-ready deployments are quickly achievable. The ecosystem is mature - first-class integration with KServe, BentoML, and Ray Serve out of the box.

This post references vLLM v0.23.0 (as of June 2026). Recent releases focused on speculative decoding optimizations, expanded multi-modal support, and improved multi-node tensor parallelism. Known limitations: edge-case memory fragmentation during prolonged uptime and configuration friction when hot-swapping multiple LoRA adapters under high concurrency.

Alternatives

TensorRT-LLM wins on peak throughput for static, NVIDIA-exclusive deployments. SGLang outperforms vLLM for complex structured outputs and multi-turn prefix reuse. Neither matches vLLM's balance of general-purpose high-concurrency serving, broad hardware support, and ecosystem integration - which is why every major deployment framework supports it natively.
That balance makes vLLM the right engine to anchor a deployment comparison.

3) Deploying vLLM in Production

The engine has been decided, now the question is how to operationalize it. Three deployment patterns dominate, each targeting a different infrastructure maturity level.

3.1) Standalone

The baseline is straightforward: vllm serve <model> or the vllm/vllm-openai Docker image behind a load balancer. The vLLM Production Stack (open-source) adds a Request Router that supersedes generic load balancers - routing by session ID or routing keys to maximize KV cache reuse across instances without application-level changes.

The appeal here is operational simplicity. There is no Kubernetes control plane required, you retain direct host-level control over GPU allocation, networking, and shared memory, and it remains the fastest path from prototype to production endpoint. For teams running on bare metal or single-node cloud instances, this eliminates an entire class of orchestration complexity.

The trade-offs are clear. There is no native metric-driven autoscaling keyed off GPU utilization or queue depth. Canary deployments, blue/green routing, and traffic shadowing are absent entirely. Model versioning and rollback remain manual processes. Observability requires assembling your own stack on top of vLLM's built-in Prometheus /metrics endpoint - functional, but undifferentiated work that scales poorly across multiple models.

This option is best suited for small teams, single-model deployments, non-K8s environments, and predictable static workloads where the operational overhead of a full orchestration layer is not justified.

3.2) KServe + vLLM

KServe abstracts model deployment through the InferenceService Custom Resource Definition, integrating vLLM into a fully declarative Kubernetes-native lifecycle. It provides what standalone deployments cannot: Knative-driven autoscaling (including scale-to-zero), native canary rollouts via traffic splitting between model revisions, concurrent model versioning, compliance with the V2 Open Inference Protocol, and built-in metrics endpoints compatible with Prometheus/Grafana observability stacks - all configured through a single YAML manifest or the KServe Python SDK.

KServe offers two deployment modes. Knative | Serverless mode provides the full feature set but introduces cold-start latency. Standard | Raw Deployment mode bypasses Knative entirely, using standard Kubernetes Deployments with HPA or KEDA for autoscaling (and no Istio requirement). In practice, most production LLM teams stay on serverless mode but set minReplicas:1 to retain traffic splitting and revision management while avoiding cold-start penalties.

The operational trade-off is significant. KServe's dependency chain is extensive: cert-manager, Istio (ingress + mesh), Knative Serving, and KServe itself - all with strict cross-version compatibility requirements. Upgrades demand synchronized rollouts across four layers. For ML teams without dedicated platform engineering support, a realistic first-time setup takes days. Debugging failed deployments means tracing requests through Istio VirtualServices, Knative route configurations, and KServe revisions simultaneously.

Cold-start latency compounds the challenge. Scaling from zero for large models incurs 30–60 second penalties (container pull, CUDA init, weight loading into VRAM). Storage I/O bottlenecks frequently surface when multiple pods pull 100GB+ weights simultaneously.

KServe is an enterprise-grade framework justified in environments requiring multi-model production systems, strict rollout governance, and advanced autoscaling - backed by a dedicated platform team. Section 4 covers the practitioner workflow in detail.

3.3) BentoML + vLLM

BentoML wraps vLLM serving logic through its Python service abstraction, consolidating model weights, dependencies, runtime configuration, and API logic into a single OCI-compliant artifact - a Bento. The workflow remains linear: bentoml build produces a deployable image that can run on Docker, Kubernetes using standard manifests and autoscaling primitives, or BentoCloud as a managed platform.

The developer experience is distinctly Python-first. Teams can create OpenAI-compatible endpoints, define request handling, preprocessing, postprocessing, routing, guardrails, or auxiliary models directly in Python rather than starting from Kubernetes manifests. This makes BentoML attractive for RAG systems, agentic workflows, and multi-stage inference pipelines where the LLM is only one component of the serving graph.

For managed production deployments, BentoCloud provides deployment automation, autoscaling, and rollout workflows. Historically, BentoML’s self-managed Kubernetes story was associated with Yatai. However, the main Yatai repository is archived, while the BentoML 1.2 / Yatai 2.0 direction appears under development rather than a clearly established production path. Teams planning self-managed Kubernetes deployments should validate its current state before relying on it.

The trade-off is that BentoML is primarily an application-level serving abstraction, not a Kubernetes-native infrastructure control plane. Capabilities such as traffic splitting, scale-to-zero, fine-grained routing, and managed rollout workflows depend on the surrounding platform. In open-source self-managed deployments, these typically require Kubernetes primitives such as HPA or KEDA, ingress or service-mesh routing, and CI/CD automation. Debugging surface area also increases, since failures may span BentoML’s execution model, vLLM, the container runtime, and Kubernetes.

This option is best suited for Python-first teams, complex multi-stage pipelines such as RAG or agents, and organizations that value unified packaging without making Kubernetes platform engineering the primary developer workflow.

3.4) Comparison Table

3.5) Also Worth Mentioning

Ray Serve + vLLM: vLLM already uses Ray for multi-GPU/multi-node tensor parallelism under the hood. Wrapping it in Ray Serve adds prefill-decode disaggregation, cache-aware routing, and Python-native inference graphs. A legitimate production alternative to KServe for teams deploying 70B+ models across multiple nodes - though it means managing a Ray cluster (via KubeRay) on top of Kubernetes.

vLLM Production Stack: vLLM's first-party Kubernetes deployment (Helm charts + Gateway API + Request Router). Offers more control than KServe with less dependency overhead, but lacks KServe's maturity in autoscaling and traffic management. Currently viable for teams wanting Kubernetes-native deployment without the Istio/Knative tax.

4) KServe in Practice

KServe's value becomes clear once you move beyond a single-model proof of concept. This section covers the two deployment interfaces and how KServe fits into a broader LLMOps platform. Code examples reference KServe 0.18.

4.1) Deploying a Model: YAML vs. SDK

Declarative YAML (GitOps)

The standard production path uses an InferenceService manifest committed to a Git repository and reconciled by ArgoCD or Flux. This ensures cluster state matches Git state, provides audit trails via commit history, and enables instant rollbacks with git revert.

The example below assumes the Knative/serverless KServe path, which is useful when teams want revision management, traffic splitting, and autoscaling semantics managed through KServe.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: vllm-llama3
  namespace: production
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 5
    scaleTarget: 1 # soft limit, can be exceeded
    scaleMetric: concurrency
    model:
      modelFormat:
        name: huggingface # using KServe HuggingFace vLLM runtime
      storageUri: hf://meta-llama/Meta-Llama-3-8B-Instruct
      args:
        - --max-model-len=4096
        - --tensor-parallel-size=1
        - --served-model-name=llama3
      resources:
        limits:
          nvidia.com/gpu: "1"
          memory: "24Gi"
      env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: huggingface-secret
              key: token

For teams preferring KServe's RawDeployment mode, the InferenceService spec remains largely identical, but autoscaling is handled via standard Kubernetes HPA with custom metrics (e.g., GPU utilization or queue depth) rather than Knative's concurrency-based scaler.

Key production considerations: secrets are pulled from Kubernetes Secrets (never hardcoded), minReplicas: 1 prevents cold starts that would otherwise take 30–60 seconds for large models, and Knative annotations drive concurrency-based autoscaling rather than CPU utilization.

In Knative/serverless mode, canary rollouts require only adding canaryTrafficPercent: 10 to the predictor spec and updating the storageUri to the new model version. KServe creates a new Knative Revision, splits traffic at the ingress layer and lets both versions scale independently. For large vLLM deployments, keep the canary and stable revisions warm with appropriate minReplicas values to avoid weight-loading cold starts during rollout validation. Once latency, error rate, GPU saturation, and model-quality metrics confirm stability, promote the new revision to 100% or roll it back to 0%.

Python SDK (Programmatic)

The KServe Python SDK is the right choice when model metadata is dynamic - injected from upstream training jobs, orchestrated by Kubeflow Pipelines or Airflow, or exposed through an internal developer portal.

from kubernetes import client
from kserve import KServeClient, V1beta1InferenceService, V1beta1InferenceServiceSpec, V1beta1PredictorSpec, V1beta1ModelSpec, V1beta1ModelFormat, constants

isvc = V1beta1InferenceService(
    api_version=constants.KSERVE_V1BETA1,
    kind=constants.KSERVE_KIND,
    metadata=client.V1ObjectMeta(
        name="vllm-llama3",
        namespace="default",
    ),
    spec=V1beta1InferenceServiceSpec(
        predictor=V1beta1PredictorSpec(
            min_replicas=1,
            max_replicas=3,
            model=V1beta1ModelSpec(
                model_format=V1beta1ModelFormat(name="vLLM"),
                storage_uri="hf://meta-llama/Meta-Llama-3-8B-Instruct",
                resources=client.V1ResourceRequirements(
                    limits={"nvidia.com/gpu": "1", "memory": "24Gi"}
                ),
                args=["--max-model-len=4096", "--dtype=auto"]
            )
        )
    )
)

KServeClient().create(isvc)

The hybrid pattern most mature teams adopt: use the SDK in dev/staging for rapid experimentation, then generate the equivalent YAML manifest and commit it to a GitOps repository for the production rollout. This gives data scientists velocity without sacrificing infrastructure auditability.

4.2) KServe in a Broader LLMOps Platform

KServe is the inference data plane - it serves models. But model serving is one stage in a lifecycle that spans training, evaluation, deployment, and monitoring. In a mature LLMOps platform, KServe integrates with four surrounding layers:

Model Registry → KServe (Artifact Handoff). KServe's Storage Initializer pulls weights from wherever they live - hf://, s3://, mlflow://, or OCI registries. The registry is the immutable source of truth; KServe simply consumes it. A growing trend is packaging LLM weights as OCI artifacts, letting KServe pull models from container registries exactly like Docker images.

CI/CD → GitOps → KServe (Promotion Engine). When a fine-tuned model is tagged as "Champion" in MLflow or W&B, a CI pipeline generates the InferenceService manifest, injects the model URI and hardware requirements, and commits it to the GitOps repo. ArgoCD detects the drift and applies the manifest - first with canaryTrafficPercent: 10, then promoted to 100% after the quality gate passes.

Evaluation Frameworks → KServe (Quality Gate). Before production promotion, KServe hosts the model in a staging namespace. The CI pipeline triggers an evaluation framework like Promptfoo or DeepEval to send test prompts to the staging endpoint and score responses for hallucinations, context relevance, and toxicity. If evaluation fails, the GitOps promotion is blocked. For continuous evaluation, KServe's shadow traffic feature mirrors live requests to an unreleased model version without affecting users.

Observability → KServe (Feedback Loop). KServe and vLLM expose Prometheus metrics natively - GPU utilization, KV cache saturation, TTFT, inter-token latency. These feed both Grafana dashboards and the Knative/KEDA autoscaler. For payload-level observability, KServe's Logger component captures prompts and completions asynchronously, pushing them to Kafka or S3 for drift detection and quality monitoring via platforms like Arize or LangSmith.

Running KServe with vLLM in production is not without friction. Version compatibility across the Istio/Knative/KServe stack requires careful pinning and synchronized upgrades. Large model weights introduce cold-start latency that makes scale-to-zero impractical for most LLM workloads - plan for warm replicas. Timeout budgets must be aligned across the entire ingress chain (Istio, Knative queue-proxy, vLLM) to prevent silent request drops during long generations. And autoscaling based on request concurrency alone is insufficient - GPU memory saturation and KV cache pressure are the metrics that actually determine when to scale.

5) Choosing Your Deployment Path

vLLM has become the inference engine standard for a reason - PagedAttention and continuous batching solve the fundamental memory and throughput constraints of LLM serving. But the engine is only half the equation. Your deployment wrapper must match your team's operational maturity, not your architectural ambition. Start with standalone, graduate to KServe or BentoML when the operational requirements - not the resume appeal - justify the complexity. The best infrastructure is the simplest one that meets your actual production constraints.

Need Help Implementing This?

At Liquid Reply, we specialize in Platform Engineering, Kubernetes, and Cloud Native infrastructure - including production-grade LLM serving architectures. From selecting the right deployment strategy to production-hardening KServe with proper autoscaling, canary rollouts, and LLMOps integration, we help engineering teams get LLM inference into production without the trial-and-error.

Whether you are evaluating vLLM for the first time or need to operationalize a multi-model KServe platform with GitOps, observability, and self-service abstractions for your ML teams - reach out for a free initial consultation.