Muhammad Ibrahim
← Writing

vLLM v0.11.0: Technical Highlights & Optimizations

·2 min read

vLLM engine architecture and throughput-optimized design

1. Engine Architecture: Full Migration to V1#

  • V0 engine has been fully removed.
    This clears away LLMEngine, AsyncLLMEngine, and all old attention backends.
    V1 is now the sole, unified engine.
  • This streamlines code maintenance and ensures all features and optimizations benefit from the V1 pathway.

2. CUDA Graph Mode Default: Improved Out-of-the-Box Performance#

  • FULL_AND_PIECEWISE is now the default CUDA graph mode.
  • This is especially beneficial for fine-grained MoE (Mixture of Experts) models and generally improves performance without manual tuning.
  • Preserves compatibility with models only supporting PIECEWISE while unlocking better throughput for most modern LLMs.

3. Model & Task Expansion#

  • Many new models are supported: DeepSeek-V3.2-Exp, Qwen3-VL, Qwen3-Next, OLMo3, Dots OCR, Ling2.0, LongCat-Flash, and more.
  • Significant encoder support (e.g., RADIO encoder, transformer backend for encoder-only).
  • New task types: BERT token classification and multimodal pooling.

4. Core Engine Optimization#

  • KV Cache Offloading: LRU-managed CPU offloading for KV cache cuts down memory overhead for long contextual requests.
  • Prompt Embeddings, Sharded State Loading, and FlexAttention: Enable more complex attention & embedding patterns with less latency and memory waste.
  • Hybrid Allocator + Pipeline Parallel: Advanced memory management for varying hidden sizes and pipeline-parallel execution.
  • Async Scheduling: Uniprocessor executor, better load balancing, and shared memory multimodal caching.

5. Hardware-Specific & Kernel Optimizations#

  • NVIDIA Blackwell/Hopper/SM100: FP8 and BF16 support, specialized MLAs, MoE overlapping, and deep kernel fusion for optimal inference and throughput.
  • 13.7x token conversion speedup and improved batch overlapping (Dual-Batch Overlap, DeepEP).
  • Out-of-the-box tuning for modern architectures—better scaling for H200, B200, and MI300X (AMD).

6. Quantization#

  • Per-token-group FP8 quantization, blocked FP8 MoE, and new support for NVFP4/Gemma3/Llama 3.1 405B.
  • W4A8 preprocessing speedups, hardware-accelerated quantized tensor operations.

7. Distributed & Scalability#

  • Data Parallelism: torchrun, Ray placement, and improved DP/EP kernels for large clusters.
  • Async and speculative decoding support for more models (EAGLE3, GPT-OSS).
  • All-reduce optimization for NCCL, reducing distributed overhead.

8. Multimodal and API#

  • Multimodal: UUID-based caches, efficient video embedding, and improved hash tracking.
  • Prompt logprobs for all tokens in OpenAI-compatible API, enhanced event streaming and response tool APIs.

9. Security & Compliance#

  • Vulnerability and API validation improvements, NVTX for profiling, PyTorch 2.8+ CPU compatibility, C++17 for the build.

Practical Impact#

  • Higher throughput, lower latency, especially for advanced model families and multimodal tasks.
  • Simpler configuration with robust V1 defaults—performance unlocks with minimal tuning.
  • Stronger support for quantization, large-scale data-parallel serving, and heterogeneous hardware clusters.
  • Simplified upgrade path and codebase, leading to rapid integration of further innovations.

Comments