Databricks Unveils Strategies for Reliable LLM Inference at Scale

Databricks has outlined its strategy for achieving reliable large language model (LLM) inference at scale, a critical challenge driven by the exponential growth in demand from agentic applications. The company highlighted the difficulties in maintaining consistent availability and low latency in multi-tenant systems that serve a diverse range of models, from open-source to proprietary, and power significant agentic applications.

The Databricks inference platform currently handles over 120 trillion tokens per month, serving models like Kimi, Qwen, OpenAI, Gemini, and Claude. This scale, coupled with the increasing use of AI agents as primary interfaces for work and life, results in highly variable and spiky demand curves, particularly during peak working hours.

Challenges in LLM Serving

Serving LLMs reliably at scale presents several interconnected challenges. Firstly, reliability itself is complex. While availability typically means a request can be processed, different use cases have distinct latency requirements that impact this. Advanced agents, for instance, cannot tolerate degradation in time to first token (TTFT) or output tokens per second (OPTS).

The infrastructure required for frontier model performance, such as high-end GPUs with fast interconnects for KV cache transfer, is inherently less reliable and more expensive than traditional CPU systems. Failures in disaggregated prefill/decode setups can necessitate complex reconfigurations across multiple nodes. Furthermore, high-bandwidth networking often relies on single-spine connectivity within a single physical rack, meaning rack-level failures can lead to widespread outages. Standard distributed system solutions like multi-AZ deployments or backup instance types are often cost-prohibitive for LLM serving due to the expense of keeping backup GPUs idle. Overprovisioning, another common strategy, is also impractical given the constrained compute supply.

Shipping new features and supporting evolving model architectures also add complexity. Innovations like image and video processing, or safety classification, each require separate, scalable preprocessing systems. The introduction of new low-level software for different architectures can also lead to opaque failures at scale, making debugging difficult.

Managing Latency

Keeping latency under control is equally challenging due to variable request costs. Even healthy servers process requests more slowly under heavier loads, creating a trade-off between throughput (cost efficiency) and the ultra-low latency demanded by products. This can also lead to servers entering unhealthy states rapidly based on the mix of incoming requests. While latency is largely determined by output token generation, predicting the duration of a model’s response is difficult, complicating capacity management, load balancing, and request prioritization.

Databricks’ Approach: Model Units and Dicer

To address these issues, Databricks has introduced an abstraction called “model units.” This allows them to reason about capacity by projecting the number of model units a replica can process per minute. This enables modeling request costs using a multi-dimensional function, where coefficients are determined by automated benchmarking for each model on specific hardware. Model units can be adjusted for optimizations like prefix caching and must account for features like multi-modality.

These estimations, while imperfect, create a more manageable system akin to cloud VMs, offering predictable performance that can be allocated to customers. This is crucial for production agentic workloads that require guarantees around low latency and capacity, moving beyond a “best-effort” model.

Optimal routing decisions are vital because LLM requests have a highly variable impact on servers. Traditional load balancing methods like P2C, which rely on queue size and sampling, may not be sufficient due to high LLM latencies and the severe cost of misrouting. Databricks employs Dicer, its auto-sharder, for dynamic workload routing. Dicer integrates model units, allowing routing decisions to be based on server load in model units rather than simpler request-based heuristics. Dicer also supports stateful sessions, directing a workload’s requests to a specific subset of servers. This improves cache hit rates, which is critical for latency-sensitive tasks like coding agents, and limits the blast radius of failures.

Autoscaling also benefits from this approach. Relying solely on pending request counts is insufficient, as spikes in long-context requests can appear identical to short ones, and CPU/memory metrics often don’t correlate with actual GPU utilization.

Datos clave

Databricks’ focus on these sophisticated strategies is essential for enterprises increasingly relying on AI agents and LLM-powered applications. By addressing the core challenges of reliability and latency through novel abstractions and intelligent routing, the company aims to provide a foundational inference platform capable of supporting the next wave of AI innovation.

Fuente: Databricks Blog – Reliable LLM Inference at Scale – https://www.databricks.com/blog/reliable-llm-inference-scale