EAServe: Up to 4.3x Higher Goodput for Multimodal LLM Serving

Multimodal models do not begin with language generation. Images, video, and audio first pass through an encoder, then Prefill builds the key-value cache, and only then can Decode produce tokens.

Those three phases have different resource profiles. Treating them as independent services looks clean architecturally, but it can leave the encoder GPU mostly idle while every downstream request waits at its gate.

EAServe turns that gate into a control point for the whole pipeline.

Multimodal serving adds a third bottleneck

Text-only LLM inference is usually divided into Prefill and Decode. Prefill processes the prompt and builds the KV cache; it is compute-heavy. Decode produces tokens autoregressively and is usually constrained by memory bandwidth.

A multimodal model adds Encode in front. A vision or audio tower converts raw input into embeddings that Prefill consumes with the text prompt. The serving path becomes Encode–Prefill–Decode, or EPD.

In a pass-through encoder service, one request triggers one small forward pass. That pass often cannot saturate a modern GPU. The encoder therefore spends much of its time idle, yet every request still has to pass through it. A queue can build at the least-utilized GPU while Prefill and Decode wait for work.

Micro-batching helps the encoder but can move the bottleneck downstream. On LLaVA-v1.6-34B, a fixed encode batch of eight raised throughput from 0.75 to 1.09 requests/s and reduced mean time-to-first-token from 367.4 to 112.1 seconds. Mean time-per-output-token, however, increased from 179 to 411 ms because batched Encode flooded Prefill and Decode.

Admission, routing, allocation, and resource sharing are coupled. EAServe controls them together.

One offline search, three online controls

Hybrid Auto Selection (HAS) chooses a deployment for a specific model, workload, cluster, and target arrival rate. It selects three values:

• the GPU allocation across Encode, Prefill, and Decode;
• the maximum encode batch size, $B$; and
• the remote-Prefill fraction, $s$.

The runtime then makes those choices stable request by request.

1Profile stages

Measure independent Encode, Prefill, and Decode capacity.

→

2Screen allocations

Discard configurations with an unavoidable bottleneck.

→

3TPE search

Tune allocation, batch bound $B$, and offload ratio $s$.

Encode control plane

Adaptive micro-batcher dispatch by load, size, or age

↓

Modality encoder vision or audio tower

↓

Deficit-counter router maintain remote fraction s

1 − s↙s↘

Local path

EncodeLocal Prefill

Dynamic SM partitioning protects Encode and fills unused compute.

Offload path

Remote Prefill

↓ KV cache

Remote Decode

Outputs

Search only the configurations that matter

Even an eight-GPU server admits hundreds of allocation, batch-size, and routing combinations. HAS first profiles the capacity of each stage independently and removes allocations that already overload one stage. In the evaluated search space, this reduces seven allocation candidates to three.

TPE-based Bayesian optimization then measures the survivors end to end. Within 30 trials, HAS reaches the 99th percentile of the best observed throughput in 1.7 hours for image, 1.3 hours for video, and 1.7 hours for audio. Profiling rejects structurally impossible deployments; measurement captures the non-monotonic interactions that remain.

Batch according to the arrival process

A fixed encode timeout is too eager at low load and too slow at high load. EAServe derives the dispatch gap from the per-worker arrival rate $\lambda$, making the threshold proportional to $1/\lambda$.

The batcher dispatches when it reaches the configured maximum, when the inter-arrival gap crosses the adaptive threshold, or when the oldest request reaches a one-second safety cap. This keeps the encoder efficient without allowing a partial batch to wait indefinitely.

Route Prefill at a precise rate

After Encode, a request can use Prefill co-located on the encode GPU or a remote Prefill worker. A deficit counter keeps cumulative remote routing within one request of the target fraction $s$ at every point in the stream. Unlike random routing, it does not send short bursts down one path and destabilize both queues.

EAServe co-locates Prefill, not Decode. Encode and Prefill are compute-heavy forward passes that can divide Streaming Multiprocessors. Decode competes for shared memory bandwidth, which software SM partitioning cannot isolate.

Repartition the GPU for every micro-batch

At startup, EAServe profiles encode latency across batch sizes and five SM allocation levels. For each batch size it stores the smallest share that keeps Encode close to its exclusive-GPU latency.

Small batches release more SMs to local Prefill; large batches reclaim them for Encode. Reconfiguration takes less than a microsecond. The policy also knows when not to share: the evaluated video encoder uses more than 90% of SMs during batched forwards, so HAS routes every request to remote Prefill.

Goodput, not throughput alone

We evaluate three representative multimodal architectures on the same server with eight NVIDIA RTX 6000 Ada GPUs.

The primary metric is goodput: requests whose time-to-first-token and time-per-output-token both satisfy the selected P99 SLO tier. Raw throughput can remain high while user-visible tail latency becomes unacceptable; goodput counts only requests that meet both promises.

Image LLaVA-34B

Dynamo

0.95

vLLM

1.11

EAServe

1.93

Video Qwen2.5-VL-32B

Dynamo

0.95

vLLM

1.32

EAServe

1.90

Audio Ultravox-27B

Dynamo

0.98

vLLM

2.77

EAServe

4.17

requests/s · moderate SLO tier · bar scales are normalized within each modality

The modality dependence is the point. Audio leaves enough encoder headroom for substantial local Prefill. Video consumes nearly the whole GPU and benefits from batching and allocation without co-location. Image lies between them. There is no single best EPD recipe independent of the model.

Tail latency falls with the queues

At an arrival rate of two requests per second, EAServe sharply reduces P99 latency as well as increasing goodput.

For image and video, Dynamo’s request-at-a-time Encode is already saturated at this rate, so the queue grows throughout the run. Adaptive batching keeps Encode at line rate. For audio, selective local Prefill relieves the downstream queue.

Reclaim idle compute without adding hardware

The utilization trace reveals why the system-level gain is possible. Request-at-a-time Encode uses less than 10% of the GPU. Fixed batching produces taller bursts but still leaves long gaps. EAServe fills those gaps with local Prefill while SM partitioning protects the encoder’s critical path.

Baseline

<10%

Fixed batch = 8

bursty

EAServe

~80%

0%25%50%75%100%

At 10 requests per second on Ultravox, dynamic partitioning also beats uncontrolled CUDA time-slicing and a static 50/50 split.

Compared with time-slicing, the dynamic policy raises throughput by about 9%, lowers P99 first-token latency by 16%, and lowers P99 per-token latency by 29%. A static split helps, but it cannot respond when the encode batch size changes.

Why this matters for models a team can own

Serving is not model learning, but it determines whether repeated evaluation, synthetic-data generation, and production feedback are affordable. Better utilization means the same hardware can process more requests that actually meet a user-facing latency promise.

EAServe also makes deployment behavior legible. Allocation, batch size, and offload ratio are concrete controls selected from measured stage capacity. A team can see where resources go and why a configuration changes across image, video, and audio workloads.

Boundaries and open directions

The current evaluation establishes the design on one eight-GPU server. Several boundaries remain.

• Multi-node serving: cross-node embedding and KV-cache transfers are not evaluated here.
• Changing traffic: HAS searches offline for a target workload and arrival rate; a complete controller for rapidly shifting demand remains future work.
• Bursts: at the same mean rate, the burstiest tested distribution reduces throughput by only 3%, but P99 TTFT rises from 10.2 to 16.5 seconds.
• Bandwidth isolation: SM partitioning isolates compute, not the shared HBM bus; this is why EAServe does not co-locate Decode.
• Serving scope: EAServe runs existing multimodal models more efficiently. It does not train them or evaluate their task quality.

The bottom line

The multimodal encoder is not just preprocessing. Because every request passes through it, Encode is the natural place to coordinate when work enters the system, where Prefill executes, and how GPU compute is shared.

EAServe makes that control explicit. Offline search finds a workload-specific deployment; adaptive batching, precise routing, and per-batch SM partitioning keep it stable online. The result is higher SLO-compliant goodput, lower tail latency, and much better use of the same GPU budget.

Citation

Please cite our original paper as:

@inproceedings{zhu2026easerve,
  title     = {EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models},
  author    = {Zhu, Kunxiong and Shu, Zhihao and Zheng, Hangyu and Qin, Minghai and Yin, Miao and Agrawal, Gagan and Niu, Wei},
  booktitle = {Proceedings of the International Conference on Parallel Architectures and Compilation Techniques (PACT)},
  year      = {2026}
}