SpecLane.
Technical report · November 2026

SpecLane: Serving Small Hybrid-Attention Models Under Agent Traffic

How should an inference engine be configured to serve small language models to AI agents? I built a serving configuration for two open-weight models, Qwen3.8-27B-FP8 and Qwen3.5-9B, on the vLLM engine. I start from the default setup, change one setting at a time, and measure speed, responsiveness and answer quality to verify each change.

vLLM 0.33.0 · one NVIDIA H100 and one L40S GPU on cloud instances · one run per measurement · code and raw results are in the repository

27B speed, one request at a time
+124%
78 to 175 tokens per second with multi-token-prediction speculation (MTP-3).
27B speed, 16 requests at once
+50%
800 to 1,199 tokens per second on the agent-session workload.
9B speed, one request at a time
+77%
43 to 75 tokens per second with multi-token-prediction speculation (MTP-2).
Change in answer accuracy
±2 of 200
At most two questions out of 200: within normal sampling noise.

Speed and latency figures are generated from the raw result files by paper/make_results.py. The accuracy check was run on live endpoints with scripts/gsm8k_endpoint.py.

In short

Key terms

Throughput

How many output tokens (word pieces) the server generates per second, summed over all requests. Higher is better.

Time to first token (TTFT)

How long a request waits before the first token appears. It governs how responsive the system feels. Lower is better.

p99

The 99th percentile: the value that all but the slowest 1% of requests stay under. With 24 requests per measurement, it is simply the slowest request.

Concurrency (c)

How many requests are in flight at the same time. c = 1 measures a single user; c = 16 measures a loaded server.

Multi-token prediction (MTP)

Alongside its normal output layer, each model carries a small extra head trained to predict several upcoming tokens at once. In speculative decoding that head proposes the next few tokens cheaply, and the full model verifies all of them in a single pass, keeping every correct proposal. The number after MTP in the detailed tables is how many tokens are proposed per step.

KV cache

Memory that stores the attention keys and values of earlier tokens so they are not recomputed. “FP8 KV cache” stores them in 8-bit precision to save memory.

Results

MTP is speculative decoding with the model’s multi-token-prediction head. Output tokens per second on the agent_sessions workload (multi-turn conversations that resend their history), one run per measurement. The change from the default (“baseline”) setup is in parentheses. c is concurrency. The last column is the p99 time to first token, in seconds, at c = 16.

modelprofilec = 1c = 16p99 TTFT, c = 16 (s)
Qwen3.8-27B-FP8baseline788001.09
Qwen3.8-27B-FP8MTP, 3 proposed tokens175 (+124%)1,199 (+50%)0.94
Qwen3.8-27B-FP8MTP (3) + FP8 KV cache163 (+110%)1,007 (+26%)1.20
Qwen3.8-27B-FP8FP8 KV cache76 (−2%)684 (−14%)1.46
Qwen3.5-9Bbaseline434750.95
Qwen3.5-9BMTP, 2 proposed tokens75 (+77%)753 (+59%)0.74
Qwen3.5-9BFP8 KV cache43 (+0%)467 (−2%)1.42
Why 3 on one model and 2 on the other?The number is how many tokens the MTP head proposes per step. Proposing more can save more work when the proposals are right, but each wrong proposal is wasted effort, so every model has a best value. I used 3 on the 27B, the value recommended for that model, and tried 1 and 2 on the 9B: 2 was faster (+77% against +44% at c = 1), and it is also the value in that model’s documentation. I did not search other values on the 27B, so 3 there is the recommended setting rather than a tuned optimum.
Throughput by profile, agent sessions
Output tokens per second. Hover a bar for the exact value.
baselinemulti-token prediction, MTP (recommended)FP8 KV cache
One run per bar, so there are no error bars. The baseline setup gave the same result within about 2% when started separately, so I do not claim differences smaller than about 5%.
Read speed and responsiveness together.A faster server can still have a slower worst case. On the 27B model with 16 requests at once (shared_prefix workload), MTP-3 raised throughput by 75% but also raised the p99 time to first token from 0.53 s to 0.79 s. The FP8 KV cache raised it to 1.47 s while also lowering throughput, so it was worse on both measures.

Detailed evaluation

Throughput

Output tokens per second, with the change from baseline in parentheses. shared_prefix sends many requests that start with the same long prompt; agent_sessions sends multi-turn conversations.

Qwen3.8-27B-FP8 (H100)shared_prefix c=1shared_prefix c=16agent_sessions c=1agent_sessions c=16
baseline7974378800
kv-fp876 (−3%)612 (−18%)76 (−2%)684 (−14%)
mtp-3176 (+123%)1,297 (+75%)175 (+124%)1,199 (+50%)
mtp-3 + kv-fp8167 (+111%)1,018 (+37%)163 (+110%)1,007 (+26%)
Qwen3.5-9B (L40S)shared_prefix c=1shared_prefix c=16agent_sessions c=1agent_sessions c=16
baseline4342043475
kv-fp843 (−0%)399 (−5%)43 (+0%)467 (−2%)
mtp-161 (+41%)499 (+19%)61 (+44%)617 (+30%)
mtp-271 (+65%)644 (+53%)75 (+77%)753 (+59%)

Tail latency

p99 time to first token, in seconds. With 24 requests per measurement, the p99 equals the slowest request.

Qwen3.8-27B-FP8shared_prefix c=1shared_prefix c=16agent_sessions c=1agent_sessions c=16
baseline0.350.530.271.09
kv-fp80.361.470.271.46
mtp-30.360.790.280.94
mtp-3 + kv-fp80.371.510.291.20
Qwen3.5-9Bshared_prefix c=1shared_prefix c=16agent_sessions c=1agent_sessions c=16
baseline0.530.690.410.95
kv-fp80.581.390.431.42
mtp-10.541.540.460.95
mtp-20.570.840.450.74

Answer quality

In principle, speculative decoding does not change what the model outputs, because every proposed token is verified by the full model. An implementation can still differ from the theory, so I checked. I deployed the default and the MTP setup of each model as live servers and asked both the same 200 grade-school math questions (a fixed random sample of GSM8K), with the sampling temperature set to 0 and the model's extended “thinking” mode turned off. An answer counts as correct only if its final number exactly matches the reference.

modelbaselineMTPdifference
Qwen3.8-27B-FP8 (MTP-3)0.9400.935−1 question
Qwen3.5-9B (MTP-2)0.9000.910+2 questions

The differences are one and two questions out of 200, which is within normal sampling noise. I did not run this check for the FP8 KV cache.

Method

Models and hardware

Qwen3.8-27B-FP8 (a 27-billion-parameter model, 31 GB, Apache-2.0 licence) on one H100 GPU with 80 GB of memory, and Qwen3.5-9B (9 billion parameters, 19 GB, Apache-2.0) on one L40S GPU with 48 GB. Each model runs on its own GPU in a separate container. Software: vLLM 0.33.0, PyTorch 2.14, CUDA 13.0.

The baseline

The engine's default settings, plus only what is needed to start the model and a fixed context length of 16,384 tokens. The 9B model needs --trust-remote-code. The 27B model needs --max-num-seqs 512 (explained below).

Settings tested

Each test changes exactly one setting from the baseline: speculative decoding with k proposed tokens, the FP8 KV cache, scheduler limits, asynchronous scheduling, or turning prefix caching off. I then combine the best ones.

Workloads

shared_prefix: a 3,000-token system prompt shared by all requests, a 150-token question, and up to 256 output tokens. agent_sessions: six-turn conversations with a 2,000-token shared system prompt, where each turn resends the full history. Prompts are random words, so no request is answered from cache by accident.

The load generator runs inside the same container as the server, so no network delay is included. Every measurement is preceded by warm-up requests drawn with a different random seed. I test 1 and 16 concurrent requests. shared_prefix sends 24 requests per measurement. agent_sessions sends 24 requests at c = 1 and 96 at c = 16. Throughput is the number of output tokens divided by the elapsed time.

A startup failure on a hybrid-attention model

Qwen3.8-27B uses a cheaper “linear” form of attention in 48 of its 64 layers. Those layers keep a fixed-size state for every active sequence, so the number of sequences an engine can hold is limited by that state as well as by the KV cache. With vLLM 0.33.0’s default settings on one H100, the 27B model failed to start. The engine stopped while preparing its optimised execution graphs, with this error:

ValueError: max_num_seqs (1024) exceeds available Mamba cache blocks (745).
Each decode sequence requires one Mamba cache block, so CUDA graph capture
cannot proceed. Please lower max_num_seqs to at most 745 or increase
gpu_memory_utilization.

The default limit of 1,024 concurrent sequences is higher than the 745 state slots that fit in memory next to the model. Setting --max-num-seqs 512 fixes it, so I include it in the baseline. In this report, “baseline” therefore means the defaults plus whatever is needed to start.

More findings

The FP8 KV cache: not recommended at this scale

Storing the KV cache in 8-bit precision lowered throughput (by 14% on the 27B model with 16 requests at once) and raised the p99 time to first token from 0.53 s to 1.47 s on shared_prefix. Its purpose is to free memory, but at these model sizes memory is not the limiting factor, and the extra conversion work shows up as lost speed. It is not part of the recommended setup.

Agent harness pilot

I also asked whether a carefully engineered prompt for the coding agent (the “harness”), combined with the tuned engine, makes agents more accurate. I compared it with the default agent on the default engine, using Terminal-Bench 2.1 (command-line tasks, Qwen3.8-27B) and τ²-bench (customer-service conversations, Qwen3.5-9B). At the sample sizes I could afford, there was no reliable difference.

task set (same tasks in both arms; valid outcomes only)tasksdefaultengineerednote
τ²-bench airline, test190.840.84tie; uncertainty about ±0.21
τ²-bench retail, test51.000.80one task worse
Terminal-Bench, first 10 test tasks100.300.40one task better, none worse
Terminal-Bench, final round82 solved2 solvedtie; the rest hit sandbox errors
Terminal-Bench, dev tasks50.200.80not used for tuning; not confirmed on the test tasks
Scope of the agent results.These are small samples with one attempt per task, so they are best read as a pilot. The development tasks favoured the engineered harness; the held-out test tasks showed no measurable difference. Two prompt changes aimed at a failure in which the model reasons until it reaches its length limit did not change the outcome. A larger, repeated study is the natural next step for any agent-accuracy claim.

Evaluation pipeline fixes

Reproduce

Everything is driven by a small, engine-agnostic configuration layer: a model registry, a set of named serving profiles (the baseline and one-change variants), a workload generator, a load-testing client and a quality check. Any machine with a supported GPU and vLLM can run it. The repository README lists the exact commands for each step:

  1. Stage the model weights and verify that every candidate setting is accepted by the installed vLLM version.
  2. Sweep the profiles and workloads at the chosen concurrency levels, with the accuracy check enabled.
  3. Serve the baseline or the tuned (MTP) profile behind an authenticated, OpenAI-compatible endpoint.
  4. Rebuild the tables, figure and paper directly from the raw result files with paper/make_results.py.

A deployed endpoint keeps its GPU allocated until it is stopped, so shut it down when a run is finished.

Acknowledgements

This work builds on vLLM and the Qwen model family. The agent-harness pilot used Terminal-Bench 2.1 (run with Harbor) and τ²-bench.

Appendix: how multi-token prediction works

What I did and did not do.I did not design or train the MTP head. It ships inside each Qwen model, trained by the model’s authors. My work is to switch it on, choose how many tokens it proposes, and measure the result. This section gives the mathematics of how it is used and why it makes generation faster, and checks the theory against my measurements.

1. The head proposes, the model verifies

Plain decoding runs the full model once per generated token. Speculative decoding adds a cheap proposer that guesses the next k tokens; the full model then checks all k guesses in a single forward pass. With MTP, the proposer is a small head attached to the model. In the general MTP design (published for DeepSeek-V3; I have not inspected the exact Qwen layer layout), the head for the j-th future token combines the model’s hidden state with the embedding of the token just proposed, passes them through one light transformer block, and reuses the model’s own output layer:

h′j = Blockj( Wj [ norm(hj−1) ; norm(E(x̂j−1)) ] )
qj = softmax( LMhead( norm(h′j) ) ),   x̂j = argmax qjproposal j = 1…k; E and LMhead are shared with the main model, so the head is small

The model authors train the head with an extra loss for each future position, added to the usual next-token loss, where λ weights the extra terms:

L = Lmain + λ · Σj=1..k Lj,   Lj = − log qj( xt+j+1 | x≤t )training objective (done by the model authors, not by me)

2. Verification keeps the output identical

Let p be the full model’s distribution at a position and q the proposer’s. A proposed token x is accepted with probability min(1, p(x)/q(x)). At the first rejection, the token is replaced by a sample from the leftover distribution, and the remaining proposals are discarded:

P(accept x) = min(1, p(x)/q(x))    on reject: x ∼ norm( max(0, p − q) )at temperature 0 this reduces to: accept if x̂ = argmax p, otherwise take argmax p

The emitted token then follows exactly p, the full model’s own distribution:

P(x) = q(x)·min(1, p(x)/q(x)) + [ 1 − Σy min(p(y), q(y)) ] · max(0, p(x) − q(x)) / Σy max(0, p(y) − q(y)) = p(x)the first term is min(p,q); the second is the leftover mass, which supplies exactly p − min(p,q)

So speculative decoding changes the speed, not the distribution of outputs. If all k proposals are accepted, the same verification pass also yields one extra token for free, so a step produces between 1 and k+1 tokens.

3. Expected tokens per step and the speedup

Let α be the probability that a proposal is accepted, assumed equal at every position. A step produces j tokens when the first j−1 proposals are accepted and the next is not, so the expected number of tokens per step is a geometric sum. Let c be the cost of one proposal step relative to one full-model step. Because verifying k+1 positions costs about one full-model step when decoding is limited by memory bandwidth, the speedup is:

E[tokens per step] = Σj=0..k αj = (1 − αk+1) / (1 − α)
Speedup S(k) = E[tokens per step] / (1 + c·k)upper limit as k grows: 1 / (1 − α)

Two forces set the best k. Each extra proposal adds a term αk to the numerator, which shrinks geometrically, while it adds c to the denominator. The speedup therefore rises, peaks and then falls. For illustration, with α = 0.6 and c = 0.1, S(1) = 1.46, S(2) = 1.63, S(3) = 1.67, S(4) = 1.65 and S(6) = 1.52, so the best value is 3. This is a worked example of the shape, not a measurement. It is why each model has a best k and why I compared 1 and 2 on the 9B.

4. Checking the theory against my measurements

My measured speedups bound the acceptance rate. Since S = E / (1 + c·k) ≤ E, setting E equal to the measured speedup and solving for α gives a lower bound on the acceptance rate. This assumes equal acceptance at every position, which is an approximation.

model, krequests at oncemeasured speedupmax possible (k+1)implied acceptance rate α ≥
Qwen3.8-27B, k = 312.24×40.62
Qwen3.8-27B, k = 3161.50×40.34
Qwen3.5-9B, k = 111.44×20.44
Qwen3.5-9B, k = 211.77×30.51
Qwen3.5-9B, k = 2161.59×30.42

Measured speedups come from the agent_sessions workload. Two observations are consistent with the model. First, the speedup is below the cap of k+1 in every row, as it must be. Second, the speedup is smaller with 16 requests at once than with one. Batching makes the full-model step more compute-bound, so verifying extra tokens costs more and the effective c rises. I did not measure α or c directly, so this explanation is a consistent reading of the numbers and not a tested claim.