How I load-test self-hosted model serving before an air-gapped rollout

On a cloud deployment, a bad serving config costs you a redeploy. On an air-gapped site it can cost you a week. Every package, container image and model weight has to be staged ahead of time, and there is no pip install to fall back on when something is missing. So most of my tuning happens on test hardware, before anything reaches the site.

This post covers the method I used this year on a rollout for a government client: speech-to-text, text-to-speech, embeddings and chat models, all self-hosted on GPUs inside a closed network. I worked in a team of three and took on most of the technical delivery. The pre-deployment numbers below come from our own test hardware, a single NVIDIA H200, in June and July 2026.

flowchart TD
    A["Offline bring-up rehearsal"] --> B["Accuracy on public datasets"]
    B --> C["Solo benchmark per service"]
    C --> D["Mixed load, all services at once"]
    D --> E["Soak"]
    E --> F["On-site burst test per service"]
    F --> G["Re-tune"]
    G --> F

Why test before rollout

Two things make air-gapped rollouts unforgiving. Fixes are slow, because a missing library cannot be installed on the spot and getting one in can take days. And anything that quietly reaches for the internet fails there and nowhere else.

The second bit us in testing. The text-to-speech service shipped with an Arabic voice preset whose reference transcript was empty. With that field empty, the service tries to transcribe the reference audio on the fly, which does not work offline, so every Arabic and mixed-language request returned HTTP 500. A review caught it, and the fix was one line of config: supply the transcript.

My rule since then: rehearse the full bring-up from offline packages on the test box before trusting any number it produces.

The test bench

One H200 (141 GB) ran three services side by side: a speech-to-text model on vLLM, an Arabic-capable text-to-speech model behind a small FastAPI service, and a multilingual embedding model. Locust generated the load with a realistic mix of Arabic, English and mixed-language traffic: real audio clips to transcribe, real sentences to synthesise, real passages to embed. Prometheus and a GPU exporter collected telemetry, along with DCGM’s SM activity counter.

That last counter matters. The “GPU utilization” figure from nvidia-smi is the fraction of time at least one kernel is running, so it can read close to 100% while most of the chip sits idle. DCGM’s SMACT averages, across every SM, the fraction of time it had a warp active, so it shows how much of the chip is actually busy. An earlier report of mine read saturation off nvidia-smi and overstated it. I now report both and base conclusions on SMACT.

The harness rules came from getting burned. Reuse HTTP sessions, discard a 5-second warmup, measure over 60 seconds, repeat, and report the median. My first quick benches used two-second windows, single runs and a fresh connection per request. Their directions held up; their absolute numbers did not.

Accuracy first, then load

Throughput is meaningless if the model gets the words wrong, so each was checked on public data first. Speech-to-text scored 2.1% WER on a 100-clip sample of LibriSpeech test-clean. On a 100-clip FLEURS Arabic sample it scored 3.3% CER and 11.8% WER. CER is the usual headline because spelling variation inflates WER, but an 11.8% word error rate is what a user will notice.

The embedding model reached nDCG@10 of 0.928 in Arabic and 0.867 in English on a 120-query MIRACL reranking subset, with about a dozen candidates per query. That is easier than full-corpus retrieval, so it is not comparable with leaderboard scores.

Text-to-speech round-tripped cleanly on general prose and stumbled on English numbers, dates and abbreviations, code-switched sentences, and very short Arabic phrases. There were one or two clips per case, so this is indicative, but it still says to put text normalisation in front of the model.

Picking the embedding server

I served the same embedding weights four ways on an identical workload (192 requests, batch 16, concurrency 8, median of three repeats; Triton was re-run on its own after a dependency fix). Text Embeddings Inference did 200.2 requests per second, Infinity 60.5, vLLM 59.6 and Triton with ONNX 18.2, with nDCG within 0.001 across all four, so the choice came down to throughput and operations.

Mixed load and soak

Models that look fine alone can fall over when they share a GPU, so the main test runs everything at once. The ceiling run on our test box used 500 concurrent simulated clients (100 speech-to-text, 100 text-to-speech and 300 embedding streams) for 300 seconds. It completed 140,189 requests with zero failures.

Zero failures says nothing about latency, which was mixed. Embedding p95 was 440 ms. Speech-to-text p95 was 4.2 s with 100 speech clients in the mix, against about 0.39 s alone at 50 concurrent requests. Part of that is the doubled load, but most of a tenfold jump is the GPU being shared. Text-to-speech p95 was 12 s.

The telemetry showed what was binding. At 500 clients SM activity reached 0.71 and peak power hit 94.6% of the card’s 700 W limit, while VRAM sat at about 29%. Compute ran out first, power was close behind, and memory had room to spare. That changes what you tune: shaving gigabytes off a model buys nothing when you are short on SM time.

For stability, the first campaign ran a 30-minute soak with all three services driven, on the untuned stack and with a synthetic audio clip for speech-to-text: 2,057,540 requests, zero failures, and 71 MiB of VRAM drift, so no sign of a memory leak. After reworking text-to-speech I re-ran a 12-minute soak on the tuned stack, which finished 80,972 requests without a failure and drifted 407 MiB of VRAM, too short a window to rule a leak in or out.

Finding the real bottleneck

Text-to-speech was the outlier. Alone at 50 concurrent requests it managed 0.5 requests per second, with a median latency of 66.5 seconds, while GPU duty cycle averaged about 4%. The GPU was mostly waiting. The model is not safe to call from several threads, so the service handled one request at a time.

The textbook levers did nothing. Halving the sampling steps left throughput at 0.5 requests per second and lowered naturalness scores. Four process replicas under NVIDIA MPS gave 0.48 requests per second at 50 concurrent and produced 30 failures at 100. On one GPU, separate processes mostly take turns.

The fix was in-process micro-batching. The model’s sampler already accepts a batch, so I added a queue to the service that waits about 40 ms, collects concurrent requests for the same voice, runs one batched pass and slices the audio back out per request. In a 100-request burst at 50 concurrent, throughput went from 0.5 to 13.4 requests per second and median latency from 66.5 s to 2.7 s. A batch-size sweep picked 16; 32, 48 and 64 were all slower. These were short single runs; the gap was too large for repeats to change the conclusion. This was in the validation service on our test box; the delivered text-to-speech server is built differently.

Batching fixed throughput, not tail latency: at 100 concurrent requests p95 stayed at 7.3 s, and in the 500-client mix it was 12 s. With a ceiling of about 14 requests per second per GPU, 100 waiting clients will queue for seconds whatever you do. For interactive use I would keep pure text-to-speech concurrency at or below about 30 per GPU and shed the rest with HTTP 429, which the service does once its queue is full. Batching also only groups requests that share a voice.

Checking that FP8 KV cache did not cost accuracy

An FP8 KV cache stores each cached token in half the memory of bf16, which is free capacity if accuracy holds.

First, check that the flag took effect. A review of an earlier round found the speech server had silently run with the default KV dtype, so the “FP8” results were nothing of the kind. An attention-backend A/B had the same problem: the environment variable was rejected and both arms ran FlashAttention. Now I read the vLLM startup log for each arm before recording anything, looking for “Using fp8 data type to store kv cache” or the attention backend’s name.

Second, measure accuracy on both sides of the change, on the same day. I only got part of the way there. Two full 900-clip speech-to-text runs with FP8 KV active agreed within 0.21 WER points across nine dataset and noise conditions (FLEURS Arabic clean: 11.58% and 11.79%; LibriSpeech clean: 2.06% and 2.12%). Both matched the headline accuracy of the earlier default-KV runs, about 2.1% English WER on the LibriSpeech sample and about 3.3% Arabic CER. I did not re-run a same-day default-KV baseline, so this is parity against earlier runs rather than a same-session A/B.

Speech-to-text is the only model where I claim FP8 KV parity, because it is the only one where I checked it. Any KV-cache dtype change should get a quality re-run as well as a throughput run. The log-verified attention A/B settled the other question: FlashAttention 3 beat FlashInfer by about 10% on throughput for the speech model, so it stayed.

On site: burst load per service, re-tune between rounds

On site the test kit is smaller. For the chat models I used vLLM’s bench serve with the request rate unbounded and concurrency capped at a fixed level, so the server stays full. For speech and embeddings, small scripts fired a fixed number of calls at once and counted the answers.

Burst load produced no failed requests across the model services. Speech-to-text answered every concurrent transcription.

Between rounds I re-tuned the Arabic chat model. The scheduler pointed at the max-sequences limit, not memory. Enabling FP8 KV cache and raising that limit lifted output throughput by more than a quarter under the same load. The gain showed up at high concurrency; at lower load the change was small, and each stream decoded somewhat slower because more requests shared the GPU. That is a throughput number only.

The toy model below shows why: raise the limit and more requests run at once, each a little slower, while more finish every second.

Try it

Why the batch limit matters under load

Concurrent users

Max sequences at once

On the GPU

Waiting

In service: 4
Waiting: 6
Last request waits: 2.4 s
Finished per second: 3.4

Toy model with example values, not measurements. Add users, then raise the limit and watch the queue drain into the GPU.

First-token latency grows quickly as prompts get longer, so benchmark with prompts as long as the documents your users will paste in.

What I would do differently

Build the careful harness on day one. Warmup, 60-second windows, medians and log-verified configs would have saved a full re-measurement round.
Generate the report straight from raw result files, so a number without an artifact cannot get in. Mine does that now.
Soak the final stack for hours, not an interim stack for minutes.
Run every service alone and in the mix on the same day, so interference shows up as a measured number and not as a surprise. The same goes for both arms of any accuracy comparison.
Use real document lengths and a realistic traffic shape from the first benchmark. Random tokens at a fixed length were the least useful test I ran.