OpenInfer ← Benchmark

Benchmark telemetry

Sellable throughput under sustained load

Identical open-loop load driven at both fleets in parallel — a dedicated one-model-per-node baseline vs. OpenInfer co-locating every model on each node — measuring the tokens delivered within the latency SLA.

3.9×

more sellable tokens than baseline — 943k vs 240k delivered within the latency SLA

Latest run · 2026-09-10 15:39 UTC

Run summary avg 2.7× min 0.4× max 5.8× 34 runs 2026-07-22 → 2026-09-10

Results

Sellable tok/s

Baseline
2k
333.1/node
OpenInfer
7.9k
1.3k/node
Delta
3.9×

Sellable tokens

Baseline
240k
40k/node
OpenInfer
943k
157k/node
Delta
3.9×

Peak fleet GPU (mean of every GPU)

Baseline
46.7%
OpenInfer
97.8%
Delta
+51.1 pts

Accounting

Where every request went

Each request the arm sent is in exactly one column. Within SLA is the headline's own test over every request sent; the headline counts those finished by its second. Pace-exempt counts the requests within SLA only because a Medium or Low replica served them: the tokens-a-second pace binds only High replicas. Tier unknown counts the served requests that named no replica, held to the pace. Check is ✓ when sent plus unsent is what the schedule planned.

Baseline

Model Planned Sent Within SLA Pace-exempt Tier unknown Late Runtime 429 Weave 429 Hub 503 Timed out Client / killed Other Unsent Check
Qwen3.5-27B None None None None None None None None None None None None None None
Qwen3.5-9B None None None None None None None None None None None None None None
Llama-3.2-1B-Instruct None None None None None None None None None None None None None None
Phi-4-14B None None None None None None None None None None None None None None
Gemma4-12B-IT None None None None None None None None None None None None None None
Qwen3-1.7B None None None None None None None None None None None None None None
Total None None None None None None None None None None None None None None

OpenInfer

Model Planned Sent Within SLA Pace-exempt Tier unknown Late Runtime 429 Weave 429 Hub 503 Timed out Client / killed Other Unsent Check
Qwen3.5-27B None None None None None None None None None None None None None None
Qwen3.5-9B None None None None None None None None None None None None None None
Llama-3.2-1B-Instruct None None None None None None None None None None None None None None
Phi-4-14B None None None None None None None None None None None None None None
Gemma4-12B-IT None None None None None None None None None None None None None None
Qwen3-1.7B None None None None None None None None None None None None None None
Total None None None None None None None None None None None None None None

Requests completed within SLA

cumulative
Baseline OpenInfer in flight (right axis)

No request completed within SLA, or this report predates the count

Active concurrency over time

in-flight
Baseline OpenInfer
0 200 400 600 800 0s 20s 40s 60s 80s 100s 120s

Throughput over time

tok/s
Baseline OpenInfer active concurrency
0 10k 20k 30k 40k 0s 20s 40s 60s 80s 100s 120s 0 200 400 600 800

Fleet GPU utilization over time

% util, mean of every GPU
Baseline OpenInfer active concurrency
0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s 0 200 400 600 800

Error rate over time

% of sent requests, max_inflight drops excluded
Baseline OpenInfer

This report predates the error-rate series

GPU utilization per node over time

% util, mean of the node's GPUs
Baseline OpenInfer

ip-172-31-10-77

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-15-161

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-16-37

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-20-102

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-23-117

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-24-138

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-25-23

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-25-232

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-27-249

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-4-194

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-4-58

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-8-104

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

Per model · TTFT p50 / decode tok/s per request

Model Baseline TTFT p50 Baseline decode tok/s per request OpenInfer TTFT p50 OpenInfer decode tok/s per request
Qwen3.5-27B 4.5s 9.4 1.7s 24.3
Qwen3.5-9B 868ms 42.8 463ms 73.1
Llama-3.2-1B-Instruct 3.7s 120.7 1.7s 157.5
Phi-4-14B 3.4s 35.4 2s 50.2
Gemma4-12B-IT 1s 44.8 579ms 55
Qwen3-1.7B 1.2s 121.6 575ms 169.7

Results

Totals and per node

Metric Baseline total Per node OpenInfer total Per node
Total tokens 956k 159k 1.3M 211k
Aggregate tok/s 8k 1.3k 10.6k 1.8k
Requests (steps) 6,038 1,006 6,435 1,073
Over capacity 1,256 20.8% 97 1.5%

Latency & throughput

Per model · p95 / tok-s / over-capacity

Model Baseline p95 / tps / over-cap OpenInfer p95 / tps / over-cap
Qwen3.5-27B 6.3s / 9.4 / 0% 5.4s / 24.3 / 1%
Qwen3.5-9B 1.1s / 42.8 / 0% 2s / 73.1 / 0%
Llama-3.2-1B-Instruct 5.2s / 120.7 / 0% 3.7s / 157.5 / 0%
Phi-4-14B 5.2s / 35.4 / 40.1% 4.3s / 50.2 / 0%
Gemma4-12B-IT 3.9s / 44.8 / 70.3% 2.6s / 55 / 5.3%
Qwen3-1.7B 2.2s / 121.6 / 1.6% 1.4s / 169.7 / 3.9%

Load balance

Requests & utilization per model

Requests admitted, Served (2xx), and GPU aggregated across the cards serving each model, for each arm the run fired. ×N copies is how many routable replicas of the model the nodes held while the arm ran; a pinned baseline arm sends each model's requests to its High replica only. Raw request counts are dominated by fast rejections under overload, so compare Served, not Requests.

Model Requests Served Over cap Tokens Avg GPU % Peak GPU %
Baseline Qwen3.5-27B 256 256 0% 51.2k 43 100
Baseline Qwen3.5-9B 384 384 0% 76.8k 38.4 97
Baseline Llama-3.2-1B-Instruct 1,820 1,820 0% 364k 33 83
Baseline Phi-4-14B 1,400 838 40.1% 168k 46.6 96
Baseline Gemma4-12B-IT 960 285 70.3% 57k 29.3 96
Baseline Qwen3-1.7B 1,218 1,199 1.6% 240k 13.7 90
OpenInfer Qwen3.5-27B ×6 copies 603 597 1% 119k 76.1 98
OpenInfer Qwen3.5-9B ×6 copies 486 486 0% 97.2k 76.1 98
OpenInfer Llama-3.2-1B-Instruct ×6 copies 1,760 1,760 0% 352k 76.1 98
OpenInfer Phi-4-14B ×6 copies 1,460 1,460 0% 292k 76.1 98
OpenInfer Gemma4-12B-IT ×6 copies 551 522 5.3% 104k 76.1 98
OpenInfer Qwen3-1.7B ×6 copies 1,575 1,513 3.9% 303k 76.1 98