BENCHMARKS.md

Append-only results ledger. Never delete old numbers — the history is what
makes "we improved performance" a checkable claim instead of an assertion.

Hardware fixture

Recorded 2026-07-03. All benchmarks must use this same machine.

• CPU: 13th Gen Intel Core i5-13600K (6P+8E cores, 20 threads)
• RAM: 31 GiB DDR5
• Kernel: Linux 7.1.2-3-cachyos x86_64
• Virtualized: No (bare metal)
• OS: CachyOS

Tooling versions

Honest caveats (where JustAPI is not yet the best)

Selective benchmarking is a credibility risk, so we state the losses too:

• Raw hello-world vs Robyn. Robyn (Rust runtime, no Python GIL in the hot
  path) is commonly cited at ~40–60% faster than FastAPI on simple endpoints.
  A head-to-head Robyn number is now recorded on this hardware fixture
  (see below). After the hot-path optimization (2026-07-13) JustAPI beats
  Robyn on all three raw-throughput workloads.
• ASGI server swap vs Granian. If you only need a faster ASGI server under
  an existing app, Granian may be the lower-friction choice — it drops in
  without changing your framework. JustAPI's win is the full Rust stack
  (routing/serialization/middleware) plus agent-native serving, not merely a
  faster transport.
• Agent-native workload is the real differentiator. The benchmark that
  matters for JustAPI is structured-LLM-output streaming + MCP tool serving,
  not hello-world req/s. Those workloads are recorded in later sections; on
  pure throughput micro-benchmarks we expect to lose to purpose-built minimal
  runtimes.

We will add a Robyn row to this ledger once measured on the same fixture.

Head-to-head: JustAPI vs Robyn (recorded 2026-07-13)

Measured on the same hardware fixture (i5-13600K, 20 threads, CachyOS) with
oha -z 10s -c 100. Both servers run single-process; justapi uses its default
tokio runtime + PyO3 handler boundary, Robyn 0.88.0 uses its actix runtime.
justapi's response path was switched to orjson (Rust serializer) for this run.

Honest conclusion: Robyn is faster than JustAPI on every workload measured,
by ~1.5–2×. This matches the expectation stated in the caveats above. The
reason is structural, not a tuning gap: JustAPI dispatches every request
through the PyO3 boundary into a Python handler (request dict built in Rust →
Python, handler runs, response dict → Rust), whereas Robyn minimizes Python
involvement on the hot path. JustAPI's own validation also runs in Rust
(jsonschema), but the surrounding Python overhead still dominates, so even the
validated workload loses to Robyn+pydantic on raw req/s.

Where JustAPI is meant to win (not yet separately benchmarked): workloads
that are impossible or awkward in Robyn — streaming structured-LLM output
with per-token schema validation in Rust, MCP tool dispatch from the Rust
registry, and durable agent session state in the Rust store. These are the
agent-native differentiators; they are not micro-benchmarks and should be
measured as end-to-end agent loops, not req/s.

To actually beat Robyn on raw throughput would require Rust-native request
handlers (handlers implemented in Rust, not Python) so the PyO3 boundary is
removed from the hot path. That is a larger architectural step and is tracked in
PLAN.md / DECISIONS.md, not claimed as done here.

Head-to-head: JustAPI vs Robyn — post hot-path optimization (recorded 2026-07-13, re-run)

The table above (×1.5–2× slower than Robyn) was measured on the debug
maturin develop build before the hot-path optimization. After the changes
below, the same workloads were re-run on release builds on the same fixture
(i5-13600K, 20 threads, CachyOS), oha -z 10s -c 100, single-process:

• Rust-side response serialization (orjson called from Rust, mirroring
  Robyn's extract_response_type_fast) — removes the Python wrap_result /
  to_dict round-trip.
• Response detected via a marker attribute and serialized directly in Rust.
• Request dict is no longer built on the hot path when the handler does not
  need it (needs_request flag).
• Trace-context propagation gated behind JUSTAPI_ENABLE_TRACE (off by default).

Result: JustAPI now beats Robyn on every raw-throughput workload measured,
by ~1.2–1.5×, on release builds of both. The earlier "Robyn ×1.5–2× faster"
conclusion is superseded by this run. The one structural cost that remained
JustAPI-crossing-the-PyO3-boundary into a Python handler — is now removed for
schema-backed routes by the native Rust fast path (see below).

justapi native fast path — schema-backed Rust handlers (recorded 2026-07-14)

The "beat Robyn on raw throughput" goal is met, but a structural cost remained:
every JustAPI route still crossed the PyO3 boundary into a Python handler. For
schema-backed routes this is now optional. Registering a route with
native=True (and a Schema) serves it entirely in Rust — the request body is
validated against the JSON schema via the Rust validator and, on success, echoed
back as the response. No Python handler is invoked and no Request object is
built.

Server: python3 benchmarks/workloads_justapi.py 8090 (release maturin develop),
single process. Load: oha -z 10s -c 100 -m POST -H "content-type: application/json".
Both endpoints take the same {"id":1,"name":"x","price":1.5} body; /validate
runs a Python handler, /validate_native runs the native Rust path.

Result: the native fast path is ~12× faster than the first optimized
measurement (724,038 vs the earlier 59,666 req/s) and ~205× faster than the
equivalent Python handler route (3,531 req/s) on the same hardware fixture. The
speed-up over the 59,666 baseline came from eliminating the per-request GIL
acquire, the tokio::spawn_blocking thread hop, and per-request JSON-schema
recompilation (the schema is now compiled once per route into a
CompiledValidator shared across threads — see ADR-048/049). The entire PyO3
handler-call boundary and Python Request build are skipped. This exceeds
Robyn's validated-workload number (32,919 req/s) by ~22× for routes that opt into
native=True — JustAPI serves schema-validated JSON with no Python involvement.

Measured range: 410k–724k req/s across re-runs on the same fixture
(machine-load variance). Both are ~7–12× the 59,666 first cut.

Caveat: native=True requires a registered Schema (the route falls back to the
normal Python path if no schema is present). The response is the validated request
body echoed verbatim; handlers that transform the body must keep using a Python
handler.

Known issue — non-native deadlock at high concurrency (see ADR-049): the
Python dispatch path (spawn_blocking + Python::attach) hard-stalls at
~100 concurrent connections for every non-native route, including the
simplest handler (≈16–20 req/s, all connections aborted). The native fast path
is immune because it never acquires the GIL or enters the blocking pool. This
is a pre-existing server-level GIL/blocking-pool deadlock, not introduced
by the native optimization; it must be fixed before non-native routes are
production-viable at scale (recommended fix: a dedicated GIL thread-pool).

Baseline: Uvicorn+FastAPI (recorded 2026-07-03)

Workload: hello-world

Server config:

uvicorn benchmarks.workloads_fastapi:app --host 127.0.0.1 --port 8080 --workers 4

Load command:

oha -z 30s -c 100 http://127.0.0.1:8080/hello

Workload: JSON-echo (nested payload)

Load command:

oha -z 30s -c 100 -m POST \
    -H "Content-Type: application/json" \
    -d '{"user":{"name":"test","id":42},"items":[1,2,3],"meta":{"version":"1.0"}}' \
    http://127.0.0.1:8080/echo

Baseline: Granian (recorded 2026-07-03)

Workload: hello-world

Server config:

granian --interface asgi benchmarks.workloads:app --host 127.0.0.1 --port 8080 --workers 4

Load command:

oha -z 30s -c 100 http://127.0.0.1:8080/hello

Workload: JSON-echo (nested payload)

Load command:

oha -z 30s -c 100 -m POST \
    -H "Content-Type: application/json" \
    -d '{"user":{"name":"test","id":42},"items":[1,2,3],"meta":{"version":"1.0"}}' \
    http://127.0.0.1:8080/echo

justapi Phase 1 — Minimal Viable Runtime (recorded 2026-07-03)

Server binary: cargo run --release -p justapi-cli -- serve --addr 127.0.0.1:8080
Response: {"message":"hello"} (19 B for hello-world, 73 B for echo)

Workload: hello-world

Workload: JSON-echo (nested payload)

Benchmark gate verdict

PASS — within 2x of Granian hello-world (actually 2.4x faster at 765k vs 314k req/s).

Comparison vs baselines

justapi Phase 2 — Native Router (recorded 2026-07-03)

Route lookup benchmark (500-route table)

Methodology: matchit-based radix trie router, per-method HashMap dispatch.
500k lookups across 5 different path patterns (static, param, nested param,
catch-all, miss). Release build on hardware fixture.

Overall throughput (no regression)

Hello-world throughput identical to Phase 1 (760k req/s, p99 0.48ms).
Router overhead is negligible in the hot path.

justapi Phase 3 — Memory & Zero-Copy Pipeline (recorded 2026-07-03)

Arena + BufferPool integration

Server binary: cargo run --release -p justapi-cli -- serve --addr 127.0.0.1:8080

SharedArena (per-connection Mutex<RequestArena>) and BufferPool (thread-safe 4-bucket pool)
integrated into the request pipeline. Arena reset per request; pool provides reusable response buffers.

Workload: hello-world

Load command:

oha -z 30s -c 100 http://127.0.0.1:8080/hello

Benchmark gate verdict

PASS — 752k req/s (no regression vs P2 760k, within measurement noise).

justapi Phase 5 — Middleware Engine + Auth (recorded 2026-07-03)

Middleware chain integration

Server binary: cargo run --release -p justapi-cli -- serve --addr 127.0.0.1:8080

Middleware chain (Middleware<B> trait, MiddlewareChain<B>, Next<'a, B>) integrated
into server pipeline. Cors + SecurityHeaders tested for overhead measurement.

Workload: hello-world (no middleware — baseline)

Workload: hello-world (CORS + SecurityHeaders)

Benchmark gate verdict

PASS — middleware overhead = 0.18% (< 5% target).

justapi Phase 6 — Serialization (recorded 2026-07-03)

Payload: nested JSON (~60 bytes, user+items+meta)

Benchmark: cargo run --release -p justapi-bench (with --features simd-json for second row).

Key findings

1. serde_json is faster than simd-json for serializing small JSON payloads
   typical of web API responses (50-200 bytes). simd-json's SIMD setup overhead
   doesn't pay off at this scale.
2. simd-json may be beneficial for deserialization of large request bodies
   (>1KB), but that isn't benchmarked here.
3. Hot path unaffected — the current hello-world/echo handlers use
   pre-formatted static strings, not runtime serialization.

Decision

Default to serde_json for all serialization. The simd-json feature flag
remains available for opt-in when request body parsing is added (Phase 7+).

Phase status

• serialize module with to_json_string/to_json_vec abstraction
• simd-json feature flag (optional dep, enables simd-json backend)
• Benchmark results recorded
• 35 tests pass, clippy + fmt clean

justapi Phase 7 — TLS + HTTP/2 (recorded 2026-07-03)

TLS termination with rustls + ALPN for HTTP/2

Server binary: cargo run --release -p justapi-cli --features tls -- serve --addr ...

TLS uses rustls 0.23 with ring-based crypto provider. Self-signed cert generated
with openssl. ALPN configured for h2 and http/1.1. HTTP/2 auto-negotiated
when the client supports it (oha uses HTTP/1.1 over TLS in these benchmarks).

Workload: hello-world (plain TCP — baseline)

Workload: hello-world (TLS)

TLS overhead

(777734 - 694233) / 777734 = 10.7% — under the 15% target.

Benchmark gate verdict

PASS — TLS overhead 10.7% (< 15% target).

Feature flag

TLS is behind the tls feature flag (justapi-core, justapi-cli):

• cargo build --release -p justapi-cli --features tls
• justapi serve --addr 0.0.0.0:8443 --tls-cert cert.pem --tls-key key.pem

Without the feature flag, the server has zero TLS overhead (no rustls dependency).

justapi Phase 8 — WebSocket + SSE (recorded 2026-07-03)

SSE streaming endpoint

GET /events — returns 10 SSE events (100ms apart), text/event-stream content type.
Built on tokio::sync::mpsc + tokio_stream::wrappers::ReceiverStream + http_body_util::StreamBody.

No throughput benchmark for SSE — streaming endpoints are long-lived connections,
not request/response hot paths.

WebSocket echo handler

GET /ws — full-duplex echo server via tokio-tungstenite.
TCP-peek approach: peek() first bytes to detect Upgrade: websocket before
passing to tokio_tungstenite::accept_async, bypassing hyper's upgrade mechanism
(which caused ResetWithoutClosingHandshake errors).

Integration test confirms round-trip: send "hello" → receive "hello".

Feature flag

WebSocket is behind the ws feature flag:

• cargo build --release -p justapi-cli --features ws
• tokio-tungstenite 0.24 + sha1 0.10 + base64 0.22 (optional deps)

SSE is always available (no feature gate).

Test results

Exit criteria

• SSE streaming endpoint working with StreamBody
• WebSocket echo handler with tokio-tungstenite
• TCP-peek approach avoids hyper upgrade issues
• Generic handle_ws_raw<S> works over plain TCP and TLS
• 37 tests pass (32 lib + 5 integration), clippy + fmt clean
• Feature combos tested: none, ws, tls, simd-json, tls+ws

Phase 16 — K8s Overhead vs Bare Metal (recorded 2026-07-05)

Methodology

JustAPI justapi serve binary deployed in two configurations:

1. Bare metal: running directly on the hardware fixture (Linux)
2. K8s (kind): same binary running in a kind (Kubernetes-in-Docker) cluster on the same machine,
   with 1 replica, no middleware, no TLS

Both configurations serve the same hello-world endpoint. Load generated with oha
from the same machine (localhost). K8s overhead includes:

• Container runtime (containerd) overhead
• Network proxy (kube-proxy + CNI) overhead
• Service routing through ClusterIP

Workload: hello-world

Bare metal (from Phase 1):

K8s (kind, 1 replica, ClusterIP):

Overhead calculation

Benchmark gate verdict

PASS — K8s overhead < 15% on throughput and latency. The 8.4% throughput
degradation is primarily from container networking (kube-proxy iptables rules +
CNI encapsulation). Memory overhead is expected from the container runtime and
is consistent with other workloads (kind adds ~10-15 MB per container).

Key findings

1. CPU-bound workloads see minimal K8s overhead (~3-5%) when running without
   CNI encryption. The overhead here (8.4%) is from kind's nested networking.
2. Memory overhead (~25 MB vs 12 MB) is dominated by the container runtime
   and OS layer, not the application itself.
3. Production K8s (GKE, EKS, AKS) typically adds 5-10% latency vs bare metal
   due to the CNI plugin (Calico, Cilium, AWS VPC CNI).
4. Recommendation: For latency-sensitive workloads, use hostNetwork or
   ALB/NEG direct pod routing to bypass kube-proxy.

Load command

oha -z 30s -c 100 http://127.0.0.1:8080/hello    # bare metal
oha -z 30s -c 100 http://localhost:30080/hello     # K8s (NodePort)

justapi Phase 18 — DX Tooling (recorded 2026-07-05)

Built-in Profiler

Server binary: justapi serve
Profiler: justapi profile --duration 5 --connections 50

Workload: 404 Not Found (baseline throughput test)

Benchmark gate verdict

PASS — justapi new -> justapi serve -> justapi profile executes in < 5 seconds.
The hot-reload and built-in profiler work efficiently without dragging down the core runtime.

justapi Phase 20 — Benchmark & Optimization (recorded 2026-07-05)

Server binary: python3 benchmarks/workloads_justapi.py 8080 (Native Tier B Python handler)

Workload: hello-world

Workload: JSON-echo (nested payload)

Benchmark gate verdict

PASS — JustAPI's native Python API beats FastAPI significantly on throughput (39.5k vs 2.4k req/sec on hello-world, ~16x faster) and JSON echo (25.9k vs 2.4k, ~10x faster). Latency is also substantially better (10ms p99 vs 44ms p99).

JustAPI native API — verified run (2026-07-09)

Hardware matches the recorded fixture exactly (i5-13600K, 20 threads, 31Gi,
CachyOS, oha 1.14.0), so these are directly comparable to the
Uvicorn+FastAPI baseline above. Server: single process (JustAPIApp.run,
one Rust server + one Python interpreter — no multi-worker fan-out).

The earlier JustAPI entry in this ledger (≈39.5k req/s hello) predates the
current server/native-API implementation and was measured under a different
(now-obsolete) configuration; this run supersedes it.

Load command (3 runs each, median reported):

oha -z 10s -c 100 http://127.0.0.1:8080/hello
oha -z 10s -c 100 -m POST -H "Content-Type: application/json" \
    -d '{"user":{"name":"test","id":42},"items":[1,2,3],"meta":{"version":"1.0"}}' \
    http://127.0.0.1:8080/echo

Workload: hello-world (GET)

Workload: JSON-echo (nested payload, POST)

Verdict vs Uvicorn+FastAPI baseline (same hardware, 4 workers)

• hello-world: ~324k vs 36k req/sec → ≈9x throughput; p99 1.10 ms vs 24.63 ms → ≈22x lower tail latency — and JustAPI used a single process vs FastAPI's 4 workers.
• JSON-echo: ~268k vs 26k req/sec → ≈10x throughput; p99 1.64 ms vs ~14.5 ms → ≈9x lower tail latency.
• Memory: 69.8 MB peak (single process) vs 29 MB baseline (note: baseline RSS was per-worker-ish; JustAPI figure is the whole process).

GATE: PASS — JustAPI's native Python API beats the Uvicorn+FastAPI baseline decisively on both throughput and tail latency, with no multi-worker fan-out required.

justapi Phase 40 — Startup Latency & Docker Image (recorded 2026-07-10)

Hardware

Same fixture as above (i5-13600K, 20 threads, 31 GiB, CachyOS). Single process,
no multi-worker fan-out.

Startup time to first response

Measured from process launch until the first successful HTTP 200 on /hello.
Lower is better; FastAPI target is sub-10 ms for the CLI.

Notes:

• CLI startup is dominated by thetokio runtime + listener bind; well under the
  10 ms target.
• Python native startup includes CPython interpreter init + PyO3 attach +
  tokio runtime spawn + OpenAPI spec generation from registered routes. The
  first-response window (~165 ms) is one-time process startup, not per-request.

Docker image

Build: docker build -t justapi:2.0.0 . (features tls,compression, binary
stripped). The runtime stage carries only the static Rust binary + ca-certificates

• libssl3 — no Python interpreter is shipped in the container (the Python
  native API is distributed separately via maturin/pip). Smoke test confirmed
  GET /hello → {"message":"hello"} and POST /echo round-trip inside the
  container.

Test-suite gate (all green)

Python packaging

• Single abi3 wheel (cp311-abi3) covers CPython 3.11–3.14 from one build.
• justapi PyPI name is available (verified 404 on pypi.org/pypi/justapi).
• Wheel installs and imports in a clean venv; from justapi import JustAPIApp, Schema, Depends, … all resolve.
• NOTE: the wheel built on this host is tagged manylinux_2_34 (Arch glibc
  2.41). For PyPI upload it must be rebuilt inside a manylinux_2_28 container
  (maturin auto-detects the policy there); see publish notes below.

Publish command (requires PyPI API token)

# Build portable manylinux_2_28 wheels (Linux x86_64; add targets for aarch64/osx)
docker run --rm -v "$PWD":/io -w /io ghcr.io/pyo3/maturin:latest \
  build --release --target x86_64-unknown-linux-gnu

# Upload
maturin publish --skip-existing   # uses MATURIN_PYPI_TOKEN env var
# or: twine upload --skip-existing target/wheels/justapi-2.0.0-*.whl

Phase 48 — Speculative decoding (draft-target), CPU proof-of-mechanism

Recorded 2026-07-10. No GPU in CI, so this is a structural benchmark: it
proves the speedup mechanism (acceptance rate → effective tokens/verify-step)
on the weight-free MockModel, not wall-clock GPU tokens/sec. The Model
trait's forward_logits single-step primitive is the same one the real
(candle, GPU) path will drive, so these ratios transfer directly.

Workload: greedy decode, max_tokens = 256, draft == target (perfect draft)

Interpretation: with a perfectly-aligned draft, gamma = 8 emits ~8.8 tokens
per verify step instead of 1 — i.e. up to ~8.8x fewer target forward passes
for the same output, which is exactly the vLLM/SGLang speculative-decoding
win. Effective tok/step ≈ gamma + 1 as long as acceptance ≈ 1.0.

Workload: imperfect draft (draft offset ≠ target offset)

Interpretation: when the draft is a poor predictor the algorithm degrades
gracefully to plain target decode (no worse than no speculation). This is the
correctness/safety guarantee that makes it safe to always run speculation.

What this does NOT yet measure (gated on GPU)

• Wall-clock tokens/sec on a real 7B-class model (needs --features real + CUDA).
• Disaggregated prefill/decode (independent GPU pools) — Phase 48 next item.
• Tree-based Medusa/EAGLE verify → delivered in Phase 49 (see below).

These are recorded as TODOs against the Phase 48 gate; the numbers above are the
mechanism proof, not the production throughput claim.

Phase 49 — Tree-based speculative decoding (acceptance comparison)

Recorded 2026-07-10. Structural benchmark: no GPU, weight-free MockModel

• RangeModel alignment control. Proves the tree-verify mechanism raises
  acceptance rate vs draft-target at the same gamma by giving the draft multiple
  candidates per position.

Method

The AcceptanceStats fields are unified for draft-target and tree modes:

• total_draft: number of positions speculated (γ per step) — comparable
  across modes.
• total_accepted: number of those positions where at least one draft
  candidate matched the target.
• tree_branch: 0 for draft-target, >0 for tree.
• tree_nodes_verified: total tree nodes the target scored (cost metric).

Workload: imperfect draft, max_tokens = 256, gamma = 4

Target: MockModel (predicts (prev+1)%V). Draft: RangeModel(offset=0, spread=2) — centered on (prev+1)%V with ±2 falloff, so the correct token
appears in the top-2 and top-3 but not in top-1 (it is not the most likely
token for the draft to predict). This models a realistic scenario where the
draft and target agree on the set of plausible tokens but disagree on
ordering.

Interpretation: with branch=3, the tree explores 3 candidates per position
(120 nodes per 4-position step), raising acceptance from 13% to 41% — 3×
higher. Verified nodes (120) vs draft-target (4) represents 30× more target
forward calls per step; in a real system this cost is amortised by
tree-attention batching (all nodes at a given depth verified in one forward
pass). Even with branch=2 (30 nodes per step) the acceptance more than
doubles.

Lossless guarantee

All three modes produce the same token stream as plain target decode (verified
by tree_and_draft_target_produce_same_output and
tree_matches_target_when_draft_contains_correct_path tests).

What this does NOT yet measure (gated on GPU)

• Wall-clock tokens/sec with tree-attention batching (NVIDIA GPU + CUDA).
• Optimal branch factor vs γ trade-off on real model loss distributions.
• Dynamic tree pruning (stop expanding branches whose score falls below a
  threshold).

Phase 50 — RadixAttention prefix caching (scheduler integration)

Recorded 2026-07-10. Two structural results, both GPU-free:

1. Tree vs flat hash-map memory (bench_nested in radix_cache.rs): a
   chat-history workload where request r shares a growing prefix chain with
   prior requests. Radix stores each block once (O(N) block-slots); the flat
   PrefixCache duplicates block IDs across every distinct prefix entry
   (O(N²) block references).
2. Scheduler reuse: the Scheduler now owns the RadixPrefixCache and
   reuses matched KV blocks on admission, skipping prefill for the shared
   prefix.

1. Memory: radix vs flat (chat-history workload)

10× more requests → radix grows ~10× (linear), flat grows ~100× (quadratic).

2. Scheduler reuse (verified by schedule_radix_reuses_* tests)

Eviction authority

A single authority owns cached blocks: the scheduler drives eviction through
RadixPrefixCache::evict_filter (skips leaves still referenced by a live
sequence) and recycles them with KvBlockPool::free_cached. Cached blocks are
pinned so the pool's clock-sweep cannot evict them out from under the tree —
this avoids the stale-block-id class of bug. Finished-sequence blocks are now
promoted into the cache (previously they leaked).

What this does NOT yet measure (gated on GPU)

• Wall-clock TTFT/ITL improvement from prefix reuse on a real model
  (--features real + CUDA + weights): the scheduler-level prefill-skip is
  proven structurally, but the token-compute savings need a real forward pass.
• Token-content hashing for cache-key validation (today the prefix key is the
  raw token sequence; a content hash would let the same block serve
  semantically-equal-but-token-distinct prompts).

Phase 48 — Disaggregated prefill/decode benchmark (structural)

Recorded 2026-07-10. Like the speculative-decoding section, this is a
structural benchmark: no GPU in CI, so we drive the real JustAPI
schedulers (PdScheduler for disaggregated P/D, Scheduler for a collocated
pool) over a parameterized synthetic GPU-cost model and measure the scheduling
metrics that define LLM-serving quality. The point is to prove the
disaggregated topology's benefit (the vLLM/SGLang thesis) at the scheduler
level, not to claim wall-clock GPU tokens/sec.

Cost + parallelism model

• Prefill cost scales with prompt tokens in a step (compute-bound).
• Decode cost is one fixed step regardless of batch size
  (memory-bandwidth-bound) — the standard vLLM/SGLang assumption.
• Disaggregated = independent prefill/decode pools modelled with parallel
  virtual clocks (separate GPUs); decode ITL is decoupled from prefill.
• Collocated = one pool, one shared clock; prefill and decode serialised on
  the same budget, so decode ITL is inflated by co-scheduled prefill work.

Prompt lengths are varied per request (64–256 tok) so sequences finish prefill
at different steps and prefill/decode overlap — the condition under which the
collocated scheduler's ITL degrades.

Harness: cargo run --release -p justapi-bench --bin justapi-bench-inference
(CPU fixture, i5-13600K).

Workload: 16 burst requests, prompt 64–256 tok, max_tokens 64

Interpretation: under concurrent load with overlapping prefill/decode, the
collocated scheduler lets prefill chunks steal the shared step budget,
spiking ITL p99 to 1.16 ms (5.8x the raw decode cost) and halving
throughput. Splitting prefill and decode onto independent pools keeps
decode ITL flat at the raw 0.20 ms decode cost and doubles effective
throughput, because the decode GPU is never starved by prefill. This is the
exact mechanism DistServe / NVIDIA Dynamo rely on — now expressed in JustAPI's
PdScheduler and reproducible without a GPU.

What this does NOT yet measure (gated on GPU)

• Real wall-clock tokens/sec on a 7B-class model (--features real + CUDA).
• KV-tensor transfer bandwidth prefill→decode (the TransferableSequence today
  carries logical block ids + token count; a real engine would copy tensors and
  bill num_prefilled tokens as transfer volume).
• Tree-based Medusa/EAGLE verify → delivered in Phase 49 (see above).

These remain TODOs against the Phase 48 gate; the table above is the
scheduler-topology proof, not the production throughput claim.

Phase 51 — Scheduler Engine real wall-clock throughput (CPU fixture)

Recorded 2026-07-10. Unlike the structural benchmarks above, this runs the
actual generation path (naive Engine::generate vs scheduler-backed
SchedulerEngine::generate) and measures real token throughput on the CPU
fixture with MockModel (instant forward pass). The goal is to quantify the
scheduler's per-token overhead — thread hop, lock contention, sampling — not to
claim LLM tokens/sec (that needs a GPU).

Harness: cargo run --release -p justapi-bench --bin justapi-bench-inference
(CPU fixture, i5-13600K, dev profile — release was not used for this first run).

Workload: 8 requests, 8 prompt tokens, max_tokens 16 each (MockModel)

The scheduler loop's per-iteration thread::sleep(Duration::from_millis(1))
when no work is ready, combined with mutex contention on every shared state
access, dominates on MockModel where forward_logits returns instantly.
On a real GPU (where a single forward pass takes 5–50 ms) this overhead is
negligible — the latency budget is dominated by the CUDA kernel, not the
CPU scheduler.

Sampling-parameter plumb

The scheduler loop previously used greedy argmax only. With this phase,
sample_token_with_params in scheduler_engine.rs now honors temperature,
top_k, and top_p from the per-sequence SamplingParams — matching the
behaviour of the naive Engine::generate path.

Benchmark gate verdict

PASS — the scheduler correctly interleaves concurrent requests with
continuous batching, exposes prefix-cache metrics, and honours all standard
sampling parameters. The real-GPU throughput gain (amortising kernel launch
overhead across a batch) is gated on --features real + CUDA.

What this does NOT yet measure (gated on GPU)

• Wall-clock tokens/sec on a 7B-class model vs vLLM/SGLang.
• TTFT improvement from prefix-cache reuse of KV blocks on GPU.
• SchedulerEngine throughput with non-trivial forward_logits duration (where
  the 1 ms sleep is negligible).

GIL-pool worker-count fix (2026-07-14)

Context. The GIL pool (crates/justapi-py/src/gil_pool.rs) previously sized
itself with available_parallelism() (20 workers on this box) for all runtimes.
For standard CPython (GIL enabled) this is catastrophic: N threads all call
Python::attach (PyGILState_Ensure/Release) per job and contend for the
single GIL, so each job paid ~170 µs of GIL-switch overhead on top of the
~30 µs FFI floor.

Fix. default_pool_size(mode) now returns 1 for GilBased (one worker
holds the GIL and drains its queue — no per-job acquire/release contention) and
available_parallelism() for GilFree (free-threaded Python runs truly in
parallel).

Measurement hygiene. Two earlier confounds were found and removed:

• benchmarks/workloads_body_test.py was broken (JustAPI(__name__) passed the
  string "__main__" as dependencies → str + list TypeError; and app.run(host=, port=) is invalid — run takes a single addr string). Its "1.2M RPS" figure never ran.
• oha 1.14.0 rejects --body; correct flag is -d. Prior RPS numbers quoted
  with --body were from oha erroring out.

Results (release build, oha -c 100 -z 6s -m POST -d '{"a":1}', CPython 3.14, GIL enabled, pool=1):

All routes ~110k RPS. Schema import and native=True registration do NOT
affect sync-route throughput — the earlier "200× slowdown" was a measurement
artifact (broken baseline + wrong oha syntax + debug build + pool=20 contention).
The ~110k ceiling is the GIL-serialized single-worker Python floor, comparable
to FastAPI for equivalent handlers.

Native fast path (native=True + Schema, validated + responded entirely
in Rust, no Python handler call) — benchmarks/workloads_native_bench.py:

The native fast path is ~4× the GIL-path ceiling, since it skips the GIL
entirely. This is the framework's headline performance path and the lever for
beating FastAPI on validated routes.

FastAPI apples-to-apples comparison (2026-07-14)

Same machine, CPython 3.14, oha -c 100 -z 6s -m POST -H "Content-Type: application/json".
justapi: release build, GIL pool = 1 worker. FastAPI: uvicorn workloads_fastapi:app
(single worker, default loop) — its per-request ASGI overhead caps it at ~8k RPS
even for a no-op in this environment.

Takeaway. justapi beats FastAPI by ~14× on equivalent sync handlers and
~60× on schema-validated routes (native fast path runs entirely in Rust, no
Python). The GIL-path ceiling (~110k) is still ~13× FastAPI, so there is headroom
to push it higher by shrinking the per-request FFI cost (see next section).

Native query fast path (GET) — 2026-07-14

The native fast path (validate-and-echo, no Python) was extended from request
body validation to request query-string validation. A GET route
registered with native=True + query_schema is validated and echoed in Rust:
the query is parsed, each value coerced to a JSON scalar ("30" -> 30,
"true" -> true) so typed JSON Schemas validate, and the parsed object is
returned — no GIL, no Python handler call. This lets body-less routes qualify
for the ~450k native path.

benchmarks/workloads_native_query.py (oha -c 100 -z 6s, CPython 3.14,
release build, GIL pool = 1 worker):

The native query fast path is ~5× the GIL-path GET ceiling (~91k), matching
the body fast path (~432k). Invalid queries return 422
(application/problem+json) without touching Python.

API. All route methods now accept query_schema (a JSON Schema dict,
Schema subclass, pydantic model, or JSON string) and native is now accepted
on get/delete/head/options/trace as well as post/put/patch/query.
Request values are coerced to JSON scalars before validation, mirroring
pydantic/FastAPI query coercion.

Implementation notes.

• crates/justapi-py/src/native/handlers.rs: new try_native_fast_path_query
  (parallel to try_native_fast_path); parses query with serde_urlencoded,
  coerces values, validates via justapi_core::validate::validate_json_schema,
  echoes on success / 422 on failure.
• query_schema_jsons added to AppState, threaded through make_native_handler
  and make_test_handler (test client) so the fast path works under AsyncTestClient.
• resolve_schema_json now also accepts a raw dict (serialized to a JSON string).

Production hardening (2026-07-14)

The 2026-07-03 production-readiness audit (the audit is being tracked in
conversation, not committed) is now closed for the runtime-crash / DoS /
exception-leak class. Items addressed:

• GIL-worker panic safety (CRITICAL). A Rust panic must not unwind across
  the pyo3 FFI boundary (undefined behaviour). The GIL job closure in
  crates/justapi-py/src/gil_pool.rs does not wrap each job in
  catch_unwind (that measured as a ~3x regression on the GIL path); instead the
  workspace root [profile.release] builds with panic = "abort", so a genuine
  panic safely aborts the process and the supervisor (Docker/k8s, systemd)
  restarts it. Python exceptions remain a separate, safe path (surfaced as
  PyErr). A handler error (not a panic) still returns a generic 500 {"error":"internal server error"} with no exception text leaked.
• Native fast-path auth bypass (CRITICAL). A native=True route that also
  declares dependencies or middlewares would silently skip auth (the native
  path validates+echoes without calling the Python handler). App._wrap_handler
  / _wrap_batch_handler now raise ValueError on that combination. Verified by
  test_native_fastpath.py::test_native_with_dependencies_rejected.
• Exception-string leakage (CRITICAL). Native-handler 500s no longer include
  the exception text in the body; they return the same generic JSON as above.
• Request / header-read timeouts (HIGH). request_timeout() (env
  JUSTAPI_REQUEST_TIMEOUT_SECS, default 60s) wraps the handler chain in both the
  plaintext and TLS serve paths; on expiry the client gets 504 {"error":"request timeout"}. header_read_timeout (30s) is set on both hyper
  builders via TokioTimer (required by hyper for header timeouts).
• Connection-flood cap (HIGH). max_connections() (env JUSTAPI_MAX_CONNECTIONS,
  default 10000) sizes an Arc<Semaphore>; every accepted connection holds a
  permit for its lifetime, bounding concurrent sockets.
• SIGTERM (HIGH). App.run now selects on ctrl_c and SIGTERM
  (unix signal::SignalKind::terminate); either triggers graceful drain.
• Builtin routes (HIGH). /health, /live, /ready, and /metrics are
  registered as Python builtin routes in JustAPIApp.__init__
  (crates/justapi-py/python/justapi/app.py) — all return 200; /metrics is
  served as text/plain via PlainTextResponse. This fixes the earlier gap
  where those endpoints 404'd. (Note: the with_default_routes() Rust call was
  reverted — calling it after with_handler() replaces the Python handler
  with the core default router and 404s every user route, so builtin routes are
  done in Python instead.)
• GIL-worker count knob (MED). default_pool_size honors the
  JUSTAPI_GIL_WORKERS env var to override the pool size (defaults to 1 for
  GilBased, available_parallelism for free-threaded).

Gates re-run this session (changed crates). cargo check -p justapi-core and
cargo check -p justapi-py clean; cargo clippy -p justapi-core -p justapi-py --tests clean; cargo test -p justapi-core -p justapi-py -> 12 integration
tests pass + doc tests pass. cargo fmt --check cannot pass on this stable
toolchain because rustfmt.toml requires nightly-only features (wrap_comments,
group_imports, imports_granularity); this is an environment limitation, not a
code regression.

Resolved (dead code removed). MED#13 (// SAFETY: for the unsafe mmap in
justapi-inference/src/real/model.rs:227) is done. MED#9 ("H3 skips middleware")
is closed: the http3 transport was dead code — crates/justapi-core/src/server/http3.rs
existed but was untracked and never declared as a module (no mod http3;), so it
was never compiled and serve_http3 was never called, and enabling the http3
feature would have failed to compile. On 2026-07-15 the dead code was removed:
deleted server/http3.rs, dropped the http3 feature + quinn/h3/h3-quinn
deps from justapi-core, removed the http3 feature from justapi-py, deleted
the enable_http3 Rust method + Python wrapper + .pyi stub + test_http3.py,
and removed the #[cfg(feature = "http3")] wiring in app.rs. The audit's
hardening (timeouts, connection cap, panic safety, middleware) therefore covers
every compiled path (HTTP/1.1 plaintext + TLS). If H3 is ever wanted, it must be
re-added as a fully-wired module with the same chain.run timeout +
connection-semaphore treatment as the other transports (and ADR-046 revisited).

Production-hardening regression check (2026-07-15)

Re-ran the AGENTS benchmark gate after the hardening changes + the panic = "abort" move to the workspace root. Built with maturin develop --release
(extension now compiled with panic = "abort"), replayed via oha -z 10s -c 100 --latency-correction against benchmarks/workloads_native_query.py (port 8263).

No regression on either path. The native fast path holds its ~440k RPS
order-of-magnitude lead over the GIL path (~89k) — i.e. ~5x, exactly the design
intent of the native path.

GIL-path environment sensitivity (note for future benches). The single-worker
GIL path is very sensitive to machine state. On a separate run earlier in the
same session it measured ~30k RPS; a bisect proved this was not caused by the
hardening (removing the (reverted) catch_unwind AND the tokio::time::timeout
wrapper both left it at ~30k, while the native path stayed healthy at ~425k, and
adding GIL workers hurt — 1→30k, 4→11k, 8→8k, classic GIL contention). The
30k reading was environmental (thermal/background noise/scheduler), not a code
regression; a subsequent run returned to ~89k. Conclusion: treat the GIL-path
number as a soft ceiling that varies run-to-run on this box; only compare it
against a fresh baseline taken on the same machine state. The native path is
the stable, regression-relevant number.

Endpoint smoke (also verified): /health → 200, /metrics → 200
(text/plain), /ready → 200, /live → 200 — the HIGH#7 gap is closed.

Full gate status: cargo clippy --workspace --tests -D warnings clean (the
earlier "profiles for the non root package will be ignored" warning is gone now
that panic = "abort" lives in the workspace root [profile.release]);
cargo test --workspace green (all suites pass: cli/core/integration/inference/
engine + doc tests); maturin develop --release builds clean;
test_native_fastpath.py → 9 passed. cargo fmt --check still cannot pass on
stable (nightly-only rustfmt.toml features) — environment limitation, unchanged.