name: benchmark-harness
description: >
Use when running or interpreting a benchmark, adding a new workload, or
comparing justapi against Uvicorn/Granian/Hypercorn. Covers the exact
invocation, hardware assumptions, and how to distinguish real improvements
from noise.

Benchmark Harness

When to use

• Before and after every phase to record numbers in BENCHMARKS.md.
• When deciding whether a p99 regression is real (>5% threshold).
• Adding a new workload type (e.g., DB-round-trip-simulated, file serving).
• Comparing against a competing server (Uvicorn, Granian, Hypercorn).
• Validating a performance claim before writing it in docs or PLAN.md.

Setup

Required tools

# Primary HTTP load generator (outputs p50/p95/p99 natively)
cargo install oha

# Allocation profiling (pick one)
# dhat: integrated via cargo, no external install needed
# heaptrack: `sudo pacman -S heaptrack` (Arch) or `apt install heaptrack`

# CPU profiling
cargo install flamegraph

Hardware fixture

Record once in BENCHMARKS.md and reuse for all runs:

• CPU model + core count
• RAM size
• Kernel version
• Whether virtualized (bare metal vs VM vs container)
• OS distribution

Protocol

1. Warm up

Minimum 5 seconds of traffic before measuring. This ensures:

• JIT compilation (Python side) is complete
• Connection pools are established
• OS caches are warm

2. Run parameters

# Standard benchmark: 30s duration, 100 concurrent connections
oha -z 30s -c 100 --latency-correction --disable-keepalive http://127.0.0.1:8080/hello

# JSON echo: POST with nested payload
oha -z 30s -c 100 -m POST \
    -H "Content-Type: application/json" \
    -d '{"user":{"name":"test","id":42},"items":[1,2,3],"meta":{"version":"1.0"}}' \
    http://127.0.0.1:8080/echo

3. Minimum runs

3 runs per workload, report the distribution:

• p50, p95, p99 latency
• Requests/sec (throughput)
• Peak RSS (via /proc/<pid>/status → VmHWM, or time -v)

4. Record in BENCHMARKS.md

Append a new section — never overwrite old numbers. Include:

• Date
• Tool version and exact command line
• All three runs' numbers
• The median or best-of-three, clearly labeled

5. Regression check

A p99 regression >5% vs. the previous phase's recorded numbers fails the
benchmark gate unless DECISIONS.md documents an accepted trade-off with
the rationale.

Workloads

Measuring peak RSS

# Option 1: /proc (during run)
cat /proc/<pid>/status | grep VmHWM

# Option 2: GNU time wrapper
/usr/bin/time -v ./target/release/justapi serve 2>&1 | grep "Maximum resident"

Gotchas

• Single-run noise: One run proves nothing. Three+ runs establish a
  distribution. If variance is >10%, investigate before drawing conclusions.
• Comparing across hardware: Absolute numbers are hardware-dependent.
  Only compare numbers from the same fixture.
• Tool version drift: Record the exact tool version in BENCHMARKS.md.
• Background load: Ensure no competing CPU/memory load. CI runners
  are noisy — prefer dedicated hardware for release benchmarks.
• Keep-alive vs no-keepalive: Document which mode was used. Keep-alive
  measures steady-state; no-keepalive includes connection setup cost.
• Python server workers: Match worker count to justapi's thread count
  for fair comparison (e.g., uvicorn --workers 4).

References

• oha: https://github.com/hatoo/oha
• wrk: https://github.com/wg/wrk
• BENCHMARKS.md in project root
• DECISIONS.md — ADR-006 for tooling choice rationale