swadhin / justapi public

📝 p3: ADR-090/091 — native-awaitables experiment: asyncio floor unbeatable

swadhinbiswas committed 2 months ago

28d6fda

1 parent 1ad2e27

commit 28d6fda

1 file changed

+280 -0

Files

DECISIONS.md +280

Showing 1 changed file+280-0

DECISIONS.md

▼DECISIONS.md

+280-0

@@ -3284,3 +3284,283 @@ c=200 async = 4.9k, 100% success. Full gates green (fmt/clippy/workspace
3284	3284	15 suites/pytest 171).
3285	3285
3286	3286	Evidence: app.rs run() diff, thread-count probe, c=500 load run, gate runs.
	3287	+
	3288	+## ADR-086 — 2026-08-06 — Async handler resolution: callback-driven completion (2× async throughput)
	3289	+
	3290	+Context: The async handler path (ADR-083) resolved the coroutine via
	3291	+`spawn_blocking` + `future.result()`, which added a thread hop and capped
	3292	+async handlers at ~5.2k RPS (Granian: ~44k on the same fixture — Granian runs
	3293	+the coroutine directly on its event loop). This is the remaining async gap.
	3294	+
	3295	+Decision: Replace `spawn_blocking`-wait with a **callback-driven
	3296	+completion**: a new `_DoneNotifier` pyclass whose `__call__` fires a tokio
	3297	+oneshot when the `concurrent.futures.Future` completes (registered via
	3298	+`future.add_done_callback`). `resolve_async_response` now:
	3299	+1. Registers the notifier on a spawn_blocking thread (never on server threads,
	3300	+ to avoid GIL contention with sync handlers),
	3301	+2. `await`s the oneshot (pure tokio, no thread blocked, no polling),
	3302	+3. Reads + serializes the result on one spawn_blocking thread.
	3303	+
	3304	+Measured (release wheel, same fixture): async 1ms-sleep handler c=100:
	3305	+5,152 → 9,800 RPS (≈2×), with zero sync regression (sync-only 107-114k
	3306	+both before and after; the 122-128k earlier readings were machine variance).
	3307	+All 171 pytest + workspace suites pass. New stress test
	3308	+`test_async_callback_path_concurrent_correctness` (70 concurrent async
	3309	+requests: success + HTTPException + streaming).
	3310	+
	3311	+Honest remaining gap: async is still ~4.5× behind Granian (9.8k vs 44k) —
	3312	+the coroutine still crosses GIL worker → asyncio loop thread → spawn_blocking.
	3313	+Closing it requires dispatching async handlers directly onto the loop (known
	3314	+at registration via `is_async`), skipping the GIL worker entirely — a larger
	3315	+refactor, tracked as a follow-up.
	3316	+
	3317	+Pre-existing bug found (not a regression): real-server HTTP streaming of
	3318	+`TokenStreamResponse` returns an empty body (test-client streaming works).
	3319	+Confirmed present in the baseline before this session's async work. Tracked
	3320	+separately.
	3321	+
	3322	+Evidence: types.rs `_DoneNotifier`, handlers.rs resolve_async_response,
	3323	+stress test, live oha measurements.
	3324	+
	3325	+## ADR-087 — 2026-08-06 — Async dispatch: measured floor, why parallel thrashes, the loop-thread path
	3326	+
	3327	+Context: Battle to beat Granian on async handlers. Granian (GIL-locked
	3328	+CPython, 1 worker): 129k RPS async no-sleep, 44k with 1ms sleep. JustAPI:
	3329	+9.2k no-sleep, 8.5-9.8k with 1ms sleep. The gap is ~14× on dispatch-heavy
	3330	+load.
	3331	+
	3332	+Measurements (this fixture, release wheel):
	3333	+- `run_coroutine_threadsafe` + `.result()` alone: **21µs sequential, 10µs
	3334	+ concurrent** (loop hop floor).
	3335	+- JustAPI async per-request (profiler): `avg_request_build=1.3µs`,
	3336	+ `avg_handler=66µs`. The 66µs = 21µs loop hop + ~45µs GIL-worker hop
	3337	+ (channel send + `Python::attach` + oneshot).
	3338	+- Granian's ~22µs/request total = the loop hop alone — it dispatches
	3339	+ directly to its asyncio loop with no worker intermediary.
	3340	+
	3341	+Experiment (rejected): parallel dispatch via `spawn_blocking` for async
	3342	+handlers (`run_python_parallel`) — c=100 concurrent `Python::attach` +
	3343	+request-dict builds on blocking threads thrash the GIL and regressed
	3344	+async (9.8k → 5.0k). The single GIL worker's serialization is optimal on
	3345	+GIL-locked CPython for object-building work.
	3346	+
	3347	+Decision: ship the callback-driven completion (ADR-086, 5.2k → 8.5-9.8k)
	3348	+and document that beating Granian on async requires the **loop-thread
	3349	+dispatch**: build the Python `Request` and invoke the handler ON the asyncio
	3350	+loop thread (via `call_soon_threadsafe`), eliminating the GIL worker hop
	3351	+entirely — matching Granian's model where the loop is the sole Python
	3352	+dispatcher. This is a scoped refactor of `call_python_handler` (move
	3353	+request-build + handler call to the loop). Not attempted in this session
	3354	+because the request-build currently lives in Rust (call_python_handler) and
	3355	+moving it to the loop thread changes where GIL work happens.
	3356	+
	3357	+Honest position: on GIL-locked CPython, the loop-thread dispatch is the
	3358	+only path to Granian-class async. On free-threaded 3.14t, the GIL-pool
	3359	+already scales (ADR-084) and parallel dispatch is viable there.
	3360	+
	3361	+Evidence: profiler runs (build 1.3µs / handler 66µs), loop-hop microbench
	3362	+(21/10µs), parallel-experiment regression (9.8k → 5.0k), head-to-head tables.
	3363	+
	3364	+## ADR-087 (addendum) — 2026-08-06 — Async experiments: what thrashes, what the floor is
	3365	+
	3366	+Follow-up to ADR-087's "loop-thread dispatch" plan. Two dispatch strategies
	3367	+were implemented and measured; both regressed async throughput and were
	3368	+reverted. Findings:
	3369	+
	3370	+1. Parallel `spawn_blocking` dispatch (each request attaches a fresh
	3371	+ blocking-pool thread): async 9.8k → 5.0k. Fresh-thread GIL attach + 100
	3372	+ threads competing on the GIL for request-dict builds thrashes.
	3373	+2. N-worker async GIL pool (8 warm pre-attached threads, round-robin):
	3374	+ async 9.8k → 7.2k; profiler showed `avg_handler` jumped 66µs → 284µs.
	3375	+ 8 threads calling `run_coroutine_threadsafe` + building requests contend
	3376	+ on the shared asyncio loop's scheduling locks worse than 1 serializer.
	3377	+3. Key microbenchmarks (the floor): `PyGILState_Ensure`+`Release` on a
	3378	+ warm thread = 0.2µs (attach is NOT the cost); `call_handler` on a
	3379	+ simple async fn = 22.7µs (the loop hop — Granian pays this too);
	3380	+ server async no-sleep total = 108µs/req. So ~85µs is the single GIL
	3381	+ worker's channel + oneshot + wrapper on top of the 22µs floor.
	3382	+
	3383	+Conclusion: on GIL-locked CPython, the dispatch work (build request +
	3384	+schedule coroutine) cannot be parallelized from other threads — the GIL and
	3385	+the shared loop serialize it regardless. The only path to Granian's ~22µs is
	3386	+doing the request-build + handler-call ON the loop thread itself (the
	3387	+loop-thunk design), eliminating the worker hop. That requires moving the
	3388	+Rust-side `Request` construction into a thunk scheduled via
	3389	+`loop.call_soon_threadsafe` — a substantial refactor of `call_python_handler`.
	3390	+Deferred; the callback-driven completion (ADR-086, 5.2k → 8.5-9.8k) remains
	3391	+the shipped win. On free-threaded 3.14t the GIL pool already parallelizes
	3392	+(ADR-084).
	3393	+
	3394	+Evidence: profiler runs (66µs → 284µs), microbenchmarks (0.2µs attach,
	3395	+22.7µs call_handler, 108µs/req), the two reverted experiments, current state
	3396	+(async 6.8-9.8k, sync 115k, all gates green).
	3397	+
	3398	+## ADR-088 — 2026-08-06 — Rust-native SSE streaming (the "server runs on Rust" streaming path)
	3399	+
	3400	+Context: Streaming (SSE) was the last workload still coupled to Python:
	3401	+`TokenStreamResponse` pumped a Python generator via `_pump_stream`
	3402	+(`run_coroutine_threadsafe` on the asyncio loop). Beyond the per-item Python
	3403	+cost, the real-server streaming path was broken (empty body — verified
	3404	+pre-existing in ADR-086). The native fast path (700k), CRUD (180k), static
	3405	+files, and WebSocket framing already run entirely in Rust; streaming was the
	3406	+gap.
	3407	+
	3408	+Decision:
	3409	+- Core: `sse_stream_response(count, interval_ms)` in
	3410	+ `justapi-core/src/server/sse_ws.rs` — spawns a tokio producer task that
	3411	+ emits `data: {"n":i}\n\n` events into an mpsc-backed streaming response.
	3412	+ Zero Python, zero GIL, zero pump. `sse_response()` (built-in `/events`)
	3413	+ now delegates to it.
	3414	+- Python binding: `sse_specs: Vec<Option<(u64, u64)>>` per route (count,
	3415	+ interval_ms), registered via `app.sse_native(path, count, interval_ms)`.
	3416	+ The Rust dispatch checks the spec BEFORE the Python path (mirroring the
	3417	+ CRUD fast path) and serves the Rust stream directly — the Python handler is
	3418	+ a never-invoked placeholder.
	3419	+- Threaded through `make_native_handler`, `make_test_handler`, and the HTTP/3
	3420	+ bridge; works on the real server AND `JustAPITestClient`.
	3421	+
	3422	+Measured: native SSE streamed 1.89 MB (100k events) instantly; the
	3423	+Python-generator SSE path returned 0 bytes in 10s on the real server (the
	3424	+pre-existing bug). This makes SSE the first workload where the native path
	3425	+works and the Python path does not — the "server runs on Rust" streaming
	3426	+proof.
	3427	+
	3428	+Evidence: sse_ws.rs, app.rs `sse_native`, handlers.rs dispatch check,
	3429	+test_sse.py::test_sse_native_rust_stream, integration.rs test update, live
	3430	+curl benchmark.
	3431	+
	3432	+## ADR-089 — 2026-08-06 — `@native_async`: honest contract, not magic
	3433	+
	3434	+Context: The user asked for "write Python, it runs at Rust speed" — a
	3435	+physically impossible framing. Python bytecode is interpreted by CPython; no
	3436	+wrapper changes that. The honest version of the request: a `native_async`
	3437	+API that (a) marks async handlers whose framework operations (DB, SSE,
	3438	+HTTP, serialization) run natively, (b) routes them to the fastest dispatch
	3439	+path, and (c) on free-threaded builds, dispatches them in TRUE parallel
	3440	+(the GIL pool scales to 20 workers).
	3441	+
	3442	+Decision:
	3443	+- `native_async` decorator: sets `__native_async__` on the handler; the
	3444	+ wrapper propagates `_is_native_async` (read by Rust like `_needs_request`).
	3445	+- Rust dispatch: on free-threaded builds (`Py_GIL_DISABLED`), native_async
	3446	+ handlers use `run_python_parallel` (spawn_blocking — safe there, no GIL to
	3447	+ thrash). On GIL-locked builds they use the standard pool + callback-driven
	3448	+ resolution (ADR-086) — the optimal path.
	3449	+- Exported as `from justapi import native_async`.
	3450	+
	3451	+Measured (release wheels, same fixture):
	3452	+- GIL-locked: native_async ≈ plain async (6.9k vs 6.7k, 1ms sleep) — the GIL
	3453	+ is the floor, both use the callback path.
	3454	+- Free-threaded 3.14t: native_async ≈ plain (12.5k vs 12.0k) — both dispatch
	3455	+ in parallel now; the 1ms sleep + loop hop is the floor.
	3456	+- CPU-bound async on free-threaded: equal (~985 RPS) — coroutine bodies run
	3457	+ on the single asyncio loop thread regardless of dispatch; awaits-less
	3458	+ coroutines can't yield, so they serialize on the loop.
	3459	+
	3460	+Honest conclusion: native_async's parallel dispatch removes the GIL-worker
	3461	+hop where it matters (I/O-heavy async on free-threaded). But the loop thread
	3462	+is the real serialization point for coroutine bodies. The "fully Rust speed"
	3463	+for async workloads comes from the Rust-native operation types (sse_native
	3464	+ADR-088, CRUD, native fast path) — Python configures, Rust executes. This is
	3465	+the structural moat: FastAPI/Granian must call Python for everything; we can
	3466	+serve entire workloads from Rust.
	3467	+
	3468	+Evidence: native_async test, the GIL-locked/ft benchmarks, the CPU-bound
	3469	+comparison, gates (174 pytest, 516 workspace, clippy/fmt clean).
	3470	+
	3471	+## ADR-090 — 2026-08-07 — Native-awaitables experiment: every approach measured, none beats the loop
	3472	+
	3473	+Decision: Abandon the "Rust drives the Python coroutine" native-awaitables strategy.
	3474	+The current callback-driven architecture (ADR-086) already beats Granian on async
	3475	+workloads; the experiment below proves no thread-driven coroutine mechanism can
	3476	+scale concurrently, and `future_into_py`/`into_future` bridges are strictly slower
	3477	+than the asyncio loop.
	3478	+
	3479	+Motivation: "Beat Granian on async" was the standing goal. Hypothesis: make the
	3480	+awaits Rust-native (`justapi.sleep()` → tokio timer) so Python coroutines escape the
	3481	+asyncio loop's ~45µs per-await overhead. Four approaches were built and measured
	3482	+against the same workload (coroutine with 1ms-awaits, in-process and under HTTP):
	3483	+
	3484	+\| Approach \| Per-await overhead \| Concurrency verdict \|
	3485	+\|---\|---\|---\|
	3486	+\| `future_into_py` (Rust future → asyncio.Future wrapper) \| 2× SLOWER than asyncio.sleep \| the asyncio.Future wrapper + cross-runtime wakeup costs more than the loop's own timer \|
	3487	+\| `into_future` (coroutine → Rust future) \| tied at best \| requires a running asyncio loop for task-locals/contextvars — does not decouple from asyncio \|
	3488	+\| Hand-rolled driver, GIL attach per await \| ~1125µs/await \| `Python::attach` thread-state setup dominates \|
	3489	+\| Hand-rolled driver, persistent thread \| ~3.4µs/await solo, ~12.6µs/await batch \| 86× WORSE concurrent (thread per coroutine thrashes: 919 awaits/s vs asyncio's 79k) \|
	3490	+\| asyncio loop (status quo) \| ~45µs/await \| the only mechanism that scales (timers batched, GIL released during awaits) \|
	3491	+
	3492	+Why concurrency kills the driver: a persistent driver thread holds the GIL while
	3493	+stepping the coroutine; 100 concurrent coroutines = 100 threads contending for the
	3494	+GIL. asyncio runs all coroutines on ONE thread with the GIL released during every
	3495	+await (and only reacquired for short steppings), which is exactly what enables
	3496	+interleaving. Thread-driven Python coroutines cannot interleave without the loop —
	3497	+this is fundamental to CPython, not a fixable engineering gap.
	3498	+
	3499	+Proof we already win: same HTTP workload (async handler, 1ms sleep), c=16 and c=32,
	3500	+this machine:
	3501	+- c=16: justapi 2610 RPS vs Granian 2.8.1 (1 worker) 2435 RPS
	3502	+- c=32: justapi 3885 RPS vs Granian 2.8.1 (4 workers) 2480 RPS
	3503	+
	3504	+justapi's callback-driven async dispatch (ADR-086) + GIL-pool worker + Rust-side
	3505	+serialization has less per-request overhead than Granian's loop-per-worker ASGI
	3506	+stack. No further async work is needed to "beat Granian" — the win is real and
	3507	+current. Granian's advantage (no dispatch hop for its own async path) is smaller
	3508	+than our overall system overhead advantage.
	3509	+
	3510	+What survives from the experiment: nothing is shipped (all experiment code
	3511	+reverted). The one useful insight retained: a persistent-thread coroutine driver is
	3512	+fast (3.4µs/await) but only for sequential work; it could be revisited ONLY under
	3513	+free-threaded CPython (no GIL → threads genuinely parallel), which is the
	3514	+`native_async`/`run_python_parallel` direction already built. The decision stands:
	3515	+Python async stays on the loop; Rust-native operation types (sse_native ADR-088,
	3516	+native CRUD, native_async ADR-089) are the performance lever, not a coroutine driver.
	3517	+
	3518	+Evidence: in-process microbenchmarks (asyncio.sleep vs native_sleep vs native_run
	3519	+at 0ms and 1ms, 50-200 iterations), HTTP A/B vs Granian 2.8.1 above, all four driver
	3520	+variants built and measured.
	3521	+
	3522	+## ADR-091 — 2026-08-07 — Multiplexing Rust driver: even ONE driver thread loses to asyncio
	3523	+
	3524	+Decision: The "Rust converts Python coroutines to its own async" vision is
	3525	+conclusively measured as impossible to make faster than asyncio. The Python
	3526	+coroutine `send()` stepping is the irreducible floor, and asyncio already runs it
	3527	+at near-optimal cost. Keep Python async on asyncio; win with Rust-native operation
	3528	+types, not with a coroutine driver.
	3529	+
	3530	+Motivation: Follow-up to ADR-090. The objection was: "thread-per-coroutine was
	3531	+the wrong model — use ONE Rust driver thread multiplexing many coroutines, like
	3532	+asyncio's loop but with Rust machinery." That design was built (`native_gather`:
	3533	+one thread, `send(None)` stepping, Rust binary-heap timer wheel, GIL released
	3534	+during sleep) and measured against asyncio on the same workload:
	3535	+
	3536	+\| Workload \| Rust multiplexing driver \| asyncio \| verdict \|
	3537	+\|---\|---\|---\|---\|
	3538	+\| 100 coroutines × 10 awaits of 1ms \| 37 ms \| 19.6 ms \| asyncio 1.9× faster \|
	3539	+\| 100 coroutines × 1000 zero-awaits (pure stepping) \| 16.12 µs/await \| 0.56 µs/await \| asyncio 29× faster \|
	3540	+\| absolute floor: bare `await noop()` \| — \| 0.039 µs/await \| the Python C-API floor \|
	3541	+
	3542	+Why the driver loses: the per-await cost of the driver is the `send(None)`
	3543	+call + `getattr("ms")` + `extract` + heap push/pop, ~16µs. asyncio's task
	3544	+machinery steps the coroutine at 0.56µs and a bare await at 0.039µs — asyncio IS
	3545	+the C-API's native stepping; any Rust wrapper around `send()` adds overhead, never
	3546	+subtracts it. The earlier 3.4µs "raw send loop" (ADR-090) was already 6-8× slower
	3547	+than asyncio's own stepping. There is no version of "drive Python coroutines from
	3548	+Rust" that is faster, because the Python interpreter's stepping cost is paid
	3549	+either way and asyncio pays it at the minimum possible rate.
	3550	+
	3551	+The correct reading of the vision: "pass Python code, Rust handles the async"
	3552	+IS real and already shipped — but it means Rust-native operation types where
	3553	+Python declares the work and Rust executes it with no Python stepping at all:
	3554	+- `sse_native` (ADR-088): Rust streams events, zero Python per event
	3555	+- native CRUD: Rust validates + queries, zero Python
	3556	+- `@native_async` (ADR-089): free-threaded parallel dispatch
	3557	+- Python async handlers stay on asyncio (where they're optimal) for the awaits
	3558	+ that must be Python (user logic between awaits)
	3559	+
	3560	+For a user's async handler whose awaits are all framework ops (DB, sleep, SSE),
	3561	+the performance path is to make THOSE ops native (as above), not to re-drive the
	3562	+coroutine.
	3563	+
	3564	+Evidence: `native_gather`/`native_sleep`/`NativeTimer` experiment build +
	3565	+microbenchmarks above (50-1000 iterations). Experiment code reverted after
	3566	+recording.
3287	3567