🧪 p4: multi-loop dispatch experiment — A/B-tested, does not help, default 1 loop (ADR-092)
swadhinbiswas committed 2 months ago
bfff404
1 parent 28d6fda
commit bfff404
2 files changed
+63 -9
Showing 2 changed files+63-9
▼DECISIONS.md
+38-0
| @@ -3564,3 +3564,41 @@ coroutine. | ||
3564 | 3564 | **Evidence:** `native_gather`/`native_sleep`/`NativeTimer` experiment build + |
3565 | 3565 | microbenchmarks above (50-1000 iterations). Experiment code reverted after |
3566 | 3566 | recording. |
3567 | + | |
3568 | +## ADR-092 — 2026-08-07 — Multi-loop async dispatch: A/B-tested, does not help, reverted | |
3569 | + | |
3570 | +**Decision:** Keep the single-asyncio-loop + callback-driven dispatch (ADR-086). | |
3571 | +Multi-loop round-robin was built, A/B-tested on identical fixtures, and does not | |
3572 | +help — it is neutral-to-slightly-worse. Reverted to 1 loop by default (the | |
3573 | +`JUSTAPI_ASYNC_LOOPS` env override stays for experimentation). | |
3574 | + | |
3575 | +**Motivation:** Phase 1 of the "beat Granian on light async" plan was multi-loop | |
3576 | +dispatch: N asyncio loops with round-robin coroutine assignment, to reduce | |
3577 | +`call_soon_threadsafe` lock contention and spread timer wheels. Implemented in | |
3578 | +`_native_helper.py` (N loop threads, round-robin `_get_loop()`, env override). | |
3579 | + | |
3580 | +**A/B results (identical fixture, same wheel, loop count the only variable):** | |
3581 | + | |
3582 | +| Workload | 1 loop | 12 loops | | |
3583 | +|---|---|---| | |
3584 | +| async + 1ms sleep, c=16 | 3,752 RPS | 3,135 RPS | | |
3585 | +| async + 1ms sleep, c=32 | 4,403 RPS | 3,651 RPS | | |
3586 | +| async + 1ms sleep, c=64 | 4,443 RPS | 3,257 RPS | | |
3587 | +| 20× 1ms sleep per req, c=16 | 543 RPS | 593 RPS | | |
3588 | +| 20× 1ms sleep per req, c=32 | 971 RPS | 928 RPS | | |
3589 | + | |
3590 | +**Why it fails:** the GIL worker is the single dispatch point — only one thread | |
3591 | +ever calls `run_coroutine_threadsafe`, so the loop's lock is never contended | |
3592 | +(no multi-thread `call_soon` traffic). The per-request cost is the loop's own | |
3593 | +stepping + the thread hop, both independent of loop count. Extra loops add | |
3594 | +GIL contention (more stepping threads). Consistent with ADR-087's finding that | |
3595 | +an 8-worker async pool regressed. | |
3596 | + | |
3597 | +**Conclusion for the async story:** the current single-loop callback-driven | |
3598 | +dispatch is already the right architecture; light-async (~4.4k RPS c=32 on this | |
3599 | +fixture, ~44k theoretical ceiling for loop-only dispatch) is capped by the | |
3600 | +thread-hop, not by the loop. The path to beat Granian on async stays as | |
3601 | +recorded: Rust-native operation types (sse_native, native CRUD, native_async), | |
3602 | +never a coroutine driver (ADR-090/091) nor multi-loop (this ADR). | |
3603 | + | |
3604 | +**Evidence:** A/B runs above, identical wheel, JUSTAPI_ASYNC_LOOPS=1 vs default. | |
3567 | 3605 | |
▼crates/justapi-py/python/justapi/_native_helper.py
+25-9
| @@ -3,6 +3,7 @@ import inspect | ||
3 | 3 | import json |
4 | 4 | import contextvars |
5 | 5 | import concurrent.futures |
6 | +import os | |
6 | 7 | import threading |
7 | 8 | import traceback |
8 | 9 | |
| @@ -42,26 +43,41 @@ def get_trace_context(): | ||
42 | 43 | "span_id": _span_id_var.get(), |
43 | 44 | } |
44 | 45 | |
45 | -_loop = None | |
46 | -_loop_thread = None | |
46 | +_loops = [] | |
47 | +_loop_threads = [] | |
47 | 48 | _loop_lock = None |
49 | +_loop_rr = 0 | |
48 | 50 | |
49 | 51 | def _start_loop(loop): |
50 | 52 | asyncio.set_event_loop(loop) |
51 | 53 | loop.run_forever() |
52 | 54 | |
55 | +def _loop_count(): | |
56 | + """Number of asyncio loops (ADR-092). A/B-tested: multi-loop does NOT | |
57 | + help — the GIL worker is the single dispatch point, so the loop's lock | |
58 | + is never contended; extra loops only add GIL contention. Default 1. | |
59 | + JUSTAPI_ASYNC_LOOPS overrides for experimentation.""" | |
60 | + if "JUSTAPI_ASYNC_LOOPS" in os.environ: | |
61 | + return max(int(os.environ["JUSTAPI_ASYNC_LOOPS"]), 1) | |
62 | + return 1 | |
63 | + | |
53 | 64 | def _get_loop(): |
54 | - global _loop, _loop_thread, _loop_lock | |
65 | + global _loops, _loop_threads, _loop_lock, _loop_rr | |
55 | 66 | if _loop_lock is None: |
56 | - # This itself is a slight race but much safer than creating loops | |
57 | 67 | _loop_lock = threading.Lock() |
58 | 68 | |
59 | 69 | with _loop_lock: |
60 | - if _loop is None: | |
61 | - _loop = asyncio.new_event_loop() | |
62 | - _loop_thread = threading.Thread(target=_start_loop, args=(_loop,), daemon=True) | |
63 | - _loop_thread.start() | |
64 | - return _loop | |
70 | + if not _loops: | |
71 | + count = _loop_count() | |
72 | + for _ in range(count): | |
73 | + loop = asyncio.new_event_loop() | |
74 | + thread = threading.Thread(target=_start_loop, args=(loop,), daemon=True) | |
75 | + thread.start() | |
76 | + _loops.append(loop) | |
77 | + _loop_threads.append(thread) | |
78 | + loop = _loops[_loop_rr % len(_loops)] | |
79 | + _loop_rr += 1 | |
80 | + return loop | |
65 | 81 | |
66 | 82 | def wrap_result(result): |
67 | 83 | # Streaming responses are returned unwrapped so the Rust side can pump the |
68 | 84 | |