p4: freeze AI/inference work until release 2.0.8 (ADR-067)
swadhinbiswas committed 3 months ago
6e791da
1 parent aacbf03
commit 6e791da
2 files changed
+31 -3
Showing 2 changed files+31-3
▼DECISIONS.md
+28-0
| @@ -2508,3 +2508,31 @@ wasm + 1 tracer). `cargo clippy --workspace --tests --features | ||
2508 | 2508 | "justapi-core/db,justapi-core/tls" -- -D warnings` clean; `cargo fmt --check` |
2509 | 2509 | clean. No new dependency, no public API change (internal function only). |
2510 | 2510 | |
2511 | +## ADR-067 — 2026-07-18 — Freeze all AI/inference work until release 2.0.8 | |
2512 | + | |
2513 | +**Context:** We spent effort on the Phase 52 GPU Benchmark Gate (real-model | |
2514 | +inference via `justapi-inference`/`justapi-bench`, CUDA PTX mismatch, Needle | |
2515 | +26M encoder-decoder weights downloaded to `~/needle-bucket`). The user has | |
2516 | +decided to **skip all AI / inference / LLM work for now** and defer it to the | |
2517 | +next release, **2.0.8**. This includes the CUDA driver upgrade (NVIDIA 610.43.03 | |
2518 | +was available and matched the running kernel `7.1.3-2-cachyos` but was | |
2519 | +deliberately NOT applied), the real-GPU benchmark run, and any Needle | |
2520 | +integration. | |
2521 | + | |
2522 | +**Decision:** | |
2523 | +- No AI/inference features will be touched until release **2.0.8**. | |
2524 | +- The CUDA driver stays at 580.173.02 (CUDA 13.0 runtime) and the toolkit at | |
2525 | + 13.3.1 — the PTX mismatch is intentionally left unresolved for now. | |
2526 | +- The Phase 52 GPU Benchmark Gate remains **🔴 blocked**; its exit criteria | |
2527 | + (real weights on real GPU, tokens/sec + TTFT/ITL) are deferred to 2.0.8. | |
2528 | +- The Needle weights in `~/needle-bucket` are kept as reference material only; | |
2529 | + no encoder-decoder loader will be added to `justapi-inference` yet (it | |
2530 | + currently supports Llama-family GGUF/safetensors only). | |
2531 | +- `justapi-gpu-bench` MockModel (CPU) run is recorded as non-gate plumbing | |
2532 | + validation only (see PLAN.md Phase 52 update, commit `a6c8cac`). | |
2533 | + | |
2534 | +**Evidence:** User directive 2026-07-18 ("skip this AI stuff, work it on next | |
2535 | +release 2.0.8, until then don't touch AI stuff"). PLAN.md Phase 52 updated to | |
2536 | +reflect the opt-out and committed (`a6c8cac`). No code change in this ADR. | |
2537 | + | |
2538 | + | |
2511 | 2539 | |
▼PLAN.md
+3-3
| @@ -6,8 +6,8 @@ | ||
6 | 6 | |
7 | 7 | ## Current status |
8 | 8 | |
9 | -- **Active phase:** Phase 52 (GPU Benchmark Gate) — 🟡 ready to run | |
10 | -- **Status:** Phase 51 complete. Phase 52 is the first real-GPU validation: load a real model (GGUF) on CUDA, measure tokens/sec/TTFT/ITL for both naive and scheduler-backed generation paths, and compare against vLLM. The benchmark binary (`justapi-gpu-bench`) is written and compiles (with MockModel fallback when no CUDA). **Blocker resolved:** CUDA toolkit IS installed — `nvcc` (CUDA 13.3) at `/opt/cuda/bin` and an NVIDIA GeForce RTX 3060 Ti (8192 MiB) are present. The gate is now runnable; it still needs an actual `cargo run --features "cuda,real"` pass with real weights (candle w/ CUDA build unverified). | |
9 | +- **Active phase:** Phase 52 (GPU Benchmark Gate) — 🔴 frozen (AI work deferred to release 2.0.8) | |
10 | +- **Status:** Phase 51 complete. **AI / inference / LLM work is FROZEN until release 2.0.8** (user directive 2026-07-18, ADR-067). Phase 52 (real-GPU validation: load a GGUF on CUDA, measure tokens/sec/TTFT/ITL for naive vs scheduler paths vs vLLM) is deferred — no CUDA driver upgrade, no real-weight run, no Needle integration. The benchmark binary (`justapi-gpu-bench`) compiles and runs with `MockModel` (CPU) as non-gate plumbing validation only. CUDA toolkit 13.3.1 + driver 580.173.02 (CUDA 13.0) remain installed; the PTX mismatch is intentionally left unresolved. Focus until 2.0.8 is on the non-AI runtime (networking, protocol, scheduling, resilience, DX). | |
11 | 11 | - **Hot-path optimization (2026-07-13) — ✅ complete, not a numbered phase.** Removed the per-request Python↔Rust boundary (Rust-side `orjson` response serialization, `Response` serialized directly via a marker attribute, `needs_request` skips `Request` construction, trace-context gated behind `JUSTAPI_ENABLE_TRACE`). Result: justapi now *beats* Robyn on raw throughput (hello 60.3k vs 39.1k RPS, echo 47.4k vs 36.9k, validated 40.1k vs 32.9k on the same fixture — see BENCHMARKS.md). Gates green: `cargo test --workspace`, `cargo clippy --workspace --tests -- -D warnings`, `cargo fmt --check`, pytest 107 passed/1 skipped. Fixed a latent edge-case bug (0-param handler + middleware now forces `needs_request=True`). See ADR-047. |
12 | 12 | - **Schema-backed native Rust fast path (2026-07-14) — ✅ complete, further optimized.** Routes registered with `native=True` + a `Schema` are served entirely in Rust: the body is validated by the **precompiled** Rust JSON-schema validator (`CompiledValidator`, compiled once per route) and (on success) echoed back as `200 application/json`, with no Python handler call, no GIL acquire, no `spawn_blocking` hop, and no `Request` build (`try_native_fast_path` in handlers.rs). `native=True` without a schema safely falls back to the Python path. Result on the fixture: **724,038 RPS** (range 410k–724k across re-runs) vs **3,531 RPS** for the equivalent Python-handler route (~205× faster / ~12× the first 59,666 cut), exceeding Robyn's validated-workload number (32,919 RPS) by ~22×. See ADR-048 + ADR-049 + BENCHMARKS.md native fast-path section. Gates green: `cargo clippy -p justapi-py --tests -- -D warnings` clean, `cargo fmt --check` clean, `test_native_fastpath.py` passes. |
13 | 13 | - **✅ Non-native dispatch deadlock at high concurrency — RESOLVED (ADR-049).** The dedicated GIL thread-pool (`gil_pool.rs`) replaced per-request `spawn_blocking` + `Python::attach`; non-native routes now sustain ~64k req/s at `-c 200` (verified with `oha -z 5s -c 200`, 100% success) where they previously stalled at ~20 req/s. Native routes were always immune. No remaining deadlock on the request path. |
| @@ -75,7 +75,7 @@ cloud-native readiness. | ||
75 | 75 | | 49 | Tree-based Speculative Decoding | ✅ complete | Medusa/EAGLE-style tree verification wired into serving path. Acceptance rate up to 3× higher than draft-target. | |
76 | 76 | | 50 | RadixAttention Prefix Caching | ✅ complete | `RadixPrefixCache` wired into `Scheduler`: O(1) prefix lookup on admission, LRU eviction, finished-seq block promotion. | |
77 | 77 | | 51 | Scheduler Serving Integration | ✅ complete | `SchedulerEngine` bridges Engine+Scheduler; prefix cache metrics in `/metrics`; sampling params plumbed; real throughput benchmark. 151 inference tests. | |
78 | -| **52** | **GPU Benchmark Gate** | **🟡 ready to run** | **Run SchedulerEngine vs naive vs vLLM on real GPU with real weights. Measure tokens/sec, TTFT, ITL. CUDA present (RTX 3060 Ti, CUDA 13.3) — blocker resolved.** | | |
78 | +| **52** | **GPU Benchmark Gate** | **🔴 frozen — AI deferred to 2.0.8** | **Run SchedulerEngine vs naive vs vLLM on real GPU with real weights. Measure tokens/sec, TTFT, ITL. DEFERRED: no CUDA driver upgrade, no real-weight run (ADR-067). MockModel CPU run only as plumbing check.** | | |
79 | 79 | |
80 | 80 | --- |
81 | 81 | |
82 | 82 | |