JustAPI Architecture

This document provides a comprehensive overview of the design, internal structure, and technical invariants of the JustAPI runtime.

High-Level System Design

JustAPI splits responsibility between a high-performance Rust core and Python application logic:

• Rust Core (justapi-core): Handles I/O, networking (HTTP/1.1, HTTP/2, WebSockets, gRPC, SSE), TLS termination, path routing, the middleware chain (auth, CORS, rate limiting, compression), schema validation, and serialization.
• Python Layer (justapi-py): Provides the decorator-based framework API (matching FastAPI's developer experience), registers routes, runs dependency injection (Depends()), and executes the user's business logic.
• Zero-Copy FFI Boundary: PyO3's buffer protocol is used to pass request parameters, headers, and bodies directly from Rust to Python as read-only memory views, eliminating serialization/deserialization overhead.

       [ Client Request ]
               │
               ▼
┌──────────────────────────────┐
│  Kernel (epoll / io_uring)   │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Connection Manager (Tokio)  │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│       TLS (rustls)           │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│      HTTP Parser (hyper)     │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│      Router (matchit)        │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│      Middleware Chain        │ (Auth, CORS, Rate Limit, Compression)
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Zero-Copy FFI Boundary      │ (PyO3 buffer protocol / Arrow / DLPack)
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│    Python Handler Coroutine  │ (Jinja2, Pydantic validation, DB access)
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│   JSON Serializer (serde)    │
└──────────────┬───────────────┘
               │
               ▼
       [ Response Write ]

Crate Layout & Dependency Graph

The workspace is organized into modular crates:

justapi-cli  ──→  justapi-core
justapi-py   ──→  justapi-core
justapi-bench ─→  justapi-core

1. justapi-core:
   • The engine room. Exposes the custom HTTP server built on hyper and tokio.
   • Implements the radix-tree path router (matchit), middlewares, JSON Schema validation (jsonschema crate), static file serving, and the database connection pooling/migration runner (sqlx).
2. justapi-py:
   • The PyO3 FFI bridge. Wraps justapi-core server handles and request views.
   • Includes the pure-Python framework code (python/justapi/) providing the JustAPIApp decorator interface, custom dependency injection container, responses, testing client, and templates.
3. justapi-cli:
   • Command-line utility (justapi binary). Runs development servers (with watch/hot-reload), compiles OpenAPI specs, profiles handlers, and executes migrations.
4. justapi-bench:
   • Internal benchmark harness. Used to prevent performance regressions by verifying throughput and latency gates.
5. justapi-inference:
   • Optional ML serving engine built on Candle (CUDA/CPU). Implements PagedAttention KV-caching, continuous batching, and speculative decoding.

Concurrency & GIL Model

JustAPI utilizes a GIL-free async worker architecture (ADR-021):

• Tokio Worker Integration: Incoming connections are accepted concurrently by Tokio worker threads.
• GIL-free attach: When a request requires Python execution, the Tokio worker thread uses PyO3's GIL-free Python::try_attach() to safely lock the GIL.
• Execution Caching: Python handler helper modules are compiled once at startup and cached in a global OnceLock<Py<PyAny>>. Invoking Python handlers does not require spawning separate OS threads or managing an external asyncio loop in Rust.
• GIL Release during I/O: The GIL is immediately released (via PyO3's py.detach()) when Python handlers complete or yield for async Rust I/O, maximizing CPU cores and avoiding interpreter deadlocks.

Key Subsystems

Radix Router

Uses the radix-trie based matchit crate. Paths are resolved in O(1) time relative to the number of registered routes. Parameters are parsed into typed path variables during the routing step.

Request Validation

Request bodies are validated against their JSON Schema entirely in Rust using the jsonschema crate before Python is invoked. Invalid requests receive an RFC 9457 structured error response and are rejected at the FFI boundary, saving CPU cycles.

Database ORM

Built on sqlx connection pools. The migrator discovers and executes plain SQL migration files, tracking applied versions in a tracking table. Transactions are automatically managed by the middleware chain: initialized on incoming requests, committed on 2xx statuses, and rolled back on error.

Observability

• Metrics: Prometheus-compatible endpoint serving bucket-based latency histograms (p50, p95, p99, p999 percentiles calculated client-side) and error counts.
• OTel Tracing: OpenTelemetry OTLP tracing hooks propagate context across the PyO3 boundary using thread-local variables.

LLM Serving Engine (justapi-inference)

To support fast AI deployment, JustAPI incorporates a Rust-native model server:

• KvBlockPool: Implements paged KV-cache block allocation to prevent GPU memory fragmentation.
• PrefixCache: A SGLang-style radix prefix cache that reuses prefill context across concurrent requests.
• PdScheduler: Implements disaggregated prefill/decode scheduling. Prefill operations run on separate resource pools from decode steps, avoiding decode starvation.
• Speculative Decoding: Compares generated draft tokens with a target model, verifying candidate tokens in parallel.