swadhin / justapi public

p4f-i: add Agent System, Streaming Output, Performance Tuning, Resilience advanced guides

swadhinbiswas committed 3 months ago
578b09d
1 parent a0a4f2a
commit 578b09d
4 files changed
+493 -0
Showing 4 changed files+493-0
docs_site/src/content/docs/advanced/agent-system.mddocs_site/src/content/docs/advanced/performance-tuning.mddocs_site/src/content/docs/advanced/resilience-patterns.mddocs_site/src/content/docs/advanced/streaming-output.md
▼docs_site/src/content/docs/advanced/agent-system.md
+123-0
@@ -0,0 +1,123 @@
1
+---
2
+title: Agent System (MCP Tools & Sessions)
3
+description: Build AI agents with native MCP tools, durable session state, and validated streaming.
4
+---
5
+
6
+JustAPI includes a first-class agent-native system that differentiates it from traditional web frameworks. It provides **MCP tools**, **durable sessions**, and **validated streaming** — all owned by Rust.
7
+
8
+## MCP Tools (`@app.tool`)
9
+
10
+Register Python functions as MCP-compatible tools. Schemas are inferred from type hints:
11
+
12
+```python
13
+from justapi import JustAPIApp
14
+
15
+app = JustAPIApp()
16
+
17
+
18
+@app.tool
19
+def add(a: int, b: int) -> int:
20
+ """Add two integers."""
21
+ return a + b
22
+
23
+
24
+@app.tool
25
+def search_docs(query: str, top_k: int = 3) -> list:
26
+ """Search documentation."""
27
+ return [{"text": "JustAPI runs routing in Rust", "score": 0.95}]
28
+```
29
+
30
+### Tool Discovery & Invocation
31
+
32
+Tools are automatically exposed via system routes:
33
+
34
+```bash
35
+# List all registered tools
36
+GET /_system/tools
37
+
38
+# Call a tool
39
+POST /_system/tools/call
40
+Content-Type: application/json
41
+
42
+{"name": "add", "arguments": {"a": 5, "b": 7}}
43
+```
44
+
45
+### MCP Stdio Server
46
+
47
+```python
48
+app.run_mcp_stdio() # Run an MCP stdio server for local AI tool use
49
+```
50
+
51
+## Durable Sessions (`app.enable_sessions()`)
52
+
53
+Rust-backed per-client state storage:
54
+
55
+```python
56
+app.enable_sessions()
57
+
58
+
59
+@app.get("/agent/state")
60
+def agent_state(request, session: Session):
61
+ prior = session.get().get("runs", 0)
62
+ session.update(runs=prior + 1)
63
+ return {"session_id": session.id, "runs": prior + 1}
64
+```
65
+
66
+Sessions are identified by the `justapi_session` cookie or `?session=` query parameter.
67
+
68
+## Validated Streaming (`@app.stream_json`)
69
+
70
+Stream generator output as NDJSON or JSON array, validating every item in Rust:
71
+
72
+```python
73
+STEP_SCHEMA = {
74
+ "type": "object",
75
+ "properties": {
76
+ "step": {"type": "string"},
77
+ "result": {},
78
+ },
79
+ "required": ["step", "result"],
80
+}
81
+
82
+
83
+@app.stream_json("/agent/run", schema=STEP_SCHEMA, mode="ndjson")
84
+def agent_run(request, session: Session):
85
+ yield {"step": "think", "result": "Analyzing..."}
86
+ yield {"step": "act", "result": "Performing task..."}
87
+ yield {"step": "observe", "result": "Task complete"}
88
+```
89
+
90
+Invalid items abort the stream — clients never see partial or invalid data.
91
+
92
+## Complete Example
93
+
94
+```python
95
+from justapi import JustAPIApp, Session
96
+
97
+app = JustAPIApp()
98
+app.enable_sessions()
99
+app.enable_system_routes()
100
+
101
+
102
+@app.tool
103
+def add(a: int, b: int) -> int:
104
+ return a + b
105
+
106
+
107
+@app.stream_json("/run", schema=STEP_SCHEMA, mode="ndjson")
108
+def run(request, session: Session):
109
+ result = add(2, 3)
110
+ prior = session.get().get("runs", 0)
111
+ session.update(runs=prior + 1)
112
+ yield {"step": "compute", "result": result}
113
+
114
+
115
+if __name__ == "__main__":
116
+ app.run("127.0.0.1:8000")
117
+```
118
+
119
+## See Also
120
+
121
+- [Session API](/api-reference/session/) — Session reference
122
+- [Streaming Output](/advanced/streaming-output/) — Validated streaming deep dive
123
+- [Dependency Injection API](/api-reference/dependency-injection/) — Session in DI
0
124
▼docs_site/src/content/docs/advanced/performance-tuning.md
+132-0
@@ -0,0 +1,132 @@
1
+---
2
+title: Performance Tuning
3
+description: Optimize your JustAPI application for maximum throughput and minimum latency.
4
+---
5
+
6
+## Worker Configuration
7
+
8
+### Multi-Worker Mode
9
+
10
+```bash
11
+justapi serve --host 0.0.0.0 --port 8080 --workers 4
12
+```
13
+
14
+Each worker is a separate OS process. Workers share no state — use Redis/Postgres for shared data.
15
+
16
+### Auto-Scaling Workers
17
+
18
+```bash
19
+justapi serve --scale --min-workers 2 --max-workers 8 \
20
+ --scale-high 1000 --scale-cooldown 30
21
+```
22
+
23
+| Flag | Default | Description |
24
+|---|---|---|
25
+| `--workers` | CPU count | Fixed number of workers |
26
+| `--scale` | off | Enable load-based auto-scaling |
27
+| `--min-workers` | 2 | Minimum workers |
28
+| `--max-workers` | CPU count | Maximum workers |
29
+| `--scale-low` | 100 | RPS threshold to scale down |
30
+| `--scale-high` | 1000 | RPS threshold to scale up |
31
+| `--scale-cooldown` | 30 | Seconds between scaling events |
32
+
33
+## Connection Pool Tuning
34
+
35
+```python
36
+app.set_database(
37
+ "postgres://localhost/mydb",
38
+ max_connections=20, # Pool size
39
+ request_acquire_timeout=1.0, # Timeout for pool saturation
40
+)
41
+```
42
+
43
+- Match pool size to expected concurrency
44
+- Set `request_acquire_timeout` low (1-3s) for fast failure under load
45
+- Use connection pooling behind the app for higher concurrency
46
+
47
+## Body Size Limits
48
+
49
+```python
50
+app.run("0.0.0.0:8000", max_body_size=1024 * 1024) # 1 MiB
51
+```
52
+
53
+Tight limits prevent memory exhaustion under load. Default is 50 MiB.
54
+
55
+## Request Timeouts
56
+
57
+Set timeouts to bound worst-case latency:
58
+
59
+```bash
60
+justapi serve --timeout 30 # 30 second request timeout
61
+```
62
+
63
+Requests exceeding the timeout return **504 Gateway Timeout**.
64
+
65
+## TLS Overhead
66
+
67
+TLS adds approximately **10.7%** overhead vs. plain HTTP. For maximum performance:
68
+
69
+1. Terminate TLS at the edge proxy (nginx, Caddy, cloud LB)
70
+2. Run JustAPI on a private loopback or Unix socket
71
+3. Pass `X-Forwarded-For` and `X-Forwarded-Proto` from the proxy
72
+
73
+## Compression Trade-offs
74
+
75
+| Algorithm | Compression Ratio | Speed |
76
+|---|---|---|
77
+| None | 1.0x | Fastest |
78
+| gzip (level 1) | ~3x | Fast |
79
+| gzip (level 9) | ~5x | Slow |
80
+| brotli | ~4x | Medium |
81
+| zstd | ~5x | Fast |
82
+
83
+Enable compression only when bandwidth is the bottleneck:
84
+
85
+```rust
86
+// In Cargo.toml features
87
+justapi-core = { features = ["compression"] }
88
+```
89
+
90
+## Route-Lookup Cache
91
+
92
+JustAPI automatically caches route lookups for high-traffic paths. No configuration needed — the cache is always active.
93
+
94
+## Native Fast Path
95
+
96
+Maximum throughput comes from the native fast path:
97
+
98
+```python
99
+@app.post("/fast", body_schema=Schema, native=True)
100
+def fast_handler(request):
101
+ return {"status": "ok"} # 724k+ RPS
102
+```
103
+
104
+See [Native Fast Path](/advanced/native-fast-path/) for details.
105
+
106
+## K8s Overhead
107
+
108
+When running in Kubernetes, expect approximately **8.4%** throughput degradation due to networking overhead. Mitigate by:
109
+
110
+1. Using host networking mode
111
+2. Setting resource requests/limits appropriately
112
+3. Tuning kube-proxy to use `iptables` mode
113
+
114
+## Benchmark Methodology
115
+
116
+```bash
117
+# Warm up: 5 seconds
118
+oha -z 5s http://localhost:8080/hello
119
+
120
+# Benchmark: 30 seconds, 100 concurrent connections
121
+oha -z 30s -c 100 http://localhost:8080/hello
122
+
123
+# Results: p50, p95, p99 latency, RPS, peak RSS
124
+```
125
+
126
+Run minimum 3 runs per workload to establish distribution.
127
+
128
+## See Also
129
+
130
+- [Native Fast Path](/advanced/native-fast-path/) — 724k+ RPS execution
131
+- [Production Checklist](/deployment/production-checklist/) — Production deployment settings
132
+- [Benchmarking Guide](/contributing/benchmarking-guide/) — Adding benchmarks
0
133
▼docs_site/src/content/docs/advanced/resilience-patterns.md
+123-0
@@ -0,0 +1,123 @@
1
+---
2
+title: Resilience Patterns
3
+description: Protect your application with circuit breakers, rate limiting, bulkheads, and graceful degradation.
4
+---
5
+
6
+JustAPI incorporates enterprise resilience primitives directly in the Rust engine.
7
+
8
+## Circuit Breakers
9
+
10
+Automatically stop calling a failing service when errors exceed a threshold:
11
+
12
+```python
13
+app.enable_circuit_breaker(
14
+ failure_threshold=5, # Open after 5 consecutive failures
15
+ reset_timeout_ms=30000, # Try again after 30 seconds
16
+)
17
+```
18
+
19
+States:
20
+- **Closed** — Normal operation, requests pass through
21
+- **Open** — Requests fail fast with 503
22
+- **Half-Open** — Probe request allowed; success → Closed, failure → Open
23
+
24
+## Request Coalescing
25
+
26
+Deduplicate concurrent identical requests so only one hits the backend:
27
+
28
+```python
29
+app.enable_request_coalescing(
30
+ headers=["accept"], # Consider these headers for dedup keys
31
+)
32
+```
33
+
34
+When 100 identical requests arrive simultaneously:
35
+- Only 1 request reaches the handler
36
+- The response is shared across all 100 callers
37
+- Saves backend resources on cacheable/read-only endpoints
38
+
39
+## Rate Limiting
40
+
41
+### Global Rate Limiter
42
+
43
+```python
44
+app.enable_rate_limiter(max_requests=1000, window_seconds=60)
45
+```
46
+
47
+### Per-IP Rate Limiter
48
+
49
+```python
50
+app.enable_ip_rate_limiter(max_requests=100, window_seconds=60)
51
+```
52
+
53
+Exceeding limits returns **429 Too Many Requests** with `Retry-After` header.
54
+
55
+## Bulkhead Pattern
56
+
57
+Limit concurrent requests to prevent resource exhaustion:
58
+
59
+```python
60
+app.enable_bulkhead(max_concurrent=100)
61
+```
62
+
63
+When the limit is exceeded, requests receive **503 Service Unavailable**.
64
+
65
+## Timeout Middleware
66
+
67
+Cancel requests that exceed the time limit:
68
+
69
+```python
70
+app.enable_timeout(seconds=30)
71
+```
72
+
73
+Returns **504 Gateway Timeout** when exceeded.
74
+
75
+## Graceful Degradation
76
+
77
+```python
78
+app.enable_fallback_middleware(
79
+ fallback_response={"status": "degraded", "message": "Please try again later"},
80
+)
81
+```
82
+
83
+## Graceful Shutdown
84
+
85
+JustAPI handles SIGINT/SIGTERM automatically:
86
+
87
+1. Stop accepting new connections
88
+2. Drain in-flight requests (with configurable timeout)
89
+3. Call plugin `on_shutdown()` hooks
90
+4. Close database connections
91
+5. Exit cleanly
92
+
93
+## Panic Model
94
+
95
+In production, Rust panics **abort the process** (not unwind). The supervisor (Docker, systemd, K8s) restarts the process immediately:
96
+
97
+```toml
98
+# Cargo.toml
99
+[profile.release]
100
+panic = "abort"
101
+```
102
+
103
+This avoids undefined behavior across the PyO3 FFI boundary during panics.
104
+
105
+## Configuration Summary
106
+
107
+```python
108
+app = JustAPIApp()
109
+
110
+# Resilience
111
+app.enable_circuit_breaker(failure_threshold=5, reset_timeout_ms=30000)
112
+app.enable_request_coalescing(headers=["accept"])
113
+app.enable_rate_limiter(max_requests=1000, window_seconds=60)
114
+app.enable_ip_rate_limiter(max_requests=100, window_seconds=60)
115
+app.enable_bulkhead(max_concurrent=100)
116
+app.enable_timeout(seconds=30)
117
+```
118
+
119
+## See Also
120
+
121
+- [Production Checklist](/deployment/production-checklist/) — Production deployment
122
+- [Security](/security/policy/) — Security policy and OWASP compliance
123
+- [Observability](/observability/metrics-monitoring/) — Monitoring resilience signals
0
124
▼docs_site/src/content/docs/advanced/streaming-output.md
+115-0
@@ -0,0 +1,115 @@
1
+---
2
+title: Streaming Output
3
+description: Stream responses with Rust-side validation using @app.stream_json and StreamingResponse.
4
+---
5
+
6
+JustAPI supports two streaming approaches: the validated `@app.stream_json` and the flexible `StreamingResponse`.
7
+
8
+## `@app.stream_json` (Validated Streaming)
9
+
10
+Every yielded item is validated against a JSON Schema **in Rust** before being written to the socket. Invalid items abort the stream.
11
+
12
+### NDJSON Mode
13
+
14
+```python
15
+from justapi import JustAPIApp
16
+
17
+app = JustAPIApp()
18
+
19
+SCHEMA = {
20
+ "type": "object",
21
+ "properties": {
22
+ "step": {"type": "string"},
23
+ "value": {"type": "number"},
24
+ },
25
+ "required": ["step", "value"],
26
+}
27
+
28
+
29
+@app.stream_json("/stream", schema=SCHEMA, mode="ndjson")
30
+def stream(request):
31
+ for i in range(10):
32
+ yield {"step": f"item-{i}", "value": i}
33
+```
34
+
35
+Output:
36
+
37
+```
38
+{"step":"item-0","value":0}
39
+{"step":"item-1","value":1}
40
+...
41
+```
42
+
43
+### Array Mode
44
+
45
+```python
46
+@app.stream_json("/array", schema=SCHEMA, mode="array")
47
+def stream_array(request):
48
+ for i in range(10):
49
+ yield {"step": f"item-{i}", "value": i}
50
+```
51
+
52
+Output:
53
+
54
+```json
55
+[{"step":"item-0","value":0},{"step":"item-1","value":1},...]
56
+```
57
+
58
+### Validation
59
+
60
+If an item doesn't match the schema, the stream is **aborted immediately** and the client receives an error. This guarantees clients never see partial or invalid data.
61
+
62
+## `StreamingResponse` (Flexible Streaming)
63
+
64
+For custom streaming needs:
65
+
66
+```python
67
+import asyncio
68
+from justapi.responses import StreamingResponse
69
+
70
+
71
+@app.get("/events")
72
+async def events(request):
73
+ async def generate():
74
+ for i in range(10):
75
+ yield f"data: Event #{i}\n\n"
76
+ await asyncio.sleep(1)
77
+
78
+ return StreamingResponse(generate(), media_type="text/event-stream")
79
+```
80
+
81
+## SSE (Server-Sent Events)
82
+
83
+The `text/event-stream` content type enables browser EventSource API:
84
+
85
+```javascript
86
+const source = new EventSource("/events");
87
+source.onmessage = (event) => console.log(event.data);
88
+```
89
+
90
+## Large File Streaming
91
+
92
+```python
93
+import aiofiles
94
+from justapi.responses import StreamingResponse
95
+
96
+
97
+@app.get("/large-file")
98
+async def download(request):
99
+ async def file_chunks():
100
+ async with aiofiles.open("data.csv", "rb") as f:
101
+ while chunk := await f.read(65536):
102
+ yield chunk
103
+
104
+ return StreamingResponse(file_chunks(), media_type="text/csv")
105
+```
106
+
107
+## Performance
108
+
109
+Streaming responses use Rust's `StreamBody` with zero-copy chunks. Overhead is minimal — measured at <0.05 ms per chunk.
110
+
111
+## See Also
112
+
113
+- [Agent System](/advanced/agent-system/) — Streaming with sessions and tools
114
+- [WebSockets & SSE Tutorial](/tutorials/websockets-sse/) — Real-time communication
115
+- [Response Classes API](/api-reference/responses/) — Response reference
0
116