# π‘οΈ AegisVision
### AI Enterprise Surveillance & Behavioral-Monitoring Platform
**Plug-and-play, multi-camera face recognition with anti-spoofing liveness, occupancy & gate analytics, spatial anti-theft memory, real-time alerting, and a Generative-AI copilot.**
`FastAPI Β· ChromaDB Β· SQLite Β· OpenCV Β· insightface/ONNX Β· dlib Β· Tailwind CSS Β· WebSocket`
[](https://www.python.org/)
[](https://fastapi.tiangolo.com/)
[](https://tailwindcss.com/)
[](tests/)
[](docker-compose.yml)
[](LICENSE)
---
## What it does
AegisVision turns any CCTV camera, webcam, or RTSP stream into an intelligent command center for offices, schools, colleges, and work sites:
- **Enroll by drag-and-drop** β drop a folder `_/` with `info.json` + photos into `data/known_persons/`, and the person is live in minutes (startup sync *and* a folder watchdog).
- **Recognize many people, in real time** β SCRFD/ArcFace via ONNX (lightweight, accurate) with an automatic dlib fallback.
- **Never saves UNKNOWN faces** β unknown people are tagged on-frame in red, *zero files written to disk* (optional, off-by-default gated snapshot).
- **Spoof-resistant** β liveness scoring from face texture, gated by `ANTI_SPOOF_THRESHOLD`.
- **Track behavior** β gate in/out & live occupancy, active/idle-lazy/bunking/phone-distraction time, drowsiness & fatigue telegraphs, spatial anti-theft memory (asset displacement vs. owner).
- **Alert instantly** β rule-based escalation (UNKNOWNΓ3, phone-overuseΓ3, repeated drowsiness) with WebSocket push to the dashboard plus Telegram / JSON webhook.
- **Ask questions in plain English** β a Generative-AI copilot over the live DB (rule-based fallback when no API key is set).
---
## Screens
| Login & onboarding | Live recognition |
|---|---|
|  |  |
> Illustration clips from an earlier build β the current pipeline is described in the diagrams below.
---
## System architecture
**Data flow at a glance**
1. **Sources β Workers** β one background capture worker per configured camera (`CAMERA_STREAMS`), sharing the latest frame with back-pressure and auto-reconnect.
2. **Frame pipeline** β detect β track β anti-spoof β embed β match, per camera.
3. **Behavioral engines** β attendance/gate, productivity, health, and spatial memory consume identities + liveness.
4. **Single source of truth** β SQLite (relational) + Chroma (vector index) stay in sync.
5. **API + push** β typed FastAPI routes, MJPEG + WebSocket streams, alert dispatcher, rule engine, and copilot.
## Face recognition pipeline
- **Detection** `SCRFD-500M` (small model, <10ms on CPU) β **IoU tracking** keeps identities across frames and only re-embeds when a track window elapses β **liveness** gate β **ArcFace** `w600k_mbf` aligned **512-d** embeddings β **top-N cosine** lookup in Chroma.
- Thresholds (`FACE_DISTANCE_THRESHOLD`, `FACE_CONFIDENCE_MIN`, `ANTI_SPOOF_THRESHOLD`) live on one place in `backend/app/config.py`.
- The backend is swappable without code changes: `FACE_BACKEND=auto` prefers `insightface`, falls back to `legacy` dlib if the package/model is unavailable, and the vector index **self-heals** if the embedding dimension changes (128 β 512).
## Behavioral analytics & alerting
| Gate & occupancy | Productivity & wellbeing | Spatial & alerts |
|---|---|---|
| In/out events per camera with a 30 s/person cooldown, live occupancy, per-person daily summaries. | Per-person active / idle-lazy / bunking / phone-distraction minutes with daily aggregates via `/api/analytics/person/{id}/{day}`. | Asset registry, displacement vs owner (THEFT_SUSPECTED), rule-engine escalation & real WS alert feed. |
---
## Data model
Relational rows are the source of truth; the Chroma vector index mirrors `person_id` β embedding so a person can own many photos (aging, angles, re-enrolment).
## Deployment topology
One host does it all (Docker Compose): **NGINX** β **uvicorn** β persistent volumes for SQLite, ChromaDB, roster, and logs. Edge inputs are added by ID; no snowflake config.
---
## Tech stack
| Layer | Tools |
|---|---|
| Backend | Python 3.12 Β· FastAPI Β· Uvicorn Β· Pydantic v2 |
| Recognition | OpenCV Β· insightface/ONNX (SCRFD + ArcFace) Β· face_recognition/dlib (fallback) Β· onnxruntime |
| Data | SQLite (SQLAlchemy 2) Β· ChromaDB (n-dim cosine) |
| Realtime | WebSocket Β· Server-Sent Events / MJPEG streaming |
| AI | Google GenAI Gemini (rule-based fallback) |
| Frontend | Vanilla-JS + Tailwind CSS SPA (served by FastAPI, no bundler) |
| Ops | Docker Compose Β· uv Β· pytest Β· watchdog roster watcher |
---
## Quickstart
### 1. Clone
```bash
git clone git@github.com:swadhinbiswas/AegisVision.git
cd AegisVision
```
### 2. Backend (Python 3.12, managed with `uv`)
```bash
# Copy the environment template and fill in camera sources / Gemini key / Telegram creds
cp backend/.env.example backend/.env
# Install locked dependencies
uv sync
# Run (from the backend/ directory)
uv run uvicorn main:app --reload --host 0.0.0.0 --port 8000
```
> The venv is locked with `uv` β no version drift on heavy native deps (`dlib`, `onnxruntime`, `insightface` optional).
### 3. Open the dashboard
The dashboard (vanilla-JS + Tailwind) is served automatically by FastAPI (`SERVE_FRONTEND=true` default) β open . API docs at .
### 4. Add people (plug-and-play)
```
data/known_persons/
βββ EMP001_Alice/
βββ info.json # {"name":"Alice","role":"Engineer","department":"Lab"}
βββ photo.jpg # one or more JPEG/PNG photos
```
Or via the API β `POST /api/personnel/enroll` (multipart). Deletion is `DELETE /api/personnel/:person_id`.
---
## Configuration reference
Core knobs in `backend/app/config.py` (all overridable via env):
| Env var | Default | Purpose |
|---|---|---|
| `CAMERA_STREAMS` | JSON | map of `camera_id` β `name` / `source` (int index or RTSP/MJPEG URL) / `type` |
| `FACE_BACKEND` | `auto` | `auto` (prefer insightface) Β· `insightface` Β· `legacy` |
| `FACE_INSIGHTFACE_MODEL` | `buffalo_sc` | model zoo name (`buffalo_sc` light Β· `buffalo_l` accurate) |
| `FACE_DET_SIZE` | `320` | detector input size |
| `FACE_DISTANCE_THRESHOLD` | `0.60` | cosine-distance ceiling for a KNOWN match |
| `ANTI_SPOOF_THRESHOLD` | `0.70` | liveness gate |
| `ENABLE_UNKNOWN_SNAPSHOTS` | `false` | never store UNKNOWN frames unless explicitly enabled |
| `TELEGRAM_BOT_TOKEN` / `NOTIFY_WEBHOOK_URL` | blank | outbound push (blank = disabled) |
| `GEMINI_API_KEY` | blank | copilot (rule fallback used when blank) |
| `JWT_SECRET` | change-me | session signing (change in prod) |
Full list is in `backend/app/config.py` with env-aware defaults β see `backend/.env.example`.
---
## API surface
All `/api/**` responses are Pydantic-typed. Feed: `GET /api/stream/video_feed?camera_id=...` (MJPEG) and `GET /api/stream/ws` (WebSocket events).
| Group | Endpoints |
|---|---|
| Personnel | `GET /api/personnel` Β· `GET /api/personnel/:id` Β· `POST /api/personnel/enroll` Β· `GET /api/personnel/photo/:id` Β· `DELETE /api/personnel/:id` |
| Analytics | `GET /api/analytics/dashboard` Β· `/occupancy` Β· `/attendance` Β· `/productivity` Β· `/health` Β· `/assets` Β· `/timeseries` Β· `/person/:id/:day` Β· `/camera-activity` |
| Alerts | `GET /api/alerts` Β· `GET /api/alerts/types` Β· `POST /api/alerts/resolve/:id` |
| Assets | `GET /api/assets` Β· `POST /api/assets` Β· `DELETE /api/assets/:name` |
| Copilot | `POST /api/copilot/query` |
| System | `GET /api/system/health` Β· `/logs` Β· `/config` Β· `GET /api/metrics` |
| Auth | `POST /api/auth/login` Β· `GET /api/auth/me` |
---
## Repository layout
```
.
βββ backend/
β βββ app/
β β βββ config.py runtime thresholds, paths, camera map
β β βββ db/ SQLite (ORM) + ChromaDB vector index
β β βββ services/ recognition, ingestion, behavioral engines,
β β β rule engine, alerts, copilot, notify, ops
β β βββ routes/ thin HTTP/WS layer (Pydantic-typed)
β β βββ schemas/ request/response contracts
β βββ main.py FastAPI entry (lifespan bootstraps everything)
β βββ requirements.txt
βββ frontend/ static Tailwind dashboard (served by FastAPI)
βββ docs/diagrams/ architecture, pipeline, data model, deployment
βββ data/ runtime: roster, SQLite, Chroma, logs (git-ignored)
βββ tests/ pytest suite (API, ingestion, vectors, backend)
βββ docker-compose.yml single-host compose stack
βββ Makefile Β· requirements.txt dev helpers
```
---
## Run with Docker
```bash
docker compose up -d --build
# dashboard β http://localhost:8000
```
A `Makefile` gives you `make dev`, `make install`, `make test`, `make build-frontend`, `make docker-up/down`.
---
## Testing
```bash
# 51 tests β API smoke, plug-and-play ingestion, vector semantics (incl. backend
# swap dim rebuild), recognition backend selection + real-photo embed
make test
```
Tests are hermetic (temp data dir, offline-safe `FACE_BACKEND=legacy`).
---
## Roadmap
| Phase | Focus | Status |
|---|---|---|
| 0β1 | Foundations, camera pool, tracker, ONNX embed path | π’ on master |
| 2 | Gate zones, live occupancy, per-person daily analytics, typed analytics | π’ on master |
| 3 | Spatial anti-theft, rule escalation, WS + Telegram alerting | π’ on master |
| 4 | Health v2 (68-pt landmarks, EAR drowsiness, posture, fall risk) | π‘ next |
| 5 | Copilot v2 (function-calling SQL, RAG, vision, multi-provider) | π‘ next |
| 6 | JWT+RBAC everywhere, CI/CD, NGINX+TLS, DB partitioning | π‘ next |
Detailed breakdown: see [PLAN.md](PLAN.md) and the repo conventions in [AGENTS.md](AGENTS.md).
---
## Important behavior notes
- **Average attendance logging is cooldown-throttled** (30 s/person/camera) to keep the store clean.
- **UNKNOWN faces** are labelled on-frame and are heavy gated; snapshots are off by default and never go to disk when disabled.
- The frontend reads `VITE_API_BASE` / a `config.js` bootstrap β it never hardcodes a production host.
---
## Contributing
1. Read [AGENTS.md](AGENTS.md) β repository conventions, run and verify instructions.
2. Branch per phase/feature. Run `make test` + `python -m compileall backend/app` before finishing.
3. Update [PLAN.md](PLAN.md) when the phase moves.
---
MIT Β© [Swadhin Biswas](https://github.com/swadhinbiswas) β built for offices, schools & workplaces.