✈️ Air Traffic Warehouse - Map of Content (MOC)

Welcome to the Air Traffic Warehouse documentation index. This central file is designed to help you navigate the entire repository documentation vault. Whether you are viewing this in Obsidian, VS Code, or GitHub, all links are relative and fully clickable.

🏛️ System Architecture Overview

Below is the high-level data flow and system architecture for the Air Traffic Platform. It follows the Medallion Architecture pattern using zero-cost, open-source modern data stack technologies.

graph TD
    %% Sources
    subgraph Sources ["External Data Sources"]
        API1["Flight APIs"]
        API2["Weather APIs"]
        API3["Fuel APIs"]
        API4["Holiday APIs"]
    end

    %% Ingestion
    subgraph Ingestion ["Ingestion Layer (FastAPI)"]
        F_API["/api/flights"]
        W_API["/api/weather"]
        FL_API["/api/fuel"]
        H_API["/api/holidays"]
    end

    %% Storage
    subgraph Storage ["Data Lake (Hugging Face Datasets)"]
        Bronze[("Bronze Layer (Raw Parquet)")]
        Silver[("Silver Layer (Cleaned Parquet)")]
        Gold[("Gold Layer (Marts Parquet)")]
    end

    %% Warehouse
    subgraph Warehouse ["Warehouse & BI (DuckDB)"]
        DB[("air_traffic.duckdb")]
        DBT["dbt Models"]
        Superset["Apache Superset Dashboards"]
    end

    %% Connections
    API1 --> F_API
    API2 --> W_API
    API3 --> FL_API
    API4 --> H_API

    F_API --> Bronze
    W_API --> Bronze
    FL_API --> Bronze
    H_API --> Bronze

    Bronze -->|"Polars Transform & Validate"| Silver
    Silver -->|"Polars & DuckDB Aggregate"| Gold
    Gold -->|"Load Schema"| DB
    DB --> DBT
    DBT --> DB
    DB --> Superset

    %% Orchestrator
    GHA["GitHub Actions (Orchestration & Cron Scheduling)"]
    GHA -.->|"Runs Ingestion"| Ingestion
    GHA -.->|"Triggers ETL"| Storage
    GHA -.->|"Executes dbt"| Warehouse

    %% Styling
    classDef source fill:#f9f,stroke:#333,stroke-width:2px;
    classDef ingest fill:#bbf,stroke:#333,stroke-width:2px;
    classDef lake fill:#bfb,stroke:#333,stroke-width:2px;
    classDef wh fill:#fbb,stroke:#333,stroke-width:2px;
    classDef orch fill:#ffb,stroke:#333,stroke-width:2px;

    class API1,API2,API3,API4 source;
    class F_API,W_API,FL_API,H_API ingest;
    class Bronze,Silver,Gold lake;
    class DB,DBT,Superset wh;
    class GHA orch;

🗺️ Documentation Map

Click on any of the sections below to navigate to the specific documentation files.

🌟 00 - Project Vision & Design

Foundational goals, requirements, and system design.

Architecture: High-level architectural diagrams, component layouts, and network topologies.
Goals: Primary objectives, business/technical success criteria, and non-goals of the project.
Requirements: Functional constraints, SLA requirements, and system properties.

✈️ 01 - Domain Knowledge

Crucial business domain context for air traffic control, airports, and airline metrics.

Airlines: Key airline indicators, fleet sizing, passenger capacities, and industry codes.
Airports: Hub structures, gate operations, runway logistics, and airport coordinates.
Aviation Basics: Airspace structure, flight scheduling (scheduled vs. actual), and status codes.
Delay Types: Detailed breakdown of delay categories (Carrier, Weather, NAS, Security, Late Aircraft).
Weather: Flight rules (VFR vs. IFR), meteorological phenomena, and weather station alignments.

🏗️ 02 - Data Engineering Concepts

Core dimensional modeling and architectural paradigms used in the platform.

Data Lake: Raw storage patterns, directory layout schemas, and Hugging Face Dataset advantages.
Data Warehouse: OLAP design principles, analytical queries, and structured indexing.
Dimension Tables: Design and attributes of dim_airport, dim_airline, dim_weather, and dim_date.
ETL vs ELT: Trade-off analysis between ETL (Polars in-memory) and ELT (DuckDB SQL-on-files).
Fact Tables: Transaction grain metrics for fact_flights and fact_delays.
Medallion Architecture: In-depth view of the Bronze (Raw) $ o$ Silver (Cleaned) $ o$ Gold (Marts) flow.
OLTP vs OLAP: Comparison of transactional workloads vs. analytical workloads.
Partitioning: Folder structures, partition pruning, and execution optimization.
Slowly Changing Dimensions (SCD): Management of changing properties over time (SCD Type 1 vs. Type 2).
Star Schema: Relational model connecting Fact tables to Dimension tables.

💾 03 - Storage & File Formats

How data is stored, compressed, and versioned.

Compression: Comparative analysis of Snappy vs. Gzip vs. Zstd in columnar files.
Dataset Versioning: How Git commit history is used on Hugging Face to track versioned releases of data.
Hugging Face Dataset: Integrating Hugging Face as a serverless, free-tier data lake storage engine.
Parquet: Apache Parquet columnar properties, metadata, and schemas.

🔄 04 - ETL Pipelines

The extract, transform, and load mechanisms.

Data Validation: Data quality rules, constraints, and validation thresholds.
Error Handling: Pipeline failure recovery, notifications, and logging.
Extraction: API collector scripts, rates, and pagination strategies.
Incremental Loading: Watermarking techniques using last_run.json to process only net-new events.
Loading: Load strategies (appending facts vs. upserting dimensions).
Transformation: Data cleaning, UTC timestamp normalization, and schema casting.

📊 05 - SQL Analytics

Standard analytical SQL expressions used to query our warehouse.

Aggregation: Fast summary metrics, SUM, AVG, and multi-column groupings.
CTE (Common Table Expressions): Improving query modularity, readability, and compilation.
Joins: Joining Star Schemas using left joins on surrogate keys.
Optimization: SQL tips (selecting specific columns, filtering early, utilizing indexes).
Ranking: Assigning order index using RANK, DENSE_RANK, and ROW_NUMBER.
Window Functions: Running calculations across partitions (AVG OVER, LAG, LEAD).

🦆 06 - DuckDB Warehouse

Leveraging DuckDB as our embedded SQL OLAP Engine.

Basics: Introduction to DuckDB, the "SQLite for Analytics".
Performance: Columnar/vectorized query execution, filter pushdowns, and parallel query scanning.
Querying Parquet: Running SQL directly over Parquet files on disk or remote URLs.
Warehouse Design: Structuring air_traffic.duckdb and linking it to BI tools.

🐻 07 - Polars Transformation

In-memory processing at scale using Polars.

Aggregation: High-speed grouping and multi-column aggregation.
DataFrames: Arrow memory layout, comparison with Pandas, and core types.
GroupBy: Fast parallel GroupBy operations.
Lazy API: Query optimization, scan operations, and predicate pushdowns.
Optimization: Polars-specific optimizations, avoiding python UDFs, streaming execution.

⏰ 08 - Orchestration

Running pipelines on schedule.

Cron: Standard cron formats and schedule trigger settings.
GitHub Actions: Serverless, zero-cost scheduling workflow (etl.yml).
Scheduling: Task dependencies, sequential execution, and alerts.

📈 09 - Analytics & BI

Deriving business value from the warehouse.

Business Questions: Key operational questions answered by our data models.
Dashboards: Visualizing the Gold Layer with Apache Superset dashboards.
KPIs: On-Time Performance (OTP), Cancellation Rates, Average Delays.

🐳 10 - Deployment & CI/CD

Production deployment and testing configurations.

CI-CD: Formatting, linting (ruff), and testing (pytest) pipelines.
Docker: Multi-container setup for FastAPI, Superset, and DBT.
Vercel: Serverless hosting for FastAPI ingestion service.

📝 11 - Architectural Decision Records (ADRs)

Documentation of critical architectural choices and their rationale.

Why DuckDB: Decision matrix leading to DuckDB.
Why GitHub Actions: Choosing GitHub Actions over Airflow/Dagster.
Why Hugging Face: Opting for Hugging Face Hub as our free storage layer.
Why Polars: Benchmarking Polars against Pandas and Spark.

🧭 Suggested Learning Path

If you are new to the codebase, we recommend reading the files in the following order:

flowchart LR
    Start([Start]) --> Vision["Project Vision & Design<br>1. Goals<br>2. Architecture"]
    Vision --> Domain["Domain Context<br>3. Aviation Basics<br>4. Delay Types"]
    Domain --> Concepts["Data Warehouse Design<br>5. Medallion Architecture<br>6. Star Schema"]
    Concepts --> Codebase["Technical Implementation<br>7. Polars Transform<br>8. DuckDB Analytics"]
    Codebase --> Ops["Operational Setup<br>9. Orchestration<br>10. Docker Deploy"]
    Ops --> End([End])

Start with the high-level design:
- Read Goals to understand what we are building.
- Read Architecture to see the structural flow.
Gain business domain context:
- Read Aviation Basics and Delay Types.
Understand the analytical modeling theory:
- Review Medallion Architecture and Star Schema.
See the code-level implementation details:
- Learn how we extract and clean data: Transformation and Lazy API.
- Learn how we query it: Querying Parquet.
Review deployment & scheduling:
- See GitHub Actions and Docker.

Note: All markdown files are equipped with custom metadata for easy filtering and query parsing in Obsidian.