swadhin / eu-air-traffic public

swadhinbiswas

[email protected]
2026-09-15 17:00:53 +0600 committed NaN years ago
8487557
commit 8487557
6 files changed
+192 -60
Showing 6 changed files+192-60
README.mdimages/eu-air-traffic.pngimages/logo.svgpaper/eu-air-traffic.pngpaper/paper.bibpaper/paper.md
▼README.md
+44-60
@@ -1,18 +1,21 @@
1
+<img src="images/logo.svg" width="48" alt="EU air traffic logo" align="left" />
2
+
1
3
# EU Air Traffic — a zero-cost air-traffic data platform
2
4
3
5
[![CI](https://github.com/swadhinbiswas/eu-air-traffic/actions/workflows/ci.yml/badge.svg)](https://github.com/swadhinbiswas/eu-air-traffic/actions/workflows/ci.yml)
4
6
[![Lake](https://github.com/swadhinbiswas/eu-air-traffic/actions/workflows/lake.yml/badge.svg)](https://github.com/swadhinbiswas/eu-air-traffic/actions/workflows/lake.yml)
5
7
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
6
8
7
-**Live dashboard:** https://airtraffic-eu.pages.dev · **Live API:** https://vps.swadhin.cv/health
8
-**Walkthrough video:** _link coming — replace `VIDEO_URL` below_
9
+<br clear="left" />
9
10
10
-[![Watch the walkthrough](https://img.shields.io/badge/YouTube-watch-red?logo=youtube)](VIDEO_URL)
11
+Live dashboard: https://airtraffic-eu.pages.dev
12
+· Live API: https://vps.swadhin.cv/health
13
+· Demo video: coming soon
11
14
12
-A real-time and analytical view of European airspace: live aircraft positions,
13
-weather, schedules, delays, emissions and official Eurostat traffic benchmarks —
14
-built end-to-end on free tiers, with every quota, credit and write budget
15
-engineered deliberately.
15
+I built this to watch European airspace without spending anything: live
16
+aircraft positions, weather, schedules, delays, emissions, and Eurostat's
17
+official passenger numbers as a benchmark. The free tiers are the whole game
18
+here, so quotas and budgets drive most of the design decisions below.
16
19
17
20
---
18
21
@@ -20,59 +23,40 @@ engineered deliberately.
20
23
21
24
| Layer | What you get |
22
25
|---|---|
23
-| **Live map** | ~2,000 aircraft with callsign, type, altitude, speed, vertical rate, route lines, METAR/TAF stations, searchable by callsign/registration/type/airport |
24
-| **Live analytics** | Airspace composition, altitude bands, operators, carbon intensity (OpenAP kinematic model, measured vs. estimated) |
25
-| **Business analysis** | Delays, punctuality, cancellations, airport leaderboard, airline rankings, route performance, weather impact, seasonal trends |
26
-| **Official benchmark** | Eurostat monthly passengers per airport cross-checked against the movements this platform observed |
27
-| **SQL workbench** | Query the serving copy directly in the browser (read-only token) |
28
-| **Ops** | Pipeline step report, data-quality report, dbt lineage, freshness |
26
+| Live map | ~2,000 aircraft with callsign, type, altitude, speed, vertical rate, route lines, METAR/TAF stations, searchable by callsign/registration/type/airport |
27
+| Live analytics | Airspace composition, altitude bands, operators, carbon intensity (OpenAP kinematic model, measured vs. estimated) |
28
+| Business analysis | Delays, punctuality, cancellations, airport leaderboard, airline rankings, route performance, weather impact, seasonal trends |
29
+| Official benchmark | Eurostat monthly passengers per airport cross-checked against the movements this platform observed |
30
+| SQL workbench | Query the serving copy directly in the browser (read-only token) |
31
+| Ops | Pipeline step report, data-quality report, dbt lineage, freshness |
29
32
30
33
## Architecture
31
34
32
-```
33
- ┌────────────────────── VPS (2 cores, systemd) ──────────────────────┐
34
- OpenSky (OAuth2) │ collector: positions · flights · weather · fuel · reference · │
35
- adsb.lol │ AirLabs schedules · live FastAPI (:8090 via Caddy TLS) │
36
- airplanes.live └───────────────┬───────────────────────────────────────────────────┘
37
- aviationweather │ publish (batched, keyed)
38
- Open-Meteo ▼
39
- AirLabs ┌───────────────┐
40
- Eurostat (monthly) │ Aiven Kafka │ 5 topics: eu-positions · eu-flights ·
41
- │ │ eu-weather · eu-fuel · eu-reference
42
- └───────┬───────┘ (records carry `_kind`, 24h retention)
43
- │ drain by consumer offset (GitHub Actions, ~15 min)
44
- ▼
45
- ┌───────────────────────── Lake ─────────────────────────┐
46
- │ Bronze Parquet → Silver (incremental) → DuckDB star │
47
- │ schema → dbt (103 models/tests) → Gold marts │
48
- └───────┬───────────────────────────────┬────────────────┘
49
- │ │
50
- Hugging Face dataset lake MotherDuck warehouse
51
- (Bronze/Silver history) (reproducible, full data)
52
- │ │
53
- └──────────► Turso serving copy ◄┘
54
- (read-only token, atomic swaps)
55
- │
56
- React + Vite dashboard
57
- (Cloudflare Pages, live API)
58
-```
35
+![EU air traffic architecture: a VPS collector publishes to Kafka, a scheduled pipeline builds bronze, silver and dbt marts, and Hugging Face, MotherDuck and Turso serve the dashboard.](images/eu-air-traffic.png)
36
+
37
+A collector on a small VPS polls upstream APIs and publishes to a five-topic
38
+Kafka cluster. Every 15 minutes a scheduled pipeline drains Kafka, builds
39
+Bronze/ Silver/ DuckDB layers and dbt marts, then pushes to Hugging Face (the
40
+dataset lake), MotherDuck (the full warehouse) and Turso (the copy the browser
41
+is allowed to read). The dashboard reads Turso plus the collector's live API.
59
42
60
43
### Why two serving stores
61
-MotherDuck has the full warehouse but its free plan cannot issue scoped
62
-read-only tokens. Turso does, so the browser reads a derived, bounded serving
63
-copy: cheap to hold, safe to expose, refreshed every cycle.
44
+
45
+MotherDuck holds the full warehouse, but its free plan cannot issue scoped
46
+read-only tokens. Turso can, so the browser reads a derived, bounded copy
47
+there. Cheap to hold, safe to expose, refreshed every cycle.
64
48
65
49
## Zero-cost constraints (and how they shaped the design)
66
50
67
51
| Constraint | Design response |
68
52
|---|---|
69
-| Kafka free plan: **5 topics only** | One topic per domain; weather (metar/taf/forecast) and reference data multiplex with a `_kind` discriminator the sink splits back into datasets |
70
-| OpenSky: **credit budgets per endpoint** | `/flights/all` (one request, both ends) + live departures for 4 hubs + a nightly arrivals backfill; ~2.7k of 4k daily credits |
71
-| AirLabs: **1,000 calls/month, 50 rows/call** | Rotating hubs, persisted monthly counter that stops at the budget, IATA→ICAO resolved from bundled data (no extra calls) |
72
-| Turso: **row-write budget** | Statics upload only when a content hash changes; positions have a refresh floor; growing tables are watermark-synced |
73
-| Hugging Face: **storage** | Silver is partitioned per source; only changed files are pushed; unchanged files are recognised as no-ops |
53
+| Kafka free plan: 5 topics only | One topic per domain; weather (metar/taf/forecast) and reference data multiplex with a `_kind` discriminator the sink splits back into datasets |
54
+| OpenSky: credit budgets per endpoint | `/flights/all` (one request, both ends) + live departures for 4 hubs + a nightly arrivals backfill; ~2.7k of 4k daily credits |
55
+| AirLabs: 1,000 calls/month, 50 rows/call | Rotating hubs, persisted monthly counter that stops at the budget, IATA→ICAO resolved from bundled data (no extra calls) |
56
+| Turso: row-write budget | Statics upload only when a content hash changes; positions have a refresh floor; growing tables are watermark-synced |
57
+| Hugging Face: storage | Silver is partitioned per source; only changed files are pushed; unchanged files are recognised as no-ops |
74
58
| Runner minutes | Public repo: free. Frontend-only pushes skip the lake entirely |
75
-| VPS: **2 cores** | The box only collects; all transformation runs in CI |
59
+| VPS: 2 cores | The box only collects; all transformation runs in CI |
76
60
77
61
## Data sources
78
62
@@ -125,9 +109,9 @@ AIR_TRAFFIC_DUCKDB_PATH=$PWD/warehouse/air_traffic.duckdb uv run dbt build --pro
125
109
| `HF_TOKEN`, `HF_REPO` | Dataset lake |
126
110
| `MOTHERDUCK_TOKEN`, `MOTHERDUCK_DATABASE` | Warehouse |
127
111
| `TURSO_DATABASE_URL`, `TURSO_AUTH_TOKEN` | Serving copy (publisher uses the read-write token) |
128
-| `VITE_LIVE_URL`, `VITE_TURSO_URL`, `VITE_TURSO_TOKEN` | Dashboard build-time values; the Turso token **must be read-only** |
112
+| `VITE_LIVE_URL`, `VITE_TURSO_URL`, `VITE_TURSO_TOKEN` | Dashboard build-time values; the Turso token must be read-only |
129
113
130
-Tuning knobs (defaults are safe): `POSITIONS_INTERVAL_SECONDS=15`,
114
+Sane defaults, override only if needed: `POSITIONS_INTERVAL_SECONDS=15`,
131
115
`POSITIONS_PUBLISH_INTERVAL_SECONDS=300`, `FLIGHTS_INTERVAL_SECONDS=1800`,
132
116
`FLIGHTS_LOOKBACK_MINUTES=90`, `FLIGHTS_ARRIVAL_INTERVAL_SECONDS=21600`,
133
117
`AIRLABS_HUBS_PER_CYCLE=4`, `AIRLABS_MONTHLY_BUDGET=800`, `LAKE_WINDOW_SECONDS=420`.
@@ -161,22 +145,22 @@ optional Caddy TLS front via `SITE_ADDRESS`. `./deploy.sh --logs`, `--ps` and
161
145
`--down` manage the stack. No local Kafka is needed: point `.env` at Aiven (or
162
146
any broker) and the containers connect out.
163
147
164
-## Reliability engineering — what broke and what it taught
148
+## Reliability engineering: what broke and what changed
165
149
166
-This platform runs unattended on free infrastructure. The interesting engineering
167
-isn't the happy path; it's these classes of failure, all fixed with tests:
150
+This platform runs unattended on free infrastructure, and most of the work went
151
+into failure handling rather than the happy path. Each item below has a test:
168
152
169
-- **Silent success.** Steps that logged an error and exited 0 (HF upload rejected
170
- by a trailing space in a configured repo id, a Turso publish that never wrote).
171
- Failures are now loud, and credentials set-but-empty raise.
153
+- **Silent success.** Steps logged an error and exited 0 (an HF upload rejected
154
+ by a trailing space in a configured repo id; a Turso publish that never
155
+ wrote). Failures are now loud, and credentials that are set but empty raise.
172
156
- **Fresh-boot assumptions.** A publish cadence compared `monotonic()` against
173
- `0.0`, which only works if system uptime exceeds the interval — CI runners and
174
- restarted services publish nothing. Missing state now means "due".
157
+ `0.0`, which only works when system uptime exceeds the interval. Missing
158
+ state now means "due".
175
159
- **Destructive retries.** A failed lake pull could push a single window over
176
160
the full Silver history; publishers could wipe serving tables mid-run. Pulls
177
161
fail the job, and static tables load into a shadow table and swap atomically.
178
162
- **Double counting.** OpenSky movements and AirLabs schedules describe the same
179
- flight with different ids; marts now dedupe on callsign + date + endpoint and
163
+ flight with different ids; marts dedupe on callsign + date + endpoint and
180
164
only average delays that are actually known.
181
165
- **Timezone traps.** Casting `TIMESTAMPTZ` to `TIMESTAMP` shifts by the session
182
166
offset, silently re-reading old rows and skipping boundary ones.
183
167
▼images/eu-air-traffic.png
+0-0
Binary file not shown.
▼images/logo.svg
+10-0
@@ -0,0 +1,10 @@
1
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-label="EU air traffic logo">
2
+ <circle cx="32" cy="32" r="29" fill="#0b1220"/>
3
+ <circle cx="32" cy="32" r="29" fill="none" stroke="#38bdf8" stroke-width="2.5"/>
4
+ <circle cx="32" cy="32" r="19" fill="none" stroke="#38bdf8" stroke-width="1.25" opacity="0.55"/>
5
+ <circle cx="32" cy="32" r="9" fill="none" stroke="#38bdf8" stroke-width="1.25" opacity="0.35"/>
6
+ <line x1="32" y1="32" x2="54" y2="12" stroke="#34d399" stroke-width="2.5" stroke-linecap="round"/>
7
+ <circle cx="32" cy="32" r="3" fill="#34d399"/>
8
+ <circle cx="46" cy="24" r="2.4" fill="#fbbf24"/>
9
+ <circle cx="22" cy="40" r="2.4" fill="#f87171"/>
10
+</svg>
0
11
▼paper/eu-air-traffic.png
+0-0
Binary file not shown.
▼paper/paper.bib
+27-0
@@ -0,0 +1,27 @@
1
+@misc{opensky-api,
2
+ author = {{OpenSky Network}},
3
+ title = {The OpenSky Network API 1.4.0 documentation: REST API},
4
+ year = {2021},
5
+ url = {https://openskynetwork.github.io/opensky-api/rest.html}
6
+}
7
+
8
+@misc{eurostat-avia-paoa,
9
+ author = {{Eurostat}},
10
+ title = {Air passenger transport by type of schedule, transport coverage and main airports (avia\_paoa)},
11
+ url = {https://ec.europa.eu/eurostat/api/dissemination/statistics/1.0/data/avia_paoa?format=JSON},
12
+ note = {Accessed 15 September 2026}
13
+}
14
+
15
+@misc{openap-handbook,
16
+ author = {Sun, Junzi and Hoekstra, Jacco},
17
+ title = {The OpenAP Handbook},
18
+ url = {https://openap.dev/},
19
+ note = {Accessed 15 September 2026}
20
+}
21
+
22
+@misc{airlabs-schedules,
23
+ author = {{AirLabs}},
24
+ title = {Flight Schedules API},
25
+ url = {https://airlabs.co/docs/schedules},
26
+ note = {Accessed 15 September 2026}
27
+}
0
28
▼paper/paper.md
+111-0
@@ -0,0 +1,111 @@
1
+---
2
+title: 'EU Air Traffic: an open platform for live and historical European airspace analytics'
3
+tags:
4
+ - Python
5
+ - aviation
6
+ - ADS-B
7
+ - data engineering
8
+ - emissions
9
+ - air transport
10
+authors:
11
+ - name: TODO-your-full-name
12
+ # orcid: 0000-0000-0000-0000
13
+ affiliation: 1
14
+affiliations:
15
+ - name: TODO-affiliation, Country
16
+ index: 1
17
+date: 15 September 2026
18
+bibliography: paper.bib
19
+---
20
+
21
+# Summary
22
+
23
+EU Air Traffic collects live and scheduled flight data over Europe and stores
24
+it as an open data lake. A small collector polls aircraft positions, flight
25
+movements, schedules, weather, and reference tables, publishes the records to
26
+Kafka, and a scheduled pipeline turns the stream into versioned Bronze, Silver,
27
+and Gold layers with dbt models and tests. The same repository ships a fuel and
28
+emissions estimator built on a precomputed OpenAP grid, a monthly Eurostat
29
+benchmark of official passenger numbers, and a browser dashboard backed by a
30
+read-only serving copy. The whole system runs on free infrastructure tiers,
31
+with quota and budget handling built into the design.
32
+
33
+# Statement of need
34
+
35
+Anyone studying European air traffic starts from scattered pieces: a movement
36
+feed here, schedules behind an API key there, weather somewhere else, official
37
+statistics in yet another format. Each source has its own credentials, rate
38
+limits, field names, and gaps. Students and researchers who want a composed
39
+picture end up writing the same glue code privately, and the result is rarely
40
+published as reusable software.
41
+
42
+This project is that glue, maintained as a system instead of a script. It is
43
+for people who need European flight movements with delays, live positions with
44
+emissions estimates, and airport-level aggregates they can query or download.
45
+The pipeline deduplicates movements seen by several providers, keeps unknown
46
+delays as missing values instead of zeros, and checks every layer with tests
47
+before publishing.
48
+
49
+# State of the field
50
+
51
+The data this system stands on is open. The OpenSky Network publishes the
52
+reference API for community flight tracking data [@opensky-api]. Eurostat
53
+publishes monthly passenger figures per airport, which this project fetches as
54
+an independent benchmark [@eurostat-avia-paoa]. Schedules and delay fields come
55
+from AirLabs [@airlabs-schedules], and the emissions grid is precomputed from
56
+the OpenAP performance handbook [@openap-handbook].
57
+
58
+Those are providers and libraries, not platforms. We are not aware of an open,
59
+end-to-end equivalent that takes these feeds from collection to a served
60
+warehouse with scheduled refreshes, quota-aware polling, and tested analytical
61
+models in one repository. The contribution here is the composition: the
62
+collectors, the lake, the models, and the serving layer are designed to run
63
+unattended on free tiers, and the repository documents each budget it respects.
64
+
65
+# Software design
66
+
67
+The runtime has two halves. A collector on a small server polls each source on
68
+its own interval, enriches positions with emissions and classification, and
69
+publishes keyed records to a five-topic Kafka cluster, multiplexing related
70
+datasets with a `_kind` discriminator. GitHub Actions drains the bus every 15
71
+minutes, appends Bronze Parquet, merges incremental Silver tables, builds a
72
+DuckDB star schema, and runs the dbt project. Results land in a Hugging Face
73
+dataset, a MotherDuck warehouse, and a Turso serving copy that the dashboard
74
+reads with a read-only token.
75
+
76
+Two decisions carry most of the design. The first is that nothing the pipeline
77
+writes can silently vanish: Silver merges are additive, serving tables swap
78
+atomically, and growing tables resume from watermarks. Any step that reports
79
+success without writing fails the run instead. The second is that every
80
+external budget is enforced in code. OpenSky requests are counted against the
81
+documented credit buckets. The schedules fetcher stops at a persisted monthly
82
+call cap. Serving tables are republished only when a content hash changes.
83
+\autoref{fig:architecture} shows the full path from data collection to the
84
+browser.
85
+
86
+![Architecture of the EU Air Traffic platform, from collection through Kafka to the lake, warehouse, and dashboard. \label{fig:architecture}](eu-air-traffic.png)
87
+
88
+# Research impact statement
89
+
90
+The author operates the platform as a public dashboard and publishes the lake
91
+as a versioned dataset. The pipeline feeds the author's own traffic, delay,
92
+and emissions analyses. There are no external publications using the software
93
+yet, so this statement records current use rather than claiming adoption. The
94
+repository is public with tests and continuous integration. The dataset and
95
+dashboard give other groups a ready-made starting point for air-transport
96
+work, which is where wider use would come from.
97
+
98
+# AI usage disclosure
99
+
100
+An AI coding assistant (Muse Spark) was used during development for
101
+implementation, refactoring, tests, documentation, and drafting of this
102
+manuscript. The author reviewed, edited, and validated all AI-assisted outputs
103
+and made the core architectural and design decisions. The author takes full
104
+responsibility for the accuracy and licensing of the software and this paper.
105
+
106
+# Acknowledgements
107
+
108
+The author thanks the maintainers of the open data providers this system
109
+builds on, and the open-source projects it depends on.
110
+
111
+# References
0
112