swadhin / eu-air-traffic public
/ eu-air-traffic / Air Traffic Warehouse / 04 - ETL / Incremental Loading.md
swadhinbiswas added:cookebook for this project · 8/1/2026
ce5d03e
16 lines · 584 Bytes
Raw Download

« Back to Index

Incremental Loading

Processing only the data that has changed since the last run, rather than reprocessing everything.

Mechanism

  1. Watermarking: We store a last_run.json containing the timestamp of the last successful extraction.
  2. Filtering: The extraction phase only requests data updated_at > last_run_timestamp.
  3. Upserting: During the load phase, we handle duplicates using primary keys to ensure idempotency.

This ensures the pipeline is fast and cost-effective.

« Back to Index