swadhin / eu-air-traffic public
/ eu-air-traffic / Air Traffic Warehouse / 04 - ETL / Error Handling.md
swadhinbiswas added:cookebook for this project · 8/1/2026
ce5d03e
15 lines · 570 Bytes
Raw Download

« Back to Index

Error Handling

Robust pipelines must anticipate and gracefully handle failures.

Strategies

  • API Failures: Retry with exponential backoff. Alert if persistent.
  • Validation Failures: Log bad records, do not fail the entire batch unless error rate > threshold (e.g., 5%).
  • Database Locks: DuckDB is single-writer. Ensure pipeline orchestrator prevents concurrent write runs.
  • Notifications: Send alerts to a webhook (Slack/Discord) if a pipeline step fails.

« Back to Index