« Back to Index

Medallion Architecture

Overview

The Medallion Architecture (Bronze → Silver → Gold) is a data organization pattern popularized by Databricks for organizing data lakes with progressive quality improvement.

┌────────────────────────────────────────────────────┐
│                 Medallion Architecture                │
│                                                      │
│  ┌──────────┐     ┌──────────┐     ┌──────────┐   │
│  │  Bronze  │ ──→ │  Silver  │ ──→ │   Gold   │   │
│  │  (Raw)   │     │(Validated)│    │(Business) │   │
│  └──────────┘     └──────────┘     └──────────┘   │
│                                                      │
│  Quality: Low → Medium → High                        │
│  Structure: Raw → Cleaned → Aggregated               │
│  Users: DEs → Analysts → Business                    │
└────────────────────────────────────────────────────┘

Bronze Layer — The "Landing Zone"

Characteristics

• Representation: Exact copy of source data
• Transformations: None (zero processing)
• Schema: Source schema preserved as-is
• Immutable: Never overwritten, append-only
• Retention: Limited (raw data replayed if needed)
• Users: Data engineers only

Rules

1. No data transformations
2. Preserve source field names and types
3. Append-only (no updates, no deletes)
4. Store ingestion timestamp (ETL metadata, not source column)
5. Partition by ingestion date

Example Structure

bronze/
├── flights/
│   ├── 2024/01/01/   ← One folder per day of ingestion
│   │   └── ingestion_20240101T120000UTC.parquet
│   └── 2024/01/02/
│       └── ingestion_20240102T120000UTC.parquet
├── weather/
├── fuel/
└── holidays/

Bronze Code Pattern

def load_to_bronze(raw_data: pl.DataFrame, source: str, ingestion_date: str):
    """Store raw data in Bronze layer. No transformations."""
    path = f"bronze/{source}/{ingestion_date}/ingestion_{ingestion_date}.parquet"
    raw_data.write_parquet(path)  # As-is, zero processing
    upload_to_hugging_face(path)  # Persist to HF
    log.info(f"Bronze: {source} | {raw_data.height} rows stored at {path}")

Silver Layer — The "Validated Zone"

Characteristics

• Source: Bronze data
• Transformations: Clean, deduplicate, standardize
• Schema: Normalized, validated schema
• Idempotent: Re-running produces identical output
• Users: Data engineers + analysts

Transformations Applied

1. Remove duplicates (by business key)
2. Normalize timestamps to UTC
3. Validate airport codes (IATA 3-char pattern)
4. Handle NULL values (defaulting or flagging)
5. Enforce data types (e.g., delay_minutes = INTEGER, not STRING)
6. Filter out corrupted records → quarantine
7. Standardize string casing (airport_code = UPPER)

Example Structure

silver/
├── flights/
│   ├── 2024/01/      ← Partitioned by month (not day)
│   │   └── flights_202401.parquet
│   └── 2024/02/
├── weather/
├── airport_metadata/
└── fuel_prices/

Silver Code Pattern

def bronze_to_silver(bronze_path: str, source: str):
    """Clean and validate Bronze data → Silver."""
    raw = pl.scan_parquet(bronze_path)

    silver = (
        raw
        # 1. Deduplicate by business key
        .unique(subset=["flight_id", "date"], keep="last")
        # 2. Normalize timestamps to UTC
        .with_columns(
            [
                pl.col("scheduled_departure").str.to_datetime().dt.convert_time_zone("UTC"),
                pl.col("actual_departure").str.to_datetime().dt.convert_time_zone("UTC"),
            ]
        )
        # 3. Validate business rules
        .filter(
            (pl.col("delay_minutes") >= 0)
            & (pl.col("airport_code").str.len_bytes() == 3)
            & (pl.col("actual_arrival") > pl.col("actual_departure"))
        )
        # 4. Cast types
        .with_columns(
            [
                pl.col("delay_minutes").cast(pl.Int32),
                pl.col("airport_code").str.to_uppercase(),
            ]
        )
        .collect()
    )

    month = extract_month(silver["date"].min())
    output_path = f"silver/{source}/{month}/"
    silver.write_parquet(output_path)

    log.info(
        f"Silver: {source} | {silver.height} clean rows | "
        f"{raw.collect().height - silver.height} rows discarded"
    )

Gold Layer — The "Business Zone"

Characteristics

• Source: Silver data
• Aggregated: Business metrics calculated
• Star Schema: Fact and dimension tables
• Time-enriched: Date dimension joined, seasonal flags added
• Users: Analysts, business, dashboards

Transformations Applied

1. Calculate delay metrics (avg delay, delay category)
2. Calculate distance (haversine between airports)
3. Calculate flight duration (actual_arrival - actual_departure)
4. Join weather data to flights
5. Add time features (hour of day, day of week, month, quarter)
6. Create fact and dimension tables
7. Apply surrogate keys

Output Tables

gold/
├── fact_flights/
│   └── partitioned by date_key/
├── fact_delays/
├── dim_airport/
├── dim_airline/
├── dim_weather/
├── dim_date/
├── dim_fuel/
├── airport_metrics/
├── airline_rankings/
├── delay_analysis/
├── weather_impact/
└── seasonal_trends/

Gold Code Pattern

def silver_to_gold(silver_path: str, airport_dim: pl.DataFrame):
    """Build star-schema tables from Silver data."""
    flights = pl.scan_parquet(silver_path)
    airport_dim_lazy = pl.LazyFrame(airport_dim)

    # Generate surrogate keys
    fact_flights = (
        flights.join(
            airport_dim_lazy.select(["airport_key", "airport_code"]),
            left_on="departure_airport",
            right_on="airport_code",
            how="left",
        )
        .with_columns(
            [
                pl.col("airport_key").alias("departure_airport_key"),
            ]
        )
        .drop(["airport_code_right", "departure_airport"])
        # Calculate metrics
        .with_columns(
            [
                (pl.col("actual_arrival") - pl.col("actual_departure"))
                .dt.total_minutes()
                .alias("flight_duration_min"),
                pl.when(pl.col("status") == "C").then(True).otherwise(False).alias("is_cancelled"),
            ]
        )
        .collect()
    )

    fact_flights.write_parquet("gold/fact_flights/", partition_by="date_key")

Benefits of Medallion Architecture

Retention Strategy

Quarantine Handling

# During validation, split:
#   → Silver (valid records)
#   → Quarantine (invalid records)

valid, invalid = split_by_validation(raw)

valid.collect().write_parquet("silver/flights/")
invalid.collect().write_parquet("quarantine/flights/flights_invalid_202401.parquet")

# Invalid record details stored for debugging
log.warning(f"Quarantine: {invalid.height} invalid rows → quarantine/flights/")

Medallion in Data Pipelines

[Source API]
    │
    ▼
[Extract] ─────────────────────────────────
    │                                       │
    ▼                                       │
[Bronze: write Parquet raw]                 │
    │                                       │
    ▼                                       │
[Silver: dedup + validate + normalize]      │
    │                                       │
    ▼                                       │
[Gold: enrich + calculate metrics + star]   │
    │                                       │
    ▼                                       │
[dbt: create analytics views]               │
    │                                       │
    ▼                                       │
[Superset: serve dashboard queries]         │

Industry Adoption

Medallion architecture is now an industry standard for data lakehouse designs.

« Back to Index