Senior Data Engineer

What you’ll own:

• Build concurrent ingestion for provider APIs, webhooks, files, and scraping with rate limits and failure handling.
• Orchestrate scheduled workflows and historical backfills using Airflow, Dagster, or Prefect.
• Implement event-driven intake with Pub/Sub or Cloud Tasks including retries and dead-letter queues.

Raw-to-curated data:

• Preserve original payloads in GCS before transforming with dbt.
• Establish medallion layers for raw, standardized, and curated data.

Correctness through recovery:

• Implement idempotent upserts and safe retries, replay, and backfills without duplicates.
• Make data traceable and safe to reprocess.

Data quality and delivery:

• Define data contracts, build automated validation and pipeline tests.
• Design and optimize PostgreSQL schemas and queries for downstream products.