Senior Databricks Engineer

Role and Responsibilities:

• Architect, build, and maintain scalable ETL/ELT pipelines on the Databricks Lakehouse Platform using PySpark, Spark SQL, and Delta Lake.
• Design and implement medallion (bronze/silver/gold) data architectures and enforce data quality, governance, and lineage standards.
• Optimize Spark jobs and cluster configurations for performance and cost, including partitioning, caching, and autoscaling strategies.

Required Qualifications:

• 5+ years of data engineering experience, with 3+ years building production solutions on Databricks and Apache Spark.
• Expert proficiency in Python (PySpark) and advanced SQL.
• Deep hands-on experience with Delta Lake, Unity Catalog, and the medallion architecture pattern.

Preferred Qualifications:

• Databricks certification (Data Engineer Associate/Professional).
• Experience with Delta Live Tables, structured streaming, and real-time data processing.
• Familiarity with MLflow and supporting machine learning workflows in production.