Job Description
Role:
- Design and implement robust, scalable data ingestion and transformation pipelines using Databricks, PySpark, and distributed processing.
- Implement Delta Lake principles, focusing on CDC and schema evolution.
- Establish and integrate data quality frameworks within CI/CD pipelines for data integrity.
Responsibilities:
- Support the implementation of data governance and cataloguing solutions.
- Proactively investigate and improve inconsistent legacy datasets.
- Guide and manage AI agents for code generation, ensuring quality and stability.
Requirements:
- 5+ years of professional experience in data engineering, focused on cloud and distributed processing environments.
- Strong experience with Databricks, PySpark, distributed processing, and Delta Lake.
- Expert SQL optimization and Python skills.
About Mobile Wave Solutions
Mobile Wave Solutions is a professional services company specializing in software development as a service, with a team of over 120 engineers.