Similar Jobs
See allData Engineer SR (Databricks | ADF | AI)
Jobgether
Brazil
Databricks
Azure Data Factory
Python
Middle Data Engineer (Azure Databricks)
Miratech
Global
PySpark
SQL
Delta Lake
Senior Research Data Engineer
PointClickCare
US
Python
SQL
PySpark
Data Engineer III
Emory University
US
Python
Scala
Azure Data Factory
Data Scientist/ Data Architect
DataDirect Networks
US
Python
SQL
Machine Learning
Responsibilities:
- Design and deploy ML-based anomaly detection pipelines layered on the Landing Zone to CDR ETL process, providing early-cycle flagging of data discrepancies.
- Build AI-assisted field mapping and classification tooling to accelerate source-to-target schema mapping across CDR cycles.
- Develop automated data quality scoring pipelines producing per-table and per-CDR-cycle quality metrics.
Requirements:
- 6+ years of applied AI/ML pipeline development and deployment for large-scale data reconciliation programs.
- 6+ years of Azure data platform engineering including Azure Databricks, Azure Data Factory, and Delta Lake.
- 10+ years of advanced T-SQL and PL/SQL development across SQL Server and Oracle.
Preferred Skills:
- 4+ years of continuous data quality enforcement using Great Expectations and parameterized pytest suites.
- 3+ years of legacy system data migration experience involving COBOL or mainframe source environments.
- 3+ years of Azure Purview data lineage and metadata management.
Derex Technologies Inc
Derex Technologies Inc specializes in providing IT consulting, staffing solutions and software services. Since 1996, the company has delivered customized IT talent solutions to global clients across North America, serving industries like finance, healthcare, and government.