Data Infrastructure Engineer

Core Data Pipelines:

• Design, implement, and maintain distributed ingestion pipelines for structured and unstructured data.
• Build scalable ETL/ELT workflows to transform, validate, and enrich datasets for AI/ML model training and analytics.

Distributed Systems & Storage:

• Architect pipelines across cloud object storage, data lakes, and metadata catalogs.
• Optimize large-scale processing with distributed frameworks.
• Implement partitioning, sharding, caching strategies, and observability for reliable pipelines.

Pretrain Data Processing:

• Support preprocessing of unstructured assets for training pipelines, including format conversion, normalization, augmentation, and metadata extraction.
• Implement validation and quality checks to ensure datasets meet ML training requirements.