Data Engineer

Data Infrastructure Setup:

• Architect and maintain robust, scalable, and secure data infrastructure on AWS leveraging Databricks.
• Data Pipeline Development:
• Design, develop, and maintain data pipelines, primarily using tools like Airbyte and custom-built services in Go, to automate data ingestion and ETL processes.
• Data Lake Management:
• Oversee the creation and maintenance of the data lake, ensuring efficient storage, high data quality, and effective partitioning, organization, performance, monitoring and alerting.

Integration and Customization:

• Integrate tools like Airbyte with various data sources and customize data flows to align with specific business needs.
• Where necessary, build custom connectors in Go to support unique data requirements.