Senior Data Engineer

Design, build, and implement generalized large-scale, sophisticated data pipelines using Nifi for downstream analytics and data science for our Sport Performance products. Design and develop scalable Nifi ingestion pipelines within AWS cloud services to consume real-time and batch data from external sources. Design and create data models for use throughout the ETL system. Utilize Kafka to efficiently and to effectively store data to move throughout the data pipeline and for downstream data science and analytics usage. Responsible for ETL development and warehousing using Python and Java. Create data pipeline triggers and filters within ETL (extract, transform, and load) process to ensure appropriate optimization of data flowing through system and resource usage. Implement monitoring and error handling for all new parts of the data pipeline to ensure observability and alerting is available. Establish rigorous unit testing across the data pipeline to ensure robustness of the system. Design data architecture and data models for both internal and external representations of data. Build the data transforms within the data pipeline to convert data from external to internal representations. Conduct data analytics and debugging bad data by writing SQL queries. Build automated cleaning of data to remove bad or unusable data from downstream consumers with logging to understand the frequency and depth of the underlying issues. Collaborate with other engineering teams to adopt standard methodologies, drive scalability, and increase consistency across systems. Follow best practices across Data Engineering to ensure scalable, consistent data architecture and system. Utilize Java language to build data processors in Nifi framework. Utilize Docker to ensure a consistent, repeatable, and isolated environment for software development and testing.