Source Job

$123,696–$254,667/yr
US

  • Design and implement robust data infrastructure in AWS, using Spark with Scala.
  • Evolve our core data pipelines to efficiently scale for our massive growth.
  • Store data in optimal engines and formats, matching your designs to our performance needs and cost factors.

Scala Spark AWS SQL Data Infrastructure

20 jobs similar to Software Engineer II, Big Data

Jobs ranked by similarity.

US Unlimited PTO

  • Build infrastructure and data automation pipelines to ingest, process, and load data from various sources.
  • Collaborate with stakeholders and data science teams to develop data products aligned with organizational goals.
  • Develop data analysis tools to provide insights and capture key metrics.

Columbia General is seeking a Senior Data Engineer to help transform data into actionable insights that drive decision-making. The company fosters a dynamic, collaborative environment that supports growth and innovation.

Global

  • Design and implement batch and real time ingestion pipelines from internal and external sources.
  • Implement automated data quality checks, observability, and SLA monitoring.
  • Optimise datasets and pipelines for analytics, ML training, and API consumption.

Software Mind develops solutions that make an impact for companies around the globe. They build cross-functional engineering teams that take ownership and crave more, always on the lookout for talented people who bring passion and creativity to every project.

US

  • Design, develop, and maintain robust and scalable data pipelines using Apache Spark and cloud-native data services.
  • Build, optimize, and support ETL/ELT workflows to enable analytics, reporting, and downstream applications.
  • Implement and manage data solutions using Databricks, Delta Lake, and Unity Catalog.

Onebridge, a Marlabs Company, is a global AI and Data Analytics Consulting Firm that empowers organizations worldwide to drive better outcomes through data and technology. Since 2005, they have partnered with some of the largest healthcare, life sciences, financial services, and government entities across the globe.

  • Design, build, and maintain scalable data pipelines using AWS Glue (PySpark), or equivalent orchestration and transformation tools.
  • Engineer and optimise the ClickHouse warehouse for sub-second query performance across all back-offices.
  • Implement data contracts between back-office and the platform.

Block Labs is a premier technology studio operating at the bleeding edge of Web3, Artificial Intelligence, and iGaming. We are a collective of senior engineers, product strategists, and builders who refuse to compromise on architecture.

$190,000–$280,500/yr
US Canada

  • Architect and evolve scalable data ingestion and egress frameworks and pipelines that are well tested and offer strong data quality monitoring.
  • Architect and evolve our CI/CD processes - enhancing the testing environment and observability.
  • Enhance our Claude Code / LLM development support capabilities - creating tools / skills / agents that give our LLMs more context and help us continually improve their abilities to debug, create code, and maintain systems.

Life360’s mission is to keep people close to the ones they love. They have a mobile app, tracking devices, and a pet GPS tracker. Life360 has more than 500 (and growing!) remote-first employees and delivers peace of mind and enhances everyday family life.

Mexico

  • Contribute to the design and implementation of scalable data solutions.
  • Build and optimize batch and streaming ingestion pipelines.
  • Ensure data quality, reliability, and performance across pipelines and datasets.

Blend is an AI services provider that co-creates impact for clients through data science, AI, technology, and people. They aim to fuel bold visions by aligning human expertise with artificial intelligence, fostering innovation, and unlocking value for their clients.

$130,000–$160,000/yr
US

  • Build, maintain, and run CI/CD pipelines and infrastructure-as-code for the Smile Digital Health platform.
  • Provision, configure, and operate cloud-based Spark clusters and distributed data processing environments.
  • Design and maintain scalable, secure infrastructure templates and deployment automation across cloud environments.

Smile Digital Health makes it easy for healthcare stakeholders to collect and exchange data with our leading FHIR-based data liberation platform. At its heart, the Smile platform enables people and organizations to better manage healthcare data; the company was #19 on Deloitte's Technology Fast 50 Ranking for 2024!

$70,560–$81,120/yr
Global

  • Enable efficient data access by creating and maintaining data pipelines.
  • Collaborate with ML engineers to design and maintain automation for machine learning training, quality assessment, and model release process.
  • Build data infrastructure from the vast amount of data for analytics, hypothesis testing and company metrics.

Eneba is building an open, safe, and sustainable marketplace for gamers. Their marketplace supports close to 20m+ active users and provides trust and safety.

US EMEA

  • Design, build, and maintain distributed data pipelines that power Spotify Wrapped data stories and personalized experiences for more than 300M users globally.
  • Partner with Data Scientists to evaluate and operationalize new Wrapped story concepts, balancing personalization, scalability, and eligibility requirements.
  • Build scalable systems that process large-scale listening data and generate insights that celebrate users’ unique listening journeys.

The Personalization team makes deciding what to play next easier and more enjoyable for every listener. They are behind some of Spotify’s most-loved features. Join them and you’ll keep millions of users listening by making great recommendations to each and every one of them.

Global

  • Query and process large datasets using Trino (SQL).
  • Work with data in AWS environment using PySpark.
  • Build audience segments based on website activity, call data, behavioral patterns and segment rules.

Kyivstar is one of the largest and most beloved telecom companies in Ukraine. They offer opportunities to work with large-scale real-world data in a friendly and collaborative team environment, with possibilities for professional development and career growth.

Global

  • Design and build end-to-end data pipelines across the RAW, Silver, and Gold layers of the Medallion Architecture.
  • Architect data ingestion, transformation, standardization, and serving processes, that structure data flows from diverse and heterogeneous sources into a coherent analytical foundation.
  • Model data for analytical consumption following Data Warehouse best practices, including Star Schema design and dimensional modeling suited for business intelligence and AI-readiness.

CI&T is a tech transformation specialist, uniting human expertise with AI to create scalable tech solutions. With over 8,000 CI&Ters around the world, they’ve built partnerships with more than 1,000 clients during their 30 years of history, valuing diverse identities and life experiences.

$110,000–$125,000/yr
US Unlimited PTO 12w paternity

  • Design, develop, and maintain robust, scalable ETL/ELT data pipelines using Python, SQL, and data processing frameworks.
  • Implement data quality checks, monitoring, and alerting across all data pipelines to ensure data integrity and reliability.
  • Work closely with data analysts, data scientists, and business intelligence engineers to understand their data requirements and deliver reliable, high-quality data access.

InStride Health delivers specialty anxiety and OCD care. They focus on expanding access to insurance-based care, increasing engagement, and improving treatment outcomes by combining clinical care and innovative technology. They are a mission-driven company.

$125,000–$150,000/yr
US Unlimited PTO

  • Build, maintain, and operate data pipelines and curated data products across Snowflake, Airflow (MWAA), AWS, Python, and SQL.
  • Implement observability and data quality controls and build monitoring for freshness, volume, schema, distribution, and lineage.
  • Define and enforce data platform standards, establish orchestration patterns, DAG anti-patterns, deployment practices, observability standards, data quality patterns, and operational runbooks used across the organization.

Attain Finance provides financial services. CURO (dba Cash Money®, LendDirect®, Heights Finance, Southern Finance, Covington Credit, Quick Credit, and First Heritage Credit).

Canada Unlimited PTO

  • Design and build an integrated data platform, unifying existing tools and pipelines into a cohesive, scalable architecture.
  • Own data pipelines and SLAs end to end, ensuring reliable data movement between systems with clear expectations.
  • Shape the data strategy and platform roadmap, researching new technologies and introducing tools as the platform evolves.

Wrapbook is a vertical fintech platform that enables companies to seamlessly onboard, pay, and insure their workforces, operating in the entertainment industry. They are at an exciting stage of growth, having raised over 30M from investors like Andreessen Horowitz.

South America

  • Design, build, and own scalable data pipelines and systems that power analytics, machine learning, and business operations.
  • Drive system design for data architecture, owning data models and storage solutions to create scalable foundations for the team.
  • Collaborate with engineering, product, and data teams to translate business needs into technical solutions, ensuring data quality and performance standards.

Goodway Group is a remote-first, data-driven, and technology-enabled digital media and marketing services firm with a 90+ year history, offering the security of an established company with a start-up feel. It is a diverse team of strategists, practitioners, technologists, and data scientists that is recognized as a top workplace and a certified partner to The Trade Desk.

India

  • Design scalable data pipelines and backend systems from the ground up.
  • Leverage AWS and GCP for real-time and batch processing.
  • Manage databases and Data Warehouses, optimizing ETL workflows.

Delivery Solutions, a UPS company, is looking for a Senior Data Engineer to join their team. They are a growing company.

LATAM

  • Design, build, and maintain scalable data pipelines
  • Develop and optimize ETL processes to support data products
  • Work with structured and unstructured data across SQL and NoSQL systems

They are seeking a Data Engineer to support the development of data products that power critical business functions. They seem to have a collaborative, cross-functional Agile environment where you'll partner closely with technical and business teams to deliver high-quality data solutions.

US Unlimited PTO

  • Build and own end-to-end data pipelines in Snowflake — from raw ingestion through transformation to serving layers for AI products.
  • Partner with ML engineers and data scientists to build and maintain AI-specific data infrastructure.
  • Consolidate fragmented data sources across the organization into reliable, automated pipelines.

Power Digital is a tech-enabled growth firm at the intersection of marketing, consulting, and data intelligence. They ignite revenue and brand recognition for leading and emerging companies. They are a people-first firm with a focus on diversity and have a dynamic team of consultative marketers, creatives, analysts and technologists.

India

  • Build and Maintain Bronze/Silver Layer Pipelines: You will ensure core data sources lands accurately, on time, and with full lineage.
  • Lead Data Ingestion, Transformation, and Enrichment: You will own the end-to-end pipeline from raw file landing through cleansed, conformed staging tables, including deduplication, standardization, code mapping, and entity resolution.
  • Develop Automated Ingestion Pipelines: You will use Snowpipe, Matillion, or custom solutions with reliability, observability, and minimal manual intervention in mind.

Precision AQ is building a centralized Data Hub to consolidate fragmented data infrastructure, establish enterprise-wide data governance, and enable AI-ready analytics across our life sciences portfolio. This is a foundational initiative, not a maintenance role.

Global

  • Design, develop, and maintain data pipelines using Azure Databricks.
  • Build and optimize data transformations using PySpark and SQL in Databricks.
  • Implement and maintain Lakehouse architectures using Delta Lake.

Miratech helps visionaries change the world with enterprise and start-up innovation, supporting digital transformation for some of the world's largest enterprises. They are a values-driven organization with nearly 1000 full-time professionals and an annual growth rate exceeding 25%.