Source Job

  • Build and design large-scale distributed crawling bots and infrastructure to extract data from diverse sources including web pages, APIs, and PDFs.
  • Develop and maintain data pipelines using Python, Scrapy, Airflow, and Spark to ingest, clean, and normalize data for search indexing and downstream ML/NLP.
  • Collaborate with ML/NLP teams to integrate classification models and utilize LLMs for precise answers from OEM repair procedures.

Python Web Scraping Data Processing ElasticSearch Airflow

20 jobs similar to Senior Engineer (Python)

Jobs ranked by similarity.

$160,000–$200,000/yr
Global Unlimited PTO

  • Contribute to the architecture and improvement of our data acquisition and processing platform, increasing reliability, throughput, and observability.
  • Use and develop web crawling technologies to capture and catalog data on the internet.
  • Build, operate, and evolve large-scale distributed systems that collect, process, and deliver data from across the web.

People Data Labs provides people and company data, integrating thousands of compliantly sourced datasets into a single source of truth. They are a remote-first company focused on building innovative data solutions, with a collaborative team that values ownership and learning from failure.

Global

  • You ship scrapers and ingestion flows against messy, sometimes adversarial sources using HTTP/2 clients, TLS-fingerprint evasion, and browser automation fallbacks.
  • You own Postgres schema design and migrations end to end across per-country and per-domain schemas.
  • You build and maintain medallion transforms that are idempotent, content-hashed, and lineage-tracked.

Labrynth builds AI-powered platforms that navigate complex regulations and generate audit-level documentation. They operate as a small, high-velocity team embedded directly with clients to rapidly ship production-quality solutions.

Global 7w PTO

  • Lead the modernization of the data platform by migrating legacy Hadoop, Spark, and Impala pipelines to a scalable Databricks architecture.
  • Accelerate migration efforts using AI coding assistants like Claude and Codex to convert SQL and modernize ETL workflows.
  • Design and optimize scalable data pipelines with Airflow and Databricks, ensuring reliability, cost-efficiency, and data quality.

LivePerson is a leader in enterprise conversational AI and digital transformation, powering nearly a billion conversational interactions monthly. Fast Company named them the #1 Most Innovative AI Company, and they foster a remote-first, innovative culture.

$210,000–$260,000/yr
Americas 3w PTO 12w maternity 12w paternity

  • Build and ship developer-facing product features from idea to production
  • Own the reliability and quality of what you ship against a web that constantly changes and fights back
  • Turn hard, ambiguous problems into clean APIs developers love to use

Firecrawl is the easiest way to turn the web into data AI agents can use. They are a small team with 147k+ GitHub stars and over $10M ARR, growing fast and operating with high urgency.

$60,000–$80,000/yr
Global 2w PTO

  • Design and maintain scalable, high-performance systems for massive data ingestion and high availability.
  • Build resilient API integrations and advance observability with comprehensive logging and monitoring.
  • Optimize software delivery through CI/CD pipelines and foster collaborative excellence through Agile practices.

Pagefreezer provides solutions for archiving online content to ensure compliance and enable litigation. The company has been recognized as one of Canada's top employers and fosters a values-led culture.

$170,000–$230,000/yr
US Unlimited PTO

  • Build, expand, and optimize data infrastructure to create the most accurate dataset of identities and their relationships.
  • Develop and operate secure, scalable, and reliable data ingestion and ETL/ELT pipelines that meet product requirements.
  • Design and maintain a data observability framework to ensure data meets strict quality and freshness standards.

SentiLink provides innovative identity and risk solutions, empowering institutions and individuals to transact with confidence. The company is growing rapidly, has verified hundreds of millions of identities, and is backed by top investors like Craft Ventures and Andreessen Horowitz, with offices across the US and India.

India

  • Own the full lifecycle of platform features from problem definition to deprecation planning.
  • Lead Search and Knowledge Graph platform evolution to reduce redundancy and optimize payload size.
  • Mentor engineers and elevate technical standards across the team.

H1 provides a platform that optimally informs every doctor interaction globally, promoting health equity and trust in healthcare systems. The company harnesses data and AI to unlock medical insights and improve patient outcomes.

$190,000–$260,000/yr
3w PTO 12w maternity 12w paternity

  • Design and build crawling, indexing, and retrieval systems for Firecrawl Search at scale.
  • Push down latency and cost per query while improving ranking quality for LLMs.
  • Own services end-to-end from design to production with a focus on speed and iteration.

Firecrawl provides the easiest way to convert web content into clean, LLM-ready data for AI agents. With 147k+ GitHub stars and rapid growth to 8-figure ARR, we are a small, fast-moving team building infrastructure that other AI companies rely on.

Global

  • Design, build, and maintain scalable data pipelines using modern tools like dbt, Airflow, and Prefect.
  • Develop curated data models and products to support analytics, AI workloads, and operational use cases.
  • Collaborate with AI, Analytics, and Product teams to deliver high-quality, scalable data solutions.

RemoFirst offers a platform that provides people management tools and employee benefits, enabling companies to hire talent globally. It is one of the fastest-growing private companies in the USA, backed by $40M+ in venture funding, with a fast-paced startup culture.

US Unlimited PTO

  • Design, build, and maintain scalable data pipelines and services using Python and modern data engineering frameworks.
  • Architect and implement data solutions on cloud platforms (AWS, Azure, or GCP) and optimize database performance.
  • Serve as a technical leader, mentoring junior engineers and collaborating with cross-functional teams to deliver data-driven solutions.

Sagent powers servicers and consumers by transforming the mortgage servicing industry with modern technology. We're a growing joint venture combining Fiserv's fintech expertise with Warburg Pincus, hiring innovators to disrupt the lending space.

$151,000–$189,000/yr
US Unlimited PTO

  • Design, develop, deploy, and maintain production-grade data pipelines using Python, Spark, and Databricks.
  • Build and improve analytical data models that support financial decisions, product development, operations, marketing, and business growth.
  • Collaborate closely with Analysts, Data Engineers, and cross-functional teams to deliver high-quality self-service analytics capabilities.

$71,600–$105,000/yr
US

  • Build and maintain batch and streaming ETL/ELT pipelines that feed a cloud data warehouse.
  • Develop, schedule, and monitor cloud data jobs, diagnosing failures and tuning for cost and performance.
  • Write idempotent load logic that handles reprocessing and backfills without creating duplicate data.

Crum & Forster provides market leading property & casualty, accident & health, specialty and standard commercial lines insurance solutions. With 3,000 employees and an employee-first focus, the company is consistently recognized as a great place to work, earning multiple workplace and wellness awards.

US

  • Design and build data ingestion pipelines and knowledge bases using enterprise search and AI retrieval technologies.
  • Implement secure data access controls and retrieval evaluation frameworks to ensure accuracy and compliance.
  • Collaborate with architects and customer teams to align data strategies and enable long-term adoption of AI solutions.

A partner company specializing in enterprise AI and data solutions is seeking a Data / Search Specialist. The company operates with a focus on secure and scalable AI adoption, though its size and culture are not specified.

$100,000–$140,000/yr
US Unlimited PTO

  • Design, build, and maintain robust, scalable ELT/ETL data pipelines from various source systems into cloud data platforms.
  • Perform data modeling, including dimensional modeling, and build transformation layers using dbt to create analytics-ready datasets.
  • Support operational reliability, monitor data pipelines, and ensure SLAs for timeliness, freshness, and accuracy.

Troveo builds the data platform that AI labs and model builders need to train the next generation of models. Backed by top investors, we’re a small, high-impact team solving one of the biggest bottlenecks in AI development.

$107,950–$127,000/yr
UK 5w PTO

  • Design, build, and maintain scalable data pipelines using Python, SQL, Snowflake, Dagster, dbt, and AWS.
  • Own end-to-end data engineering projects from ingestion through to analytics enablement.
  • Improve monitoring, alerting, and data quality across key pipelines.

Midnite is a next-generation sports betting and gaming platform built for a new wave of players. Over 400,000 players have joined, and the team operates with high ownership and fast iteration in a scale-up environment.

US Unlimited PTO 12w maternity 12w paternity

  • Design, build, and own data pipelines moving data from application and third-party sources into relational databases.
  • Build and ship production AI systems, including data infrastructure and evaluation systems for feature reliability.
  • Set standards for data work through clear writing, early context sharing, and team efficiency.

Dscout builds a flexible UX research platform trusted by top brands in finance, healthcare, and tech. They are a remote-first team of passionate professionals that prioritize learning, diversity, and inclusion.

Brazil

  • Design, develop, and maintain scalable ETL/ELT pipelines and data integration processes.
  • Build and optimize cloud-based data architectures to support analytics and business intelligence initiatives.
  • Ensure data quality, consistency, governance, and reliability through validation, monitoring, and automated quality checks.

CI&T helps large enterprises transform AI potential into business impact with AI deployment, AI-native execution, and tech-integrated solutions. With 30 years of experience and 8,000 employees across 25 countries, they collaborate to build solutions with real impact.

$115,000–$175,000/yr
US Unlimited PTO

  • Own the Airflow codebase end-to-end, building reusable templates and enforcing standards.
  • Build and extend large-scale Spark pipelines on AWS Glue and support migration to Databricks.
  • Drive data model improvements around commercial pharma data with focus on structure and lineage.

Veeva Systems is a mission-driven organization and pioneer in industry cloud, helping life sciences companies bring therapies to patients faster. As one of the fastest-growing SaaS companies in history, it surpassed $3B in revenue and values doing the right thing, customer success, employee success, and speed.

Brazil

  • Design, build, and maintain scalable data pipelines for ingestion and transformation.
  • Work with Python, SQL, Apache Airflow, and Google Cloud Platform.
  • Collaborate with cross-functional teams to deliver reliable and high-quality data solutions.

Our partner is a technology company focused on AI and data solutions. The team values collaboration, innovation, and continuous learning, and offers a remote work environment.

$172,000–$242,000/yr
US Unlimited PTO

  • Own and scale distributed systems for orchestration, APIs, and data paths.
  • Apply query-engine expertise to optimize work execution and data movement.
  • Exercise strong technical judgment on ambiguous problems.

Prefect builds and operates resilient, Pythonic orchestration and MCP platforms used for mission-critical workloads. This remote-first company fosters a supportive, high-performance culture that empowers team members to do their best work.