Source Job

Ireland

  • Design and scale web scraping infrastructure for collecting millions of social media records daily.
  • Build fault-tolerant pipelines using Python/Go, browser automation, and anti-bot bypass techniques.
  • Manage distributed queues, storage lakes, and monitoring for reliable data delivery at scale.

Python Go Web Scraping Distributed Systems SQL

6 jobs similar to Senior Data Infrastructure Engineer (Scraping & Scale)

Jobs ranked by similarity.

UK Unlimited PTO 26w maternity 9w paternity

  • Design and maintain large-scale Python scraping systems against Google and other highly defended websites.
  • Build scrapers for dynamic, JavaScript-heavy pages using browser automation and hybrid approaches.
  • Implement proxy, fingerprinting, and request-routing strategies to maximize stability and throughput.

We are an award-winning search intelligence platform powered by patented AI technology, helping brands and agencies dominate paid search with insights like Whole Market View and Smart Monitor. We are a scaling tech organization backed by Updata Partners, with a remote-first culture, a 93% approval rating, and an eNPS of 27.

India

  • Design and build production-grade Python scrapers and scalable ingestion pipelines for web-data acquisition.
  • Own the full engineering lifecycle from investigation to deployment and production support.
  • Work independently while collaborating with a distributed team through code reviews and structured workflows.

The company specializes in data engineering and web-data acquisition for research and data products. It embraces a remote-first culture with a distributed, collaborative engineering team.

Spain Unlimited PTO

  • Design and develop high-performance backend systems in Rust that collect billions of data points from the web.
  • Work on our in-house proxy layer, browser tooling, and protocol-level behavior to handle close to 200 million requests a day.
  • Collaborate with crawler engineers, DevOps, and Security to build secure, scalable, and reliable crawling infrastructure.

Lighthouse is on a mission to revolutionize commercial strategy as hospitality's most powerful AI platform, transforming the world's largest hotel data network into real-time intelligence. With 850+ teammates across 35 countries and $370M in Series C funding, we foster a collaborative, customer-centric culture focused on innovation and growth.

  • Own and continuously improve the systems for collecting job opportunities from thousands of companies.
  • Design scalable data architectures and operate reliable ETL/ELT pipelines.
  • Collaborate with product and AI/ML teams to enhance matching quality and data foundations.

Jobgether is building a job search platform for experienced professionals in the global remote job market. They aggregate and enrich job data from around the world. We are a fully remote, international team focused on accountability and outcomes.

US

  • Build and maintain data scrapers and ingestion pipelines for government data sources across the US
  • Validate data for accuracy, completeness, consistency, and reliability before it reaches the platform
  • Document source behavior, extraction logic, and data-quality checks to ensure maintainable systems

GovSignals is the AI system of work for government contracting, helping close the gap between government acquisition speed and adversary capability deployment. It's a high-velocity startup managing government contract data with AI in both FedRAMP High and DoW Impact Level 5 environments, monitoring over 5,000 live data sources and serving customers from small contractors to Fortune 500 primes.

US

  • Build and maintain data scrapers and ingestion pipelines for government data sources across the US.
  • Research new data sources using internal tooling, AI agents, and technical investigation.
  • Validate incoming data for accuracy, completeness, and reliability before it reaches production systems.

A partner company building a high-impact public-sector technology platform that relies on reliable government data. It operates as a fast-moving startup with a mission-driven culture focused on data quality and technical innovation.