Senior Software Engineer - Crawler

Responsibilities

• Design and implement scalable, fault-tolerant web crawling and extraction pipeline components.
• Write clean, production-grade Java and Python code.
• Build and operate large-scale ETL/ELT pipelines for data extraction and transformation.
• Work with GCP and AWS cloud infrastructure, primarily on GKE.
• Improve observability, reliability, and operational excellence across contributed systems.
• Partner with product and data science teams to deliver solutions.
• Contribute to code reviews, documentation, and knowledge sharing.
• Keep current with web technologies, anti-crawling mechanisms, and AI-powered extraction approaches.

Requirements

• At least 5 years of professional software engineering experience building production systems.
• Strong knowledge of algorithms, data structures, concurrency, and distributed systems.
• Proficiency in Java and/or Python.
• Experience owning features end-to-end from design through deployment and operation.
• Experience making component-level architectural decisions.
• Hands-on experience with cloud data warehouses such as BigQuery or Snowflake.
• Experience designing and operating large-scale ETL/ELT pipelines.
• Experience with orchestration tools such as Apache Airflow.
• Experience with streaming or event-driven systems such as Apache Kafka.
• Production experience on GCP or AWS, with GCP preferred.
• Hands-on experience with Kubernetes, including GKE or EKS, for distributed workloads.
• Familiarity with infrastructure-as-code tools such as Terraform.
• Strong communication, pragmatic problem-solving, adaptability, and self-direction.
• Web crawling at scale, Scrapy, proxy infrastructure, anti-bot evasion, SERP extraction, AI/LLM-based extraction, or B2B data-company experience is preferred.

Benefits

• Remote work arrangement is indicated by the LI-REMOTE designation.
• Comprehensive benefits are offered, along with holistic mind, body, and lifestyle well-being programs.