Senior Data Engineer - Python / Spark / AWS & GCP

Senior Data Engineer – Python / Spark / AWS & GCP

Miejsce pracy: Warszawa

Technologies we use

Expected

• Python
• Java
• Scala
• SQL
• Airflow
• AWS
• Spark
• Google Cloud Platform

About the project

We are looking for an experienced Senior Data Engineer / Data Platform Engineer to join an international Data team within the e-commerce and digital marketplace industry.

The team is responsible for building next-generation, cloud-based data solutions capable of ingesting and processing petabytes of data across data lakes and data warehouses.

This is how we organize our work

This is how we work

• at the client's site

Your responsibilities

• Design and develop high-volume batch and streaming data ingestion pipelines across AWS and GCP.
• Build and launch next-generation data ingestion and data curation platforms.
• Participate in system and data architecture discussions.
• Design scalable and high-performance distributed data solutions.
• Troubleshoot and resolve issues related to ETL, data lineage, data quality, backfills, and data pipelines.
• Design, implement, and support production services with strict SLAs.
• Collaborate with Software Engineers, Data Engineers, ML Engineers, Data Analysts, and other stakeholders.
• Provide technical guidance and mentor junior engineers.
• Contribute to best practices across software development, data engineering, testing, and CI/CD.
• Work with logging, monitoring, metrics, and alerting solutions to ensure platform reliability.

Our requirements

• 10+ years of experience in Data Engineering, Software Engineering, Distributed Systems, or a related field.
• Strong programming skills in Python, Java, or Scala.
• Strong knowledge of SQL and experience with SQL and NoSQL databases such as BigQuery, Teradata, MySQL, PostgreSQL, or Cassandra.
• Experience with Big Data technologies, including Hadoop, Hive, and Apache Spark.
• Hands-on experience with Airflow or another orchestration tool.
• Strong understanding of ETL, data lineage, data quality, backfills, and data pipeline troubleshooting.
• Experience developing both batch and streaming data pipelines.
• Experience across both software development and data engineering.
• Experience with AWS or GCP, particularly data processing at scale.
• Experience designing and supporting production services with strict SLAs.
• Experience with distributed applications and monitoring/logging tools such as Elasticsearch and Wavefront.
• Good understanding of testing methodologies, CI/CD, and software engineering best practices.
• Excellent written and verbal communication skills.

Optional

• Scrum Master experience.Strong expertise in Python.
• Experience with Google Data Streams and Google Dataproc.
• Experience with Change Data Capture (CDC) technologies.
• Experience with modern data warehouse technologies such as Delta Lake.

What we offer

• Full remote work
• Long-term cooperation
• B2B contract

Benefits

• sharing the costs of sports activities
• private medical care

Recruitment stages

• Interview with the client
• Final decision

SQUARE ONE RESOURCES sp. z o.o.

At Square One Poland we link IT experts with the business. With over 25 years of experience, we specialize in recruitment processes on a global scale. Despite years of experience, we still have a startup DNA and this is our advantage. Our offices are located in London and Warsaw, however, we can reach clients from all over the world, from start-ups to big worldwide corporations.