Python & Apache Spark Engineer - Data Processing Specialist

Python & Apache Spark Engineer - Data Processing Specialist

Miejsce pracy: Warszawa

Technologies we use

Expected

• Python
• Azure DevOps
• Databricks
• Apache Hive
• Microsoft Fabric
• Spark
• Delta Tables
• Delta Lake

About the project

You will join a dynamic technology team responsible for designing, developing, and deploying innovative enterprise technology and AI-driven solutions supporting the delivery of tax services.

The team combines expertise across software engineering, data engineering, AI, tax technology, change management, and project management. The projects cover the full technology lifecycle — from solution design and development through deployment and optimization, to training, adoption, and stakeholder engagement.

You will work in a modern, cloud-based environment and contribute to the development of scalable data processing solutions, with a strong focus on Python, Apache Spark, cloud technologies, microservices, and modern data platforms.

This is how we organize our work

This is how we work

• at the client's site

Your responsibilities

• Design, develop, and maintain robust, scalable, and production-ready Apache Spark applications.
• Develop efficient data processing, transformation, and analysis workflows using Python and Spark.
• Build solutions capable of handling large-scale datasets while maintaining high performance and reliability.
• Work with columnar data formats, particularly Parquet, and implement data solutions using Delta Tables.
• Develop clean, maintainable, testable, and efficient code in accordance with established engineering standards.
• Optimize Spark applications, data processing pipelines, and code for performance, scalability, and efficient resource utilization.
• Analyze and resolve technical issues affecting application reliability, performance, and data processing.
• Contribute to the design and implementation of scalable microservices-based solutions.
• Define, promote, and enforce software engineering best practices and coding standards across the project.
• Conduct detailed code reviews, ensuring code quality, maintainability, security, performance, and compliance with project standards.
• Identify potential technical risks and proactively propose improvements.
• Support the team in solving complex technical and architectural problems.
• Provide technical guidance to other team members and help establish consistent development practices.
• Contribute to continuous improvement of development processes, workflows, and engineering standards.
• Work both independently and collaboratively within a distributed, cross-functional team.
• Cooperate closely with software engineers, data specialists, architects, project managers, and business stakeholders.
• Communicate clearly with team members, coordinate day-to-day technical activities, and help ensure smooth delivery.
• Maintain regular communication with the client and participate in technical discussions and project meetings.
• Translate business and technical requirements into practical and scalable software solutions.
• Take ownership of assigned tasks and proactively communicate progress, risks, dependencies, and potential blockers.
• Create and maintain comprehensive technical documentation covering code, data processing workflows, architecture, processes, and operational procedures.
• Ensure developed solutions are maintainable and understandable for other team members.
• Support the delivery of high-quality software solutions within agreed timelines.
• Balance standardized engineering processes with a pragmatic and flexible approach to project-specific requirements.

Our requirements

• 5+ years of hands-on experience in software development, preferably in data-intensive or enterprise environments.
• Strong commercial experience with Apache Spark and large-scale data processing.
• Practical experience with one or more modern Spark-based platforms, such as:
• Databricks
• Azure Synapse
• Microsoft Fabric
• Strong programming skills in Python.
• Experience with data manipulation and processing using Pandas, Polars, or similar libraries.
• Strong understanding of Parquet and other columnar data storage formats.
• Practical experience working with Delta Tables / Delta Lake.
• Proven experience designing and implementing data processing, transformation, and analytical workflows.
• Solid understanding of microservices architecture and its implementation in scalable enterprise systems.
• Experience writing clean, maintainable, efficient, and production-ready code.
• Understanding of software development best practices, coding standards, testing, and code review processes.
• Strong analytical and problem-solving capabilities.
• Detail-oriented approach with the ability to identify potential technical issues and bottlenecks.
• Practical and pragmatic mindset — able to balance standardized processes with the flexibility required to achieve project objectives.
• Strong organizational and time-management skills.
• Ability to independently manage workload, prioritize tasks, and meet tight deadlines.
• Excellent communication skills and the ability to explain technical topics clearly to both technical and non-technical stakeholders.
• Comfortable working independently while also being an active member of a collaborative, international team.
• Willingness to take ownership and proactively drive technical topics forward.

Optional

• Experience with Microsoft Azure or other major cloud platforms.
• Practical knowledge of Azure services, particularly:
• Azure Service Bus
• Azure Data Lake
• Azure Blob Storage
• Azure Redis
• Other Azure SaaS/PaaS services
• Experience building APIs and backend services with FastAPI.
• Hands-on experience with Docker.
• Experience with Kubernetes and container orchestration.
• Familiarity with modern AI/GenAI technologies, including LangChain, LangGraph, RAG pipelines, or LLM-based applications.
• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Experience working in enterprise environments with complex data processing requirements.

SQUARE ONE RESOURCES sp. z o.o.

At Square One Poland we link IT experts with the business. With over 25 years of experience, we specialize in recruitment processes on a global scale. Despite years of experience, we still have a startup DNA and this is our advantage. Our offices are located in London and Warsaw, however, we can reach clients from all over the world, from start-ups to big worldwide corporations.