Lead Data Platform Engineer - Apache Iceberg & PySpark

Expected, PySpark, Apache Iceberg

Operating system, Linux

About the project, As a Lead Data Platform Engineer, you will be working for our client, a global financial services organization, dedicated to delivering innovative technology solutions for Investment Banking. You will play a pivotal role in shaping and optimizing data pipelines and storage strategies that drive strategic insights and operational excellence. Join us to lead transformative initiatives at the forefront of data engineering and unlock new levels of performance and efficiency., , Ignite data innovation — master the future of large-scale analytics with cutting-edge technologies!, , Krakow-based opportunity with hybrid work model., , Only candidates with an existing legal right to work in Europe will be considered for this role.

This is how we work, agile, scrum, kanban

Your responsibilities, Lead the design, development, and implementation of data pipelines and table management workflows using PySpark and Apache Iceberg., Drive performance and cost optimization for Apache Iceberg by refining partitioning strategies, file sizing, and query efficiency., Manage and improve Apache Iceberg maintenance operations, including snapshot management, compaction, and metadata cleanup., Troubleshoot platform issues by applying deep understanding of Apache Iceberg internals such as metadata layout, schemas, and schema evolution., Promote engineering excellence through code reviews, automation, and best practices in CI/CD and observability., Support operational reliability by participating in on-call duties, incident response, and root cause analysis., Collaborate with cross-functional teams, influence architectural decisions, and mentor less experienced engineers., Partner with stakeholders to ensure solutions are secure, scalable, and compliant.

7+ years of experience in data engineering or related roles., Strong hands-on expertise in PySpark, including DataFrame APIs, performance tuning, and job optimization., Practical experience with Apache Iceberg, especially in optimizing file layout, partitioning, and understanding internal architecture., Proven ability to automate maintenance operations and operate production data workloads independently., Solid software engineering fundamentals with a focus on reliable, maintainable systems., Excellent communication skills in English and the ability to influence across geographies., Ownership mindset with a focus on end-to-end delivery, discipline, and initiative.

Optional, Certifications or additional skills in Big Data, Cloud platforms, or related fields.

What we offer, Stable and long-term cooperation with very good conditions, Enhance your skills and develop your expertise in the financial industry, Work on the most strategic projects available in the market, Define your career roadmap and develop yourself in the best and fastest possible way by delivering strategic projects for different clients of ITDS over several years, Participate in Social Events, training, and work in an international environment, Access to attractive Medical Package, Access to Multisport Program, Access to Pluralsight, Flexible hours

Benefits, sharing the costs of sports activities, private medical care, flexible working time, fruits, integration events, corporate gym, saving & investment scheme, no dress code, coffee / tea, drinks, christmas gifts, birthday celebration, sharing the costs of a streaming platform subscription, access to +100 projects, access to Pluralsight

Recruitment stages, online interview, online interview

#GETREADY to meet with us!, ITDS Business Consultants is involved in many various, innovative and professional IT projects for international companies in the financial industry in Europe. We offer an environment for professional, ambitious, and driven people., , We would like to meet you. If you are interested please apply and attach your CV in English or Polish, including a statement that you agree to our processing and storing of your personal data.

ITDS’s Whistleblower Procedure, You can report violations in accordance with ITDS’s Whistleblower Procedure available here:

ITDS Polska Sp. z o.o., ITDS supports financial service providers to take the next steps., , We identify what’s possible, every day. Opportunities in the areas of technology, organization, and digitization., , We see where banks, insurers, payment companies, or fintech can go and how they can get there., , That’s why we want to stimulate you to ramp up your ambition. Forget what you perceive as restraints and step towards the new reality., , ITDS in Poland - Pure player in new Technologies & Financial Industry, +300 IT implementation professionals, +20 clients in the Banking, Insurance, Payment & Fintech Industry, , ITDS excels in digital strategy delivery and implementation of best-of-breed lending solutions. ITDS has delivered successful strategic projects throughout Europe since 1998. We combine the experience we’ve accumulated with in-depth knowledge of technologies, business processes, and EU legislation to unlock new business opportunities.

This is how we work,