Senior Data Engineer (Databricks)

We’re expanding our team! 🚀DS STREAM is growing! We are looking for SENIOR Data Engineers with solid Databricks experience. You will need at least one of these At least 5 years of commercial experience in Data Engineering with focus on large scale data pipelines. Proven track record using advanced solutions in Azure cloud and Databricks Proficiency in Python and SQL Ability to build pipelines using Github or similar CI/CD tools Nice to have Previous exposure to GCP infrastructure and tools (Airflow, Pub/Sub, BigQuery, VertexAI) Knowledge of Spark Structured Streaming Ability to build enterprise API integrations with FastAPI or similar Experience in leading/mentoring engineers [REMOTE] Senior Data Engineer We're expanding our AI Data Engineering Team at DS STREAM! This is a 100% remote role WITHIN POLAND. Project description: This initiative centers on a mature, high-availability data hub architected within Azure Databricks, utilizing Unity Catalog, Spark Structured Streaming, and Asset Bundles to ensure robust performance and governance. The solution is built upon the Medallion Architecture framework, providing a scalable foundation that is now being expanded to ingest, transform, and seamlessly integrate new data sources into the existing ecosystem. You will get 100% remote work with the possibility to use an office in Warsaw Flexible working hours and engaging company events Access to Azure, GCP and Databricks certifications Possibility to use an in-house incubator to develop your own ventures A supportive community of 50+ senior data consultants with extensive experience in Data Warehouse and Big Data and much more! ,[Development of a program for processing large volumes of data, Designing and implementing of new PySpark pipelines in Databricks, Ingesting data from APIs , Handling Spark Structure Streaming pipelines , Designing and managing objects in Unity Catalog according to Medallion Architecture, Webscraping] Requirements: Python, Azure, Azure Databricks, Docker, SQL, Databricks, PySpark, CI/CD, GCP, Airflow, BigQuery, API, FastAPI Tools: Agile, Scrum. Additionally: Remote work, Competitive compensation, Sport subscription, Training budget, Private healthcare, Flat structure, Small teams, International projects, Free coffee, Bike parking, In-house trainings, In-house hack days, Modern office, Startup atmosphere, No dress code, Free beverages.