Senior Automation Test Engineer - AI Systems (f/m/x)

Expected, Python, JavaScript, TypeScript, Node.js, Angular

About the project, We are looking for a Senior QA Engineer to join our team and contribute to building an AI-powered operating system for healthcare marketing. In this role, you will go beyond traditional end-to-end testing, owning quality, evaluation, and monitoring for complex AI systems. You will work on real-world production-grade products that create monetizable value, ensuring the stability of our full-stack architecture while designing and scaling AI evaluation pipelines and ML drift monitoring systems. If you are passionate about developing robust test and evaluation frameworks for modern systems powered by AI and ML, we want to hear from you.

Your responsibilities, Design, build, and maintain evaluation frameworks for Large Language Models (LLMs) and custom agentic harnesses, Develop automated scoring mechanisms, assertions, and test suites to measure agent accuracy, reasoning reliability, hallucination rates, and tool-use correctness, Simulate complex, multi-step agent workflows to uncover edge cases in autonomous decision-making, Collaborate with engineering teams to set up robust monitoring frameworks for data drift, concept drift, and model performance degradation in production, Implement telemetry and alerting systems to track latency, token usage, cost efficiency, and output quality over time, Own the end-to-end testing strategy across our entire stack, ensuring seamless integration between the Angular frontend, Node.js backend services, and Python-based AI microservices, Write clean, maintainable, and scalable automated test scripts in Python and JavaScript/TypeScript, Perform rigorous API testing, integration testing, and performance/load testing for high-throughput microservices, Integrate AI evaluation and testing harnesses into our CI/CD pipelines to ensure continuous quality gate enforcement before deployment, Define and track key quality metrics (KPIs) for software reliability and AI model behavior

Minimum 5 years of experience in software testing or quality engineering, with at least 2 years focused on testing or evaluating AI/ML applications, LLM workflows, or agentic systems, Proficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems, Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow, and the ability to select appropriate metrics for classification or generation-quality problems, Familiarity with ML drift monitoring concepts, metrics tracking, and observability tools like Datadog or Grafana, Strong system-level design skills and the ability to build production-grade test and evaluation infrastructure from the ground up, Advanced level of English

Optional, Understanding of agentic design patterns and their decomposition into testable units, Experience working under the regulatory conditions of healthcare systems

What we offer, Great Place to Work since 2015 - it’s thanks to feedback from our workers that we get this special title and constantly implement new ideas, Employment stability - revenue of PLN 2.1BN, no debts, since 2006 on the market, We share the profit with Workers - over PLN 76M has already been allocated for this aim since 2022, Attractive benefits package - private healthcare, benefits cafeteria platform, car discounts and more, Comfortable workplace – class A offices or remote work, Dozens of fascinating projects for prestigious brands from all over the world – you can change them thanks to Job Changer application, PLN 1 000 000 per year for your ideas - with this amount, we support the passions and voluntary actions of our workers, Investment in your growth – meetups, webinars, training platform and technology blog – you choose, Fantastic atmosphere created by all Sii Power People

Recruitment stages, Send your CV, Talk to us about your expectations, Learn more about our projects and choose the best, Start your adventure with Sii!

Sii Sp. z o.o., Sii is the leading technology consulting, digital transformation, engineering, and business services vendor in Poland. We already employ more than 7 300 professionals and implement projects in a variety of industries for clients from many countries around the world. The Great Place to Work title, won 10 times in a row, proves that at Sii, we create a friendly work environment., , Sii ensures that all employment-related decisions are made solely on the basis of qualifications and competencies. We are committed to equal treatment of all individuals, regardless of any protected characteristics, and promoting a diverse and inclusive work environment.

This is how we work,