Automation QA Engineer

Automation QA Engineer

Ciklum

Remote, Poland

About the role:

As a Automation QA Engineer, become a part of a cross-functional development team engineering experiences of tomorrow.

Responsibilities:

• Programmatic RAG Evaluation: Build and run automated validation test suites using Python frameworks like RAGAS to continuously score pipeline faithfulness, answer relevance, and context precision
• CI/CD Quality Gates: Integrate automated LLM evaluation scorecards directly into the client's Azure DevOps pipelines to prevent regression in code-generation quality
• Data Integrity Validation: Develop automated checks to monitor the performance of documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy
• Phase-Gate Reporting: Generate programmatic quality scorecards and regression reports to satisfy technical criteria at each project gate approval milestone
• Squad Collaboration: Work closely with senior and mid-level ML implementation engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by the validation metrics

Requirements:

• Core Experience Baseline: 3+ years of experience in backend test automation, data quality engineering, or Python software development within structured delivery environments
• Automation Engineering Stack: Mastery of Python for backend testing, scripting, and data manipulation. Traditional UI automation tools like Selenium or Cypress are not applicable for this scope
• AI Evaluation Frameworks: Direct, hands-on experience or deep technical familiarity with LLM validation frameworks (specifically RAGAS or directly equivalent automated validation engines)
• CI/CD Infrastructure: Practical experience configuring automated script execution, regression test triggers, and quality gates inside Git-based pipelines, preferably Azure DevOps
• Data & Search Familiarity: Basic functional understanding of relational databases (PostgreSQL), data profiling techniques, and search indexing concepts
• Location Constraint: Must be physically located in Poland or Eastern Europe to remain aligned with the embedded co-delivery squad