SDET/QA Engineer, AI Evaluation

🔥 15 hours ago

🇪🇸 Spain – Remote

💵 €38k - €43k / year

⏰ Full Time

🟡 Mid-level

🟠 Senior

🔧 QA Engineer (Quality Assurance)

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Codurance

WebsiteLinkedInAll Job Openings

51 - 200 employees

💼 Consulting

📣 Marketing

☁️ SaaS

Consulting • Marketing • SaaS

Codurance is a global software consultancy firm focused on helping businesses modernize their technical capabilities and build sustainable growth. With a dedication to Software Craftsmanship, Codurance emphasizes professionalism and technical excellence in software development. The company offers a range of services, including software modernization, product development, and cloud engineering, tailored to help clients solve complex challenges and achieve their business goals. Codurance partners with leading technology teams and utilizes agile and lean processes to ensure efficient and impactful delivery.

📋 Description

• Evaluate, validate, and improve quality across teams • Participate in developing and executing exploratory and automated tests • Review system requirements and track quality assurance metrics such as pass rate, defect densities, and open defect counts • Liaise with technical and business stakeholders from Codurance and client teams • Take ownership of a Python LLM evaluation tool, diagnostic suite, and golden datasets to raise pass rates on production AI agents • Refine scoring criteria and design multi-turn conversational evaluations for AI agents • Set up automated regression testing for core product flows using Playwright, covering integrated workflows and third-party channels • Implement SonarQube quality gate monitoring and daily deviation reporting • Partner with senior stakeholders on QA strategy, standards, and continuous improvement • Estimate, prioritize, plan, and coordinate testing activities • Stay up to date with new testing tools and test strategies

🎯 Requirements

• At least 4 years of experience in a similar role • Fluent in Spanish and English (speaking and writing) • Consultancy experience and an open mind around what is quality (adapting to clients) • Hands-on experience with Playwright, SonarQube and other testing tools (Cucumber, Selenium, Appium, Jenkins) • Knowledge of CI/CD tools and processes • Continuous learning and curiosity • Strong programming skills with Python • Practical experience in AI evaluation, including LLM evals, agent testing, multi-turn chat scoring, and golden dataset management • QA strategy knowledge (holistic testing, Agile testing Manifesto, ...) (bonus) • Anthropic certification (bonus) • Experience with performance/load testing for AI agents/call center agents and multi-channel integration (bonus) • Familiarity with BDD and TDD (bonus) • ISTQB certificate (bonus) • Eligible to work in the EU; visa sponsorship and relocation assistance are not provided

🏖️ Benefits

• Transparency - all of our salary bandings and company finances are available to everyone from day one. • Personal Learning Budget - Access to Learnerbly, our online learning & development tool, to support your personal development and career progression here. • Training - No fixed training budget. So long as the course is relevant for the company and you'll do a lightning talk on the subject, we’re happy to pay for it. • Working 100% remote (or hybrid if you want to drop by the office from time to time) • Private medical and dental insurance • Restaurant, transport and nursery tickets (retribución flexible/ tax deductible) • Gym discounts • Language classes (Spanish, English, French, German) • You choose between Linux, Mac or Windows! • Our People - You’ll be working alongside Craftspeople whether that’s on a client project or contributing to our internal projects. • Equal Opportunities Employer and inclusive workplace