AI SDET – Software Development Engineer in Test

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

CI&T

WebsiteLinkedInAll Job Openings

5001 - 10000 employees

Founded 1995

💼 Consulting

🏥 Healthcare

📣 Marketing

💰 $5.5M Venture Round on 2014-04

Consulting • Healthcare • Marketing

CI&T is a global tech transformation specialist focusing on helping organizations navigate their technology journey. With services spanning from application modernization and cloud solutions to AI-driven data analytics and customer experience, CI&T empowers businesses to accelerate their growth and maximize operational efficiency. The company emphasizes digital product design, strategy consulting, and immersive experiences, ensuring a robust support system for enterprises in various industries.

📋 Description

• Own and manage the Langfuse evaluation harness, including creation of "Golden Paths" and adversarial test scenarios • Validate regulatory quality gates (SR 26-2, NYDFS Reg 187) • Run shadow mode validation to compare AI performance against human benchmarks • Test for data quality degradation, population segmentation errors, and human-in-the-loop (HITL) checkpoint reliability • Verify that the Designer’s output standards, including explanation blocks and signal types, are maintained in all evaluation scenarios • Find edge cases through skeptical and adversarial testing • Collaborate with engineers to fix logic failures

🎯 Requirements

• Advanced proficiency in Python 3.11+, Pytest, and automated testing frameworks • Direct experience with AI evaluation tools such as Langfuse, RAGAS, or similar • Experience with LLM-as-a-judge patterns • Knowledge of model risk management and regulatory compliance testing • Experience with shadow mode deployment strategies and statistical validation • Skeptical and adversarial mindset to find edge cases • Uncompromising attention to detail regarding regulatory evidence • Collaborative approach to working with engineers to fix logic failures

🏖️ Benefits

• Health and dental insurance • Meal and food allowance • Childcare assistance • Extended paternity leave • Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass; • Profit Sharing and Results Participation (PLR); • Life insurance • Continuous learning platform (CI&T University); • Discount club • Free online platform dedicated to physical, mental, and overall well-being • Pregnancy and responsible parenting course • Partnerships with online learning platforms • Language learning platform