Senior AI Quality Engineer – LLM Evaluation, Automation

🕒 January 7

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Softgic

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2011

🔒 Cybersecurity

🤖 Artificial Intelligence

Cloud • Cybersecurity • Artificial Intelligence

Softgic is a technology company focused on driving digital and cognitive transformation for its clients. With over 12 years of experience, Softgic provides innovative and tailored solutions across various areas, including software development, cloud data security, the metaverse, and automation. Their approach combines design, coding, and implementation to enhance business processes and ensure security and efficiency in technological infrastructure.

📋 Description

• Owns the eval harness and quality gate from the beginning. • Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs. • Wire evals into CI so quality regressions fail builds and releases. • Define and maintain release-gate thresholds with Product and the Tech Lead. • Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.

🎯 Requirements

• Experience evaluating ML, LLM, or non-deterministic systems. • Strong test and benchmark design capability. • Comfort working with noisy metrics, thresholds, and probabilistic behavior. • Good scripting and automation skills.

🏖️ Benefits

• Professional development opportunities