Senior AI Quality Engineer – LLM Evaluation, Automation

🕒 January 7

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Softgic

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2011

💼 Consulting

🤝 B2B

🤖 Artificial Intelligence

Consulting • B2B • Artificial Intelligence

Softgic is a digital and cognitive transformation services company that provides custom software development, automation, cloud and cybersecurity, and emerging-technology solutions to business clients. With more than a decade of experience, Softgic offers specialized “labs” for Solutions Design (UX, Agile, process design), Build (coding, low-code, mobile, DevOps), Metaverse (VR/AR, blockchain, NFTs, virtual economy), Data (Big Data, BI, AI/ML), Automation (RPA, BPM) and Go Live (cloud, IT operations, cybersecurity). The firm emphasizes tailored, enterprise-focused services, dedicated teams, and project-based engagements to help clients modernize infrastructure, streamline processes, and adopt advanced technologies.

📋 Description

• Build and maintain the MVP eval harness, including golden tasks, exception tasks, scorecard metrics, and regression packs • Integrate evaluations into CI so quality regressions fail builds and releases • Define and maintain release-gate thresholds with Product and the Tech Lead • Prepare for later adversarial and drift-testing expansion without overbuilding MVP scope • Use AI to generate candidate evaluation cases and failure hypotheses while validating generated tests • Establish measurable agent-quality standards and release gates • Publish the first reference agent scorecard and gated evaluation path • Automate golden and exception test execution • Define measurable criteria for when the product is good enough to ship

🎯 Requirements

• Experience evaluating ML, LLM, or non-deterministic systems • Strong test and benchmark design capability • Comfort working with noisy metrics, thresholds, and probabilistic behavior • Good scripting and automation skills • Uses AI to generate candidate evaluation cases and failure hypotheses, while distinguishing generated tests from validated quality • Approaches AI quality as an operating system rather than a QA afterthought