Agent Quality, Evals Engineer

🕒 June 23

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

👷🏻‍♀️ Engineer

👻 Ghost score 63%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Softgic

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2011

💼 Consulting

🤝 B2B

🤖 Artificial Intelligence

Consulting • B2B • Artificial Intelligence

Softgic is a digital and cognitive transformation services company that provides custom software development, automation, cloud and cybersecurity, and emerging-technology solutions to business clients. With more than a decade of experience, Softgic offers specialized “labs” for Solutions Design (UX, Agile, process design), Build (coding, low-code, mobile, DevOps), Metaverse (VR/AR, blockchain, NFTs, virtual economy), Data (Big Data, BI, AI/ML), Automation (RPA, BPM) and Go Live (cloud, IT operations, cybersecurity). The firm emphasizes tailored, enterprise-focused services, dedicated teams, and project-based engagements to help clients modernize infrastructure, streamline processes, and adopt advanced technologies.

📋 Description

• Own the eval harness and quality gate from the beginning • Build and maintain the MVP eval harness, including golden tasks, exception tasks, scorecard metrics, and regression packs • Wire evaluations into CI so quality regressions fail builds and releases • Define and maintain release-gate thresholds with Product and the Tech Lead • Prepare for later adversarial and drift-testing expansion without overbuilding MVP scope • Use AI to generate candidate evaluation cases and failure hypotheses while validating generated tests • Establish the first reference agent's published scorecard and gated evaluation path • Automate golden and exception tests • Define measurable 'good enough to ship' criteria

🎯 Requirements

• Experience evaluating ML, LLM, or non-deterministic systems • Strong test and benchmark design capability • Comfort working with noisy metrics, thresholds, and probabilistic behavior • Good scripting and automation skills • Uses AI to generate candidate eval cases and failure hypotheses, while distinguishing generated tests from validated quality • Treats AI quality as an operating system rather than a QA afterthought

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume