AI Engineer – Agent Evaluation Specialist

Job not on LinkedIn

🔥 1 minute ago

🏄 California – Remote

🏄 California – Remote

⏳ Contract/Temporary

🟠 Senior

🔴 Lead

🤖 AI Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 10%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

TMS, LLC

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2009

💼 Consulting

🏛️ Government

Consulting • Government

TMS, LLC is a technology management and advisory firm (Technology Management Solutions) that provides IT advisory and consulting, program and project management, financial management, and organizational assessment services. The company works with U. S. federal government clients — including DHS OCIO, FEMA, the National Institute of Mental Health OCIO, and TSA — to improve mission performance, reduce costs, and modernize operations. TMS emphasizes experienced staff, proven processes, mentoring and exercises for system readiness, and cost allocation/charge-back methodologies for transparency and efficiency.

📋 Description

• Evaluate LLM-powered agents as the primary focus of the role • Design, implement, and run LLM-as-a-Judge evaluation frameworks • Assess agent response quality, accuracy, relevance, and reliability • Collaborate with Business Analysts and stakeholders to create, validate, and maintain golden datasets • Instrument AI agents with Langfuse • Trace agent execution, capture evaluation scores, monitor performance, and troubleshoot failures • Test agent behavior, tool usage, reasoning workflows, and response quality

🎯 Requirements

• 14+ years of experience • Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability • Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets for agent evaluation • Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures • Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality • Actual implementation experience with agent evaluation frameworks; theoretical knowledge or general GenAI/LLM development alone is insufficient

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume