Senior AI Quality, Evaluation Engineer

πŸ•’ 6 days ago

πŸ‡ΊπŸ‡Έ United States – Remote

⏰ Full Time

🟑 Mid-level

🟠 Senior

πŸ€– Artificial Intelligence

Where this job comes from

Posted on Solvd, Inc.'s careers page β†—Solvd, Inc.'s careers page, not through a job board

First seen by Remote Rocketship 6 days ago

Still open on Solvd, Inc.'s site. Checked 1 hour ago.

Apply Now
Logo of Solvd, Inc.

Solvd, Inc.

501 - 1000 employees

Founded 2010

πŸ’Ό Consulting

πŸ₯ Healthcare

πŸ“¦ Logistics

Consulting β€’ Healthcare β€’ Logistics

Solvd, Inc. is a global software engineering and consulting company that delivers comprehensive end-to-end solutions. Founded in 2011, Solvd offers core engineering services, including software development for web and mobile platforms, digital experience and design, data and AI/ML solutions, and cloud platform modernization. The company is an AWS partner and is known for optimizing resources, solving complex problems, and enhancing user satisfaction. Solvd emphasizes innovation and quality assurance, employing over 800 international professionals across 8 offices worldwide. The company prides itself on transforming businesses by providing feature-rich digital products, manual testing, automation, and employing cutting-edge technology solutions. Their proprietary tools like Zebrunner and Carina aid in improving development and QA processes. Solvd operates with a focus on collaboration, top talent cultivation, and customized technical solutions tailored to individual client needs, serving clients across 15 countries.

πŸ“‹ Description

β€’ Own independent proof that retrieval works and permissions hold β€’ Build the golden question suite with expected sources, passages, and acceptance thresholds for relevance, grounding, and citation accuracy β€’ Build negative entitlement tests that verify restricted passages are not returned and raise severity-one alarms on failures β€’ Build freshness and retirement tests for indexing lag and removal of retired items β€’ Automate the test harness to run on schedule and after content, configuration, or model changes β€’ Publish scores to the quality dashboard with trend lines and regression alarms β€’ Define the QA gate content analysts apply to remediated content β€’ Run the suite before pilot changes in weeks 7–9 and report misses with root causes to the appropriate owner β€’ Write the evaluation runbook and train client stewards and platform owners

🎯 Requirements

β€’ 3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule β€’ Hands-on evaluation of a search or RAG system in production, including writing ground truth sets, scoring relevance and grounding, and tracking drift over time β€’ Experience designing security-focused negative tests for authorization, entitlement, and data exposure β€’ Ability to work in AWS, invoking Bedrock and OpenSearch APIs, reading CloudWatch, and deploying a scheduled job with infrastructure code β€’ Clear written reporting for client stewards and engineers β€’ AWS Certified AI Practitioner or Machine Learning Engineer Associate (preferred) β€’ Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration (preferred) β€’ Contact center or knowledge management domain exposure (preferred) β€’ Familiarity with SOC 2 or ISO 27001 evidence practices (preferred) β€’ Your level of English

πŸ–οΈ Benefits

β€’ Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation β€’ Global team with collaboration across continents and cultures β€’ Inclusive environment prioritizing continuous learning, innovation, and ethical AI standards

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ 6 days ago

Generator Health

51 - 200

πŸ₯ Healthcare

πŸ’Ό Consulting

πŸ“¦ Logistics

Clinical AI Specialist auditing prior authorizations and AI-generated clinical outputs at Generator Health. Improving clinical quality, payer alignment, and review workflows.

πŸ•’ November 12

GE HealthCare

10,000+ employees

πŸ₯ Healthcare

πŸ’Š Pharmaceuticals

Digital and AI Change Management Lead driving adoption, readiness, and sustainment for GE HealthCare transformation programs. Translating technology and process changes into employee capabilities and behaviors.

πŸ•’ November 12

CDW

10,000+ employees

πŸ’Ό Consulting

πŸ₯ Healthcare

πŸ“š Education

AI consulting engineer designing and deploying secure enterprise AI and MLOps solutions for CDW customers. Integrating applications, APIs, NVIDIA technologies, and data platforms.

πŸ•’ November 12

Bask Health

11 - 50

πŸ₯ Healthcare

πŸ’Ό Consulting

πŸ“¦ Logistics

AI Evaluation Engineer building automated evaluations for Bask Health’s AI-powered telehealth platform. Testing LLM outputs, retrieval, tool calling, permissions, and healthcare safety.

πŸ•’ November 12

Next Step Systems - National IT Recruiting Firm

1 - 10

🀝 B2B

πŸ‘₯ HR Tech

Desktop Engineer building cross-platform Electron applications for AI-powered messaging and collaboration. Developing React interfaces, real-time integrations, and desktop performance features for US-based remote work.

Built byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok