Senior AI Quality, Evaluation Engineer

Job not on LinkedIn

🔥 5 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 11%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 48 days.
• No salary listed.

Apply Now

Customize resume + cover letter

Find Similar Remote Jobs

☆ Save☑️ Mark as applied❌ Hide

Report problem

Solvd, Inc.

WebsiteLinkedInAll Job Openings

501 - 1000 employees

Founded 2010

💼 Consulting

🏥 Healthcare

📦 Logistics

Consulting • Healthcare • Logistics

Solvd, Inc. is a global software engineering and consulting company that delivers comprehensive end-to-end solutions. Founded in 2011, Solvd offers core engineering services, including software development for web and mobile platforms, digital experience and design, data and AI/ML solutions, and cloud platform modernization. The company is an AWS partner and is known for optimizing resources, solving complex problems, and enhancing user satisfaction. Solvd emphasizes innovation and quality assurance, employing over 800 international professionals across 8 offices worldwide. The company prides itself on transforming businesses by providing feature-rich digital products, manual testing, automation, and employing cutting-edge technology solutions. Their proprietary tools like Zebrunner and Carina aid in improving development and QA processes. Solvd operates with a focus on collaboration, top talent cultivation, and customized technical solutions tailored to individual client needs, serving clients across 15 countries.

📋 Description

• Own independent proof that retrieval works and permissions hold • Build the golden question suite with expected sources, passages, and acceptance thresholds for relevance, grounding, and citation accuracy • Build negative entitlement tests that verify restricted passages are not returned and raise severity-one alarms on failures • Build freshness and retirement tests for indexing lag and removal of retired items • Automate the test harness to run on schedule and after content, configuration, or model changes • Publish scores to the quality dashboard with trend lines and regression alarms • Define the QA gate content analysts apply to remediated content • Run the suite before pilot changes in weeks 7–9 and report misses with root causes to the appropriate owner • Write the evaluation runbook and train client stewards and platform owners

🎯 Requirements

• 3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule • Hands-on evaluation of a search or RAG system in production, including writing ground truth sets, scoring relevance and grounding, and tracking drift over time • Experience designing security-focused negative tests for authorization, entitlement, and data exposure • Ability to work in AWS, invoking Bedrock and OpenSearch APIs, reading CloudWatch, and deploying a scheduled job with infrastructure code • Clear written reporting for client stewards and engineers • AWS Certified AI Practitioner or Machine Learning Engineer Associate (preferred) • Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration (preferred) • Contact center or knowledge management domain exposure (preferred) • Familiarity with SOC 2 or ISO 27001 evidence practices (preferred) • Your level of English

🏖️ Benefits

• Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation • Global team with collaboration across continents and cultures • Inclusive environment prioritizing continuous learning, innovation, and ethical AI standards

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume