Quality Engineering Consultant – AI/ML Testing

Job not on LinkedIn

🕒 September 2

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🔧 QA Engineer (Quality Assurance)

👻 Ghost score 35%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

FindErnest

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2022

💼 Consulting

🏢 Enterprise

🤝 B2B

💰 $20k Pre Seed Round - FindErnest on 2023-01

Consulting • Enterprise • B2B

FindErnest is a global technology consulting and services company that helps enterprises, GCCs, startups, and SMBs accelerate digital transformation through a combination of technology consulting, product/platform engineering, managed IT services, AI & intelligent automation, cloud solutions, cybersecurity, enterprise applications, and workforce/talent solutions. Founded in 2022, the firm emphasizes integrating strategy and execution—combining advisory, implementation, managed services, and staffing under one roof—to deliver measurable business outcomes quickly and cost-effectively. Key offerings include digital transformation, product development, platform engineering, managed IT, AI automation, cloud and data services, and talent solutions; industries served span Technology & Internet, Healthcare & Life Sciences, BFSI, TME, Engineering & Construction, Supply Chain & Manufacturing, and Energy & Utilities. FindErnest positions itself as a mid-market, agile alternative to large SIs and pure staffing firms by delivering end-to-end transformation with a security-first, outcome-driven approach.

📋 Description

• Design and execute test strategies for AI/GenAI applications, including LLM-powered solutions, AI agents, copilots, and conversational systems • Validate AI outputs for accuracy, relevance, groundedness, consistency, hallucination detection, and toxicity and safety compliance • Create and maintain prompt test suites and benchmark datasets • Perform functional, regression, performance, and reliability testing for AI-enabled applications • Define and execute AI evaluation metrics and quality gates • Develop and maintain Playwright test automation frameworks • Automate UI, API, and end-to-end business workflows • Create reusable automation assets and testing accelerators • Integrate automated tests into CI/CD pipelines • Support shift-left testing and quality engineering practices • Design and execute API test cases using Postman, REST Assured, and Playwright API Testing • Validate API contracts, authentication, authorization, error handling, and performance • Conduct integration testing across distributed systems and third-party services • Establish AI evaluation frameworks and quality scorecards • Measure model quality using precision/recall, relevancy scoring, semantic similarity, groundedness validation, and human-in-the-loop evaluation • Analyze AI quality trends and recommend improvements • Support Responsible AI and model governance requirements • Collaborate with engineering, product, and AI teams to identify quality risks early • Participate in requirement reviews and design discussions • Communicate testing progress, risks, and quality metrics to stakeholders • Contribute to QE best practices, standards, and reusable playbooks

🎯 Requirements

• 4 - 6 years of experience in Quality Engineering, Software Testing, or Test Automation • Hands-on experience testing AI/Generative AI applications • Experience with modern web application testing and API testing • Strong expertise in Playwright automation • Hands-on experience with API testing and automation • Experience validating AI/LLM-based applications and AI agents • Good understanding of AI evaluation methodologies and testing techniques • Familiarity with M365 Copilot, OpenAI/Azure OpenAI, Claude, Gemini, and Agentic AI frameworks • Knowledge of test management tools such as Azure DevOps (ADO), Jira, or similar platforms • Experience working with CI/CD pipelines and DevOps practices • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline

🏖️ Benefits

• Remote work arrangement