AI Evaluation Engineer

Job not on LinkedIn

🔥 0 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 25%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 60 days.
• 102 open roles for a team of 16.
• No salary listed.

Apply Now

Find Similar Remote Jobs

Bask Health

WebsiteLinkedInAll Job Openings

11 - 50 employees

🏥 Healthcare

💼 Consulting

📦 Logistics

💰 Pre Seed Round on 2022-05

Healthcare • Consulting • Logistics

Bask Health is a robust platform providing a comprehensive suite of customizable telehealth solutions. It offers tools such as a questionnaire builder, patient portals, and pharmacy fulfillment services, all secured with strong encryption and HIPAA-compliance. Bask Health seamlessly integrates telehealth and e-commerce, enabling streamlined patient care from consultation to delivery. The platform supports entrepreneurs, doctors, and developers by offering scalable no-code solutions that enhance user experience and business growth. With its nationwide pharmacy network, Bask Health empowers businesses to efficiently manage prescriptions, patient data, and order fulfillment across the entire digital health landscape.

📋 Description

• Design and implement evaluation workflows for AI-powered applications across Bask Health. • Build automated tests for large language model outputs and AI application behavior. • Develop evaluation datasets representing realistic user requests and product workflows. • Create test scenarios for patient-facing, merchant-facing, and internal AI features. • Implement checks for response accuracy, relevance, completeness, and consistency. • Evaluate AI-generated content for unsupported claims and misleading information. • Test structured model outputs for schema compliance and application compatibility. • Build evaluations for AI tool calling and authorized action execution. • Validate that AI assistants respect application permissions and tenant boundaries. • Test retrieval-augmented generation systems for grounding and information relevance. • Develop regression tests identifying unexpected changes in AI behavior. • Integrate evaluation checks into engineering development and release workflows. • Create automated comparison tests for prompts, model versions, and configuration changes. • Develop scripts and utilities simplifying AI evaluation execution. • Build reporting tools summarizing evaluation results and failure patterns. • Investigate failed evaluations and help engineers reproduce underlying issues. • Work with product managers to define expected AI behavior and acceptance criteria. • Collaborate with applied AI engineers to improve prompts and application logic. • Test AI applications against unusual inputs, ambiguous requests, and difficult edge cases. • Develop evaluation cases for prompt injection and unauthorized tool-use attempts. • Support human review workflows for outputs requiring subjective assessment. • Track recurring AI failures and help prioritize reliability improvements. • Monitor production evaluation signals and investigate quality regressions. • Document test coverage, evaluation procedures, and known limitations. • Improve evaluation efficiency, repeatability, and maintainability. • Participate in code reviews, engineering planning, and technical discussions. • Support controlled testing and rollout of new AI-powered features. • Own assigned AI evaluation projects from test design through implementation, reporting, and ongoing improvement.

🎯 Requirements

• 3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field. • Experience developing automated tests for production software applications. • Strong programming skills in Python, TypeScript, JavaScript, or comparable languages. • Familiarity with large language models and modern AI-powered applications. • Understanding of software testing methodologies and test case design. • Experience building automated testing scripts, frameworks, or validation tools. • Familiarity with API testing and backend application integrations. • Understanding of prompt engineering and common AI model failure modes. • Ability to evaluate AI-generated responses against expected requirements. • Experience working with structured data and validating application outputs. • Familiarity with regression testing and software release processes. • Experience debugging application failures and reproducing technical issues. • Understanding of authentication, authorization, and secure application behavior. • Familiarity with databases and application data models. • Experience working with Git and collaborative software development workflows. • Understanding of continuous integration and automated testing pipelines. • Familiarity with logging, monitoring, and production troubleshooting. • Ability to create clear, reproducible test cases. • Strong analytical thinking and attention to detail. • Ability to distinguish application defects from model behavior limitations. • Experience documenting technical issues and communicating findings. • Familiarity with cloud-based application environments. • Ability to collaborate effectively with engineers, product managers, and other stakeholders. • Strong problem-solving skills and a quality-focused engineering mindset. • Ability to independently own evaluation and test automation projects. • Nice to Have: Experience evaluating large language models or generative AI applications. • Nice to Have: Familiarity with OpenAI, Anthropic, or comparable model APIs. • Nice to Have: Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools. • Nice to Have: Familiarity with LLM-as-a-judge evaluation techniques. • Nice to Have: Experience developing AI evaluation datasets and test benchmarks. • Nice to Have: Understanding of human evaluation and response labeling workflows. • Nice to Have: Experience testing retrieval-augmented generation systems. • Nice to Have: Familiarity with embeddings, semantic search, and vector databases. • Nice to Have: Experience evaluating AI agents and tool-calling behavior. • Nice to Have: Familiarity with prompt injection testing and AI security evaluations. • Nice to Have: Experience implementing AI regression tests in CI/CD pipelines. • Nice to Have: Experience building automated evaluation reports or dashboards. • Nice to Have: Familiarity with Playwright, Jest, Vitest, Pytest, or comparable testing frameworks. • Nice to Have: Experience testing TypeScript or Node.js applications. • Nice to Have: Familiarity with React or Next.js application testing. • Nice to Have: Experience working with AWS or serverless infrastructure. • Nice to Have: Experience evaluating AI features in healthcare or other sensitive environments. • Nice to Have: Understanding of HIPAA and healthcare data privacy considerations. • Nice to Have: Experience developing human-in-the-loop quality review processes. • Nice to Have: Experience using AI-assisted development tools such as Cursor or Claude Code.

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume