AI/ML Evaluation Reviewer – Healthcare AI

Job not on LinkedIn

🔥 15 hours ago

🇺🇸 United States – Remote

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

👻 Ghost score 25%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

mpathic

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2021

🏥 Healthcare

💼 Consulting

🍽️ Food & Beverage

Healthcare • Consulting • Food & Beverage

mpathic is an AI-driven safety and analytics company that offers SaaS products and APIs to record, transcribe, analyze, and improve human–AI and human–human interactions in clinical, life‑science, and enterprise settings. Their offerings include speech-to-text transcription, PII redaction, behavior and checklist analytics, human-curated benchmarking datasets, red‑teaming expertise, and an Observing Agent API to add domain-informed guardrails for healthcare and medical agents. mpathic targets life sciences, clinical research organizations, healthcare providers, insurers, and HR/enterprise teams focused on empathy, safety, bias reduction, and compliance.

📋 Description

• Independently assess technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, proposed monitoring, and safeguards • Evaluate model performance and validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty • Assess data quality and representativeness, including data leakage, subgroup performance differences, and generalizability across patient populations and healthcare settings • Evaluate technical reliability, robustness, reproducibility, known failure modes, and performance under atypical or out-of-distribution inputs • Review proposed evaluation plans, performance benchmarks, and success criteria for pilot deployment • Assess technical safeguards, escalation mechanisms, and human-in-the-loop processes • Evaluate monitoring and change-management plans, including performance degradation detection, model behavior monitoring, updates, intervention, and rollback criteria • Collaborate with clinicians and other technical experts on cross-disciplinary concerns • Document strengths, limitations, missing evidence, and technical concerns using a standardized review framework • Provide clear, evidence-based recommendations and identify additional evidence or changes needed • Participate in reviewer calibration and quality assurance activities • Maintain independence and confidentiality while following conflict-of-interest, confidentiality, and data-handling requirements

🎯 Requirements

• At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems • Demonstrated understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error • Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset • Hands-on experience evaluating at least one relevant class of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models • Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions • Strong analytical writing skills, including the ability to communicate complex findings clearly to non-technical audiences • Ability to work independently while contributing to multidisciplinary expert review and calibration processes • Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training requirements • Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments • Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards • Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards • Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis • Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices • Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments • Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device • Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations • Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment

🏖️ Benefits

• Contract, per-review opportunity • Reviewer orientation to the evaluation framework and project-specific requirements prior to the first assignment