AI/ML Evaluation Reviewer – Healthcare AI

Job not on LinkedIn

🕒 October 20

🇺🇸 United States – Remote

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

Where this job comes from

Posted on mpathic's careers page ↗mpathic's careers page, not through a job board

First seen by Remote Rocketship October 20

Still open on mpathic's site. Checked 4 hours ago.

Apply Now
Logo of mpathic

mpathic

11 - 50 employees

Founded 2021

🏥 Healthcare

💼 Consulting

🍽️ Food & Beverage

Healthcare • Consulting • Food & Beverage

mpathic is an AI-driven safety and analytics company that offers SaaS products and APIs to record, transcribe, analyze, and improve human–AI and human–human interactions in clinical, life‑science, and enterprise settings. Their offerings include speech-to-text transcription, PII redaction, behavior and checklist analytics, human-curated benchmarking datasets, red‑teaming expertise, and an Observing Agent API to add domain-informed guardrails for healthcare and medical agents. mpathic targets life sciences, clinical research organizations, healthcare providers, insurers, and HR/enterprise teams focused on empathy, safety, bias reduction, and compliance.

📋 Description

• Independently assess technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, proposed monitoring, and safeguards • Evaluate model performance and validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty • Assess data quality and representativeness, including data leakage, subgroup performance differences, and generalizability across patient populations and healthcare settings • Evaluate technical reliability, robustness, reproducibility, known failure modes, and performance under atypical or out-of-distribution inputs • Review proposed evaluation plans, performance benchmarks, and success criteria for pilot deployment • Assess technical safeguards, escalation mechanisms, and human-in-the-loop processes • Evaluate monitoring and change-management plans, including performance degradation detection, model behavior monitoring, updates, intervention, and rollback criteria • Collaborate with clinicians and other technical experts on cross-disciplinary concerns • Document strengths, limitations, missing evidence, and technical concerns using a standardized review framework • Provide clear, evidence-based recommendations and identify additional evidence or changes needed • Participate in reviewer calibration and quality assurance activities • Maintain independence and confidentiality while following conflict-of-interest, confidentiality, and data-handling requirements

🎯 Requirements

• At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems • Demonstrated understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error • Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset • Hands-on experience evaluating at least one relevant class of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models • Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions • Strong analytical writing skills, including the ability to communicate complex findings clearly to non-technical audiences • Ability to work independently while contributing to multidisciplinary expert review and calibration processes • Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training requirements • Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments • Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards • Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards • Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis • Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices • Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments • Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device • Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations • Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment

🏖️ Benefits

• Contract, per-review opportunity • Reviewer orientation to the evaluation framework and project-specific requirements prior to the first assignment

Apply Now

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

🕒 October 20

Volga Partners

1001 - 5000

🤖 Artificial Intelligence

🤝 B2B

🏢 Enterprise

Medical trainees evaluating AI-generated healthcare content for Volga Partners. Reviewing clinical reasoning, verifying evidence, and improving healthcare AI accuracy remotely.

🕒 October 19

Sigma AI

501 - 1000

💼 Consulting

🏥 Healthcare

📦 Logistics

Freelance Cantonese-English linguistic annotator performing annotation, transcription, validation, and voice tasks. Supporting Sigma.AI’s global artificial intelligence data-collection and annotation projects remotely.

🇺🇸 United States – Remote

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

🕒 October 16

Pado

1 - 10

🤖 Artificial Intelligence

⚡ Energy

☁️ SaaS

Reserved GPU deal broker matching AI compute supply and demand for PADO’s LG-backed marketplace. Structuring pricing, margins, contracts, and capacity commitments for AI-native buyers.

🇺🇸 United States – Remote

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

🕒 September 29

Heritage Werks

51 - 200

📱 Media

📣 Marketing

Fractional strategic alliances consultant activating consulting and systems-integrator relationships for Heritage Werks Group’s Authority AI. Building partner-sourced opportunities, pilots, proposals, and revenue.

🇺🇸 United States – Remote

⏳ Contract/Temporary

🟠 Senior

🔴 Lead

🤖 Artificial Intelligence

🕒 September 22

Invisible Technologies

201 - 500

💼 Consulting

🏥 Healthcare

📦 Logistics

SMB business operator evaluating bundled and standalone AI assistants for Meridial’s technology clients. Testing real business workflows and documenting reproducible failures across social and commerce tools.

🇺🇸 United States – Remote

💵 $40 - $50 / hour

🔥 Funding within the last year

💰 $100M Series unknown on 2025-10

⏳ Contract/Temporary

🟡 Mid-level

🟠 Senior

🤖 Artificial Intelligence

Developed by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok