AI Agent Evaluator

Job not on LinkedIn

🔥 2 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Hippocratic AI

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2023

🤖 Artificial Intelligence

⚕️ Healthcare Insurance

🏥 Healthcare

💰 $50M Seed Round on 2023-05

Artificial Intelligence • Healthcare Insurance • Healthcare

Hippocratic AI is an innovative healthcare technology company that specializes in artificial intelligence-driven solutions designed to improve patient care and clinical decision-making. Their products leverage advanced data analysis and machine learning to support healthcare providers, enhancing efficiency and accuracy in medical practices. By integrating cutting-edge AI technology into healthcare processes, Hippocratic AI aims to revolutionize the medical field, ultimately improving outcomes for patients and practitioners alike.

📋 Description

• Evaluate recorded and simulated AI agent interactions involving acute symptom triage and assess escalation and de-escalation decisions • Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence • Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment • Identify clinical safety concerns, edge cases, and gaps in triage logic or clinical reasoning • Score and annotate agent performance using structured evaluation rubrics • Provide actionable written feedback to clinical and product teams to inform agent refinement • Participate in case review and calibration sessions with other clinical SMEs • Help develop and refine test scenarios for ongoing agent quality assurance

🎯 Requirements

• Registered Nurses: active, unrestricted RN License • Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP) • Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting • Clinical experience managing patients with diabetes, including insulin titration • Strong, current working knowledge of relevant triage protocols and clinical practice guidelines • Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions • Comfortable working independently in a remote, asynchronous contract role • Strong written communication skills for delivering structured clinical feedback • Experience in telephone triage or nurse advice line settings (recommended) • Experience in travel medicine, occupational health, or pre-travel consultation (recommended) • Familiarity with CDC and other travel medicine guidelines (recommended) • Prior experience in clinical quality assurance, utilization review, or case review (recommended) • Familiarity with AI-assisted clinical tools or digital health platforms (recommended)

🏖️ Benefits

• Fully remote work • Flexible scheduling • Hourly compensation commensurate with experience