AI Agent Evaluator

πŸ•’ August 6

Where this job comes from

Posted on Hippocratic AI's careers page β†—Hippocratic AI's careers page, not through a job board

First seen by Remote Rocketship August 6

Still open on Hippocratic AI's site. Checked 8 hours ago.

Apply Now
Logo of Hippocratic AI

Hippocratic AI

11 - 50 employees

Founded 2023

πŸ€– Artificial Intelligence

βš•οΈ Healthcare Insurance

πŸ₯ Healthcare

πŸ’° $50M Seed Round on 2023-05

Artificial Intelligence β€’ Healthcare Insurance β€’ Healthcare

Hippocratic AI is an innovative healthcare technology company that specializes in artificial intelligence-driven solutions designed to improve patient care and clinical decision-making. Their products leverage advanced data analysis and machine learning to support healthcare providers, enhancing efficiency and accuracy in medical practices. By integrating cutting-edge AI technology into healthcare processes, Hippocratic AI aims to revolutionize the medical field, ultimately improving outcomes for patients and practitioners alike.

πŸ“‹ Description

β€’ Evaluate recorded and simulated AI agent interactions involving acute symptom triage and assess escalation and de-escalation decisions β€’ Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence β€’ Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment β€’ Identify clinical safety concerns, edge cases, and gaps in triage logic or clinical reasoning β€’ Score and annotate agent performance using structured evaluation rubrics β€’ Provide actionable written feedback to clinical and product teams to inform agent refinement β€’ Participate in case review and calibration sessions with other clinical SMEs β€’ Help develop and refine test scenarios for ongoing agent quality assurance

🎯 Requirements

β€’ Registered Nurses: active, unrestricted RN License β€’ Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP) β€’ Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting β€’ Clinical experience managing patients with diabetes, including insulin titration β€’ Strong, current working knowledge of relevant triage protocols and clinical practice guidelines β€’ Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions β€’ Comfortable working independently in a remote, asynchronous contract role β€’ Strong written communication skills for delivering structured clinical feedback β€’ Experience in telephone triage or nurse advice line settings (recommended) β€’ Experience in travel medicine, occupational health, or pre-travel consultation (recommended) β€’ Familiarity with CDC and other travel medicine guidelines (recommended) β€’ Prior experience in clinical quality assurance, utilization review, or case review (recommended) β€’ Familiarity with AI-assisted clinical tools or digital health platforms (recommended)

πŸ–οΈ Benefits

β€’ Fully remote work β€’ Flexible scheduling β€’ Hourly compensation commensurate with experience

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ August 4

Welo Global

1001 - 5000

πŸ€– Artificial Intelligence

🀝 B2B

☁️ SaaS

Generative AI Analyst annotating text, image, audio, and video data for Welo Data’s AI training systems. Supporting quality checks and dataset development for multilingual AI projects.

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’΅ $15 / hour

⏳ Contract/Temporary

🟒 Junior

🟑 Mid-level

πŸ€– Artificial Intelligence

πŸš«πŸ‘¨β€πŸŽ“ No degree required

πŸ—£οΈπŸ‡¨πŸ‡³ Chinese Required

πŸ•’ August 2

Sunshine Enterprise USA

51 - 200

🎯 Recruiter

πŸ‘₯ HR Tech

🀝 B2B

AI Consultant leading enterprise AI initiatives focused on evaluating, designing, and implementing AI solutions. Collaborating with stakeholders and ensuring compliance with governance and security standards.

πŸ•’ July 31

Welo Global

1001 - 5000

πŸ€– Artificial Intelligence

🀝 B2B

☁️ SaaS

Generative AI Analyst supporting AI projects by reviewing and annotating content across media types. Collaborate in a global community to enhance the development of advanced AI systems.

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ’΅ $15 / hour

⏳ Contract/Temporary

🟒 Junior

🟑 Mid-level

πŸ€– Artificial Intelligence

πŸš«πŸ‘¨β€πŸŽ“ No degree required

πŸ•’ July 29

Benchmark Insurance Group

2 - 10

πŸ›‘οΈ Insurance

🀝 B2B

AI Workflow Architect designing and building automation systems for insurance and real estate operations. Deploying end-to-end AI workflows and collaborating with leadership to optimize processes.

πŸ•’ July 14

Invisible Technologies

201 - 500

πŸ’Ό Consulting

πŸ₯ Healthcare

πŸ“¦ Logistics

Language Alignment & Resource Partner involved in QA and resource development for AI language project. Requires Serbian proficiency and linguistic detail orientation.

πŸ‡ΊπŸ‡Έ United States – Remote

πŸ”₯ Funding within the last year

πŸ’° $100M Series unknown on 2025-10

⏳ Contract/Temporary

🟒 Junior

🟑 Mid-level

πŸ€– Artificial Intelligence

πŸš«πŸ‘¨β€πŸŽ“ No degree required

Developed byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok