Senior Data Scientist, AI Evaluation & Improvement

🔥 44 minutes ago

🇺🇸 United States – Remote

💵 $120k - $145k / year

⏰ Full Time

🟠 Senior

📊 Data Scientist

Where this job comes from

Posted on Evolent's careers page ↗Evolent's careers page, not through a job board

First seen by Remote Rocketship 44 minutes ago

Apply Now
Logo of Evolent

Evolent

1001 - 5000 employees

⚕️ Healthcare Insurance

🏥 Healthcare

☁️ SaaS

Healthcare Insurance • Healthcare • SaaS

Evolent is a healthcare company focused on improving care outcomes through its comprehensive management solutions across multiple medical specialties. They aim to enhance the patient care journey by offering coordinated services in areas like oncology, cardiology, musculoskeletal disorders, and primary care, while ensuring high-quality treatment pathways and cost management. Evolent believes every person deserves quality care, striving to align treatment guidelines and innovative approaches to meet patient needs effectively.

📋 Description

• Own the diagnostic loop for LLM-based clinical services by investigating failure modes, forming root-cause hypotheses, and designing isolating experiments • Test prompt, configuration, context, and model variants using structured evaluations and verify improvements without regressions • Own the AI services evaluation roadmap and quality bar, including deployment metrics, golden sets, regression suites, and shipping criteria • Use and thoughtfully extend the evaluation platform; build evaluation sets, define metrics, and run and interpret evaluation batteries • Partner with clinical reviewers to turn findings into labeled evidence and executable evaluation criteria • Partner with product managers to prioritize important failure modes • Mentor others in evaluation methods and grow the evaluation function • Support annual clinical-guidelines updates with regression evaluation as guidelines, prompts, and models change • Document diagnostic playbooks, failure taxonomies, experiment templates, and variant history • Handle clinical data, including PHI, according to security, privacy, and compliance requirements

🎯 Requirements

• Bachelor's degree in Data Science, Computer Science, Statistics, or a related quantitative field — or equivalent experience • 5+ years of data science or applied machine learning experience, including shipping and maintaining models or AI systems in production • 2+ years of recent, hands-on experience evaluating and improving LLM-based systems • Demonstrated experience with LLM evaluation methods and tools — golden/regression sets, LLM-as-judge with validation, tracing and observability tooling • Strong Python and solid data-analysis skills; SQL a plus • Comfort computing and reasoning about sensitivity, specificity, and PPV • Hypothesis-driven working style • Strong written communication • High speed internet over 10 Mbps at home • Comprehensive background check • In-person I-9 verification • May be subject to drug screening prior to employment • Government-issued photo ID submission for identity verification • Preferred: healthcare experience; mentoring or leading technical workstreams; experience working with clinical reviewers or domain experts; LLM observability/telemetry stacks; statistics or experimentation background; Master's degree in a quantitative field

🏖️ Benefits

• Work/life balance • Flexibility to suit work to employees' lives • Autonomy to get things done • Diversity and inclusion • Comprehensive benefits, including health insurance benefits, for qualifying employees

Apply Now

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

🔥 1 hour ago

Honor

501 - 1000

🏥 Healthcare

💼 Consulting

📦 Logistics

Staff Data Scientist building LLM-powered agentic systems for Honor’s aging-care technology platform. Deploying reliable AI workflows and predictive models for older-adult care operations.

🔥 2 hours ago

CrossCountry Consulting

501 - 1000

🏥 Healthcare

🏨 Hospitality

💼 Consulting

Data strategy leader helping CrossCountry Consulting modernize CFO organizations’ analytics and technology. Architecting cloud data solutions, leading implementations, and mentoring technical teams.

🔥 2 hours ago

Highmark Health

10,000+ employees

🛡️ Insurance

💼 Consulting

📦 Logistics

Lead Data Scientist developing predictive and machine-learning solutions for Highmark Health’s healthcare operations. Translating complex data analyses into business workflows and mentoring analytics teams.

🔥 7 hours ago

Auto Approve

201 - 500

🚘 Automotive

💼 Consulting

📦 Logistics

Direct mail analytics strategist owning targeting, modeling, testing, and ROI for Auto Approve’s auto-finance campaigns. Partnering with executives, Marketing, Product, bureaus, and mailhouses.

🔥 8 hours ago

Castellum Inc.

51 - 200

💼 Consulting

📦 Logistics

🏭 Manufacturing

Senior Data Scientist developing secure machine-learning solutions for Army Vantage and DoD platforms. Building models, APIs, data pipelines, and analytics supporting national defense missions.

🇺🇸 United States – Remote

💵 $170k - $184k / year

💰 $4M Post-IPO Debt on 2023-11

⏰ Full Time

🟡 Mid-level

🟠 Senior

📊 Data Scientist

Developed by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok