Senior Data Scientist – AI Evaluation, Improvement

Job not on LinkedIn

🔥 17 minutes ago

🇺🇸 United States – Remote

💵 $135k - $165k / year

⏰ Full Time

🟠 Senior

📊 Data Scientist

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Evolent

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

⚕️ Healthcare Insurance

🏥 Healthcare

☁️ SaaS

Healthcare Insurance • Healthcare • SaaS

Evolent is a healthcare company focused on improving care outcomes through its comprehensive management solutions across multiple medical specialties. They aim to enhance the patient care journey by offering coordinated services in areas like oncology, cardiology, musculoskeletal disorders, and primary care, while ensuring high-quality treatment pathways and cost management. Evolent believes every person deserves quality care, striving to align treatment guidelines and innovative approaches to meet patient needs effectively.

📋 Description

• Own the diagnostic loop for LLM-based clinical services • Form root-cause hypotheses for failure modes involving prompts, context, retrieval, guidelines, model behavior, and upstream data • Design experiments to isolate causes • Test prompt, configuration, context, and model variants • Verify improvements through structured evaluations and confirm no regressions • Own the evaluation roadmap and quality bar for AI services • Build golden sets and regression suites • Define and interpret metrics including accuracy, guideline adherence, and grounding/faithfulness • Extend the team's evaluation platform thoughtfully • Partner with clinical reviewers to turn findings into labeled evidence and evaluation criteria • Partner with product managers to prioritize important failure modes • Mentor others and help grow the evaluations function • Support the annual clinical-guidelines update cycle with regression evaluation • Document failure taxonomies, experiment templates, and variant history • Handle clinical data, including PHI, under security, privacy, and compliance requirements

🎯 Requirements

• Bachelor's degree in Data Science, Computer Science, Statistics, or a related quantitative field — or equivalent experience • 5+ years of data science or applied machine learning experience, including shipping and maintaining models or AI systems in production • 2+ years of recent, hands-on experience evaluating and improving LLM-based systems • Experience with structured error analysis, prompt/configuration iteration, experiment design, and metrics interpretation • Experience with LLM evaluation methods and tools, including golden/regression sets, LLM-as-judge with validation, and tracing/observability tooling • Strong Python and solid data-analysis skills; SQL is a plus • Comfort computing and reasoning about sensitivity, specificity, and PPV • Hypothesis-driven working style • Strong written communication • Must handle clinical data, including PHI, according to security, privacy, and compliance requirements • Comprehensive background check required • In-person I-9 verification required • May be subject to drug screening prior to employment • Government-issued photo ID required for identity verification • High-speed internet over 10 Mbps at home • Preferred: healthcare experience; mentoring or technical workstream leadership; experience working with clinical reviewers or domain experts; LLM observability/telemetry stacks; statistics or experimentation background; Master's degree in a quantitative field

🏖️ Benefits

• Work/life balance • Flexible work arrangements and autonomy • Comprehensive benefits, including health insurance benefits, for qualifying employees • High-speed home internet requirement supported as a home-work capability