Applied Data Scientist, Health AI Evaluation, Datasets

Job not on LinkedIn

🔥 1 hour ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

InnoData

WebsiteLinkedInAll Job Openings

2 - 10 employees

Founded 2019

🤝 B2B

💼 Consulting

🌍 Social Impact

B2B • Consulting • Social Impact

InnoData is INNOvation DATA SCS, an Italian social cooperative based in Foggia that identifies itself as a provider of technological solutions. The company website (currently under maintenance) highlights "Soluzioni tecnologiche" (technological solutions) and emphasizes social impact ("Impatto sociale"). Contact details listed include Via Francesco Crispi 65, 71121 Foggia, Italy. Based on the available information, InnoData appears to operate at the intersection of technology and social impact, likely offering tech-focused services to other organizations.

📋 Description

• Translate customer goals — such as improving differential diagnosis, evaluating a clinical note summarizer, testing a RAG-based medical literature assistant, or creating preference data for patient-facing chatbots — into dataset specifications, taxonomies, rubrics, sampling plans, and acceptance criteria. • Make multimodal health AI a core focus: design training and evaluation datasets across clinical text, medical images, waveforms, structured EHR data, claims, trial data, medical literature, patient communications, payer policies, drug information, and other clinical artifacts, as well as use cases such as clinical reasoning, medical QA, note summarization, medical coding, patient communication, utilization management, and literature synthesis. • Design evaluations for retrieval-augmented and source-grounded health AI systems, including evidence citation, faithfulness, contraindication handling, guideline adherence, source freshness, and failure modes caused by incomplete, conflicting, or stale context. • Define sampling strategies, label schemas, inter-annotator agreement targets, adjudication workflows, SME review patterns, and quality thresholds in partnership with Language Data Scientists, clinicians, biomedical experts, and quality teams. • Build statistical and ML checks that make healthcare datasets trustworthy: stratified sampling across specialties and patient subgroups, bias and representation analysis, leakage detection, distribution shift checks, uncertainty estimates, reliability metrics, and subgroup performance analysis. • Partner with Applied Research Scientists and AI/ML Research Engineers to instrument datasets into evaluation and post-training pipelines, including rubric-grounded LLM-as-judge prompts, regression suites, model comparison workflows, experiment tracking, and model-improvement feedback loops. • Evaluate health AI behavior beyond surface accuracy: calibration, hallucination on safety-critical content, refusal appropriateness, robustness under ambiguity, equity across patient subgroups, and safe handoff in agentic or workflow-integrated systems. Reason concretely about clinical workflow fit: where outputs enter care delivery, what evidence a clinician or reviewer would need to trust them, when uncertainty must be surfaced, and how patient-facing, clinician-facing, payer, pharma, and operational use cases differ in risk. • Own data quality from source intake through delivery, including de-identified clinical text, medical literature, synthetic cases, structured records, client policies, and knowledge bases, with attention to PHI/PII handling, provenance, audit trails, versioning, and compliance documentation. • Stay current on the health AI landscape — regulatory developments such as FDA guidance on AI/ML-enabled medical devices and EU AI Act health provisions, benchmark releases such as MedQA, MedMCQA, and HealthBench, and emerging clinical evaluation methodology. • Support customer discovery and proposal work by scoping dataset programs, sizing annotation and SME review effort, identifying regulatory or data-access constraints, and explaining methodology choices to client clinical and ML leadership. • Contribute to Innodata internal IP: reusable health-domain taxonomies, evaluation rubrics, golden datasets, clinical review playbooks, dataset quality checks, and methodology templates.

🎯 Requirements

• 5+ years of data science experience, including at least 2+ years with healthcare, clinical, biomedical, payer, provider, pharma, life sciences, or comparable regulated health data. • Working knowledge of healthcare data and standards: EHR structure, clinical documentation conventions, ICD-10, CPT, SNOMED CT, LOINC, RxNorm, and at least passing familiarity with FHIR, HL7, or equivalent interoperability concepts. • Hands-on experience designing ML datasets, not just consuming them: writing annotation guidelines, sizing cohorts, setting quality thresholds, designing QA checks, and shipping data that downstream teams can train or evaluate on. • Familiarity with LLM-based health AI workflows, including prompt design, rubric-based evaluation, retrieval-augmented generation, LLM-as-judge methods, model comparison, and the limitations of automated evaluation in clinical contexts. • Strong Python and SQL; comfort with pandas, scikit-learn, statsmodels or equivalent tools; and working familiarity with modern LLM tooling such as Hugging Face, evaluation frameworks, prompt development tools, or model APIs. • Statistical literacy across sampling design, bias and fairness analysis, inter-annotator agreement metrics (Cohen or Fleiss kappa, Krippendorff alpha), confidence intervals, significance testing where appropriate, error analysis, and the ability to push back when a number is being over-interpreted. • Solid grasp of healthcare privacy, compliance, and governance: HIPAA, de-identification standards (Safe Harbor and Expert Determination), practical mechanics of working with PHI safely, auditability, access control, and documentation fit for high-stakes or regulated AI programs. • Ability to work credibly with clinicians, biomedical SMEs, research scientists, engineers, technical solutions teams, annotators, and customer stakeholders. • A bias toward clinical realism: you would rather build a smaller dataset that reflects what clinicians, reviewers, patients, or care teams actually see than a larger dataset that looks impressive on paper but fails in practice. • Degree in a relevant field such as biostatistics, epidemiology, computational biology, health informatics, computer science with a health focus, statistics, a clinical degree with quantitative training, or equivalent demonstrated experience. • Clinical credentials are not required, but candidates must be able to work credibly with clinicians, biomedical SMEs, and health AI customers; candidates with MD, RN, PharmD, MPH, PhD, or health informatics backgrounds are especially encouraged.