Senior Data Scientist – LLM Evaluation, Business Intelligence

Job not on LinkedIn

🔥 11 minutes ago

🗽 New York, Ohio – Remote

🗽 New York – Remote 🏈 Ohio – Remote

💵 $148k - $184k / year

⏰ Full Time

🟠 Senior

📚 Business Intelligence Analyst

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Paradigm Health

WebsiteLinkedInAll Job Openings

51 - 200 employees

🏥 Healthcare

💊 Pharmaceuticals

☁️ SaaS

💰 $203M Series A on 2023-02

Healthcare • Pharmaceuticals • SaaS

Paradigm Health is a clinical research company that re-engineers clinical trials around patient care by combining technology and services to support sponsors and research sites. Their platform and offerings include real-time protocol optimization, feasibility and site selection, patient identification and enrollment acceleration, trial data collection and delivery, and post-approval/late-stage evidence solutions. Paradigm integrates directly with health provider organizations to streamline collaboration, improve enrollment outcomes, and reduce operational burden on research teams. The company positions itself as a bridge between providers and sponsors to make clinical research more efficient, patient-centric, and scalable.

📋 Description

• Design and run evaluations of production LLM pipelines, developing accuracy and quality metrics that tell us how well our systems perform on real clinical tasks. • Support evaluation of our Patient Trial Evaluation (PTE) pipeline — assessing prompts and trial-matching logic for accuracy in reporting, and surfacing where and why they fail. • Measure and improve the effectiveness of our retrieval-augmented generation (RAG) systems, from retrieval quality through final output. • Use accuracy metrics to inform cost/quality tradeoffs across trials, prompts, and models, giving the team a clear basis for which approaches to ship. • Establish and maintain a library of "best prompts" for recurring clinical concepts, backed by evidence rather than intuition. • Partner with AI Engineering to close the loop between evaluation findings and production improvements. • Build and maintain production-grade data models using SQL and dbt on Databricks, ensuring analytics logic is reliable, maintainable, and production-ready. • Help establish and grow a semantic layer that enables consistent, trusted reporting across the organization. • Advance data governance practices across dbt, Databricks, and Hex — documentation, testing, versioning, and clear ownership of metrics. • Collaborate with non-technical internal stakeholders to develop reporting logic, answer data questions, and translate business needs into well-defined data products. • Support both internal and external reporting needs with accurate, well-tested datasets. • For senior applicants: mentor and support junior data scientists — providing guidance on evaluation and analytics approaches, technical best practices, and professional growth.

🎯 Requirements

• Bachelor's degree or higher in data science, statistics, biostatistics, computer science, mathematics, epidemiology, or a related field. • 4+ years of experience as a data scientist, analytics engineer, ML/data professional, or other highly analytical role in life sciences, biotech, healthcare, or a related industry. • Solid understanding of how production LLM pipelines work, and hands-on experience evaluating LLM or NLP systems — accuracy metrics, error analysis, RAG/retrieval quality, prompt evaluation, or similar. • Strong proficiency in SQL, with experience building production data models in dbt (or a comparable transformation framework). • Proficiency in at least one programming language (Python preferred; PySpark a plus). • Experience with modern data platforms such as Databricks, and BI/analytics tools such as Hex, or similar. • Fluency with data science best practices: reusability, version control, documentation, testing, code review, and reproducibility. • Excellent written and verbal communication, with the ability to translate technical concepts into actionable insights for non-technical stakeholders and to develop reporting logic collaboratively with them. • Comfort with AI tooling and agentic workflows in day-to-day work. • Comfort with ambiguity and adaptability to a mission-driven, fast-paced startup environment.

🏖️ Benefits

• Offers Equity