Senior/Staff Data Scientist – Product Experimentation & Evaluation, LLMs & AI

September 29

Apply Now

Receive Emails with Similar Jobs

ItsaCheckmate

WebsiteLinkedInAll Job Openings

SaaS • Hospitality

ItsaCheckmate is a technology company that provides powerful ordering solutions for busy restaurants. Their platform enables restaurants to drive sales and connect with customers through various ordering channels, including web, app, kiosk, and third-party integrations. With its scalable enterprise technology, ItsaCheckmate helps restaurants manage and streamline operations, reduce order errors, and increase productivity. They offer features such as loyalty programs, catering management, QR-code ordering, and unified channel management. Trusted by over 2,400 brands, ItsaCheckmate is dedicated to supporting restaurant growth with innovative digital solutions and a marketplace of integrations.

201 - 500 employees

☁️ SaaS

📋 Description

• Strategic partner to product, engineering, and trust and safety teams; define evaluation frameworks and influence roadmap. • Lead end-to-end experimentation: hypothesis generation, metric design, experiment design (A/B, multivariate, sequential), analysis, and interpretation. • Build and maintain evaluation frameworks for LLMs: correctness, consistency, safety, hallucination detection, bias/fairness. • Develop predictive models, classification/ranking systems, and heuristics to improve AI/language generation product features. • Collaborate with prompt engineers and model builders to test prompt strategies, fine-tuning, or model selection; perform failure modes/error analysis. • Automate experiment pipelines: dashboards, monitoring, alerting, instrumentation; ensure data quality and measurement integrity. • Use causal inference and observational studies when randomized experiments are not feasible. • Present findings and recommendations to technical and non-technical leadership; influence product decisions. • Drive rapid, startup-like experimentation: iterate quickly with scrappy but rigorous methods and learn from limited data. • Shape large-scale product experimentation frameworks and integrate results into product strategy. • Lead and mentor teams of data scientists, analysts, and engineers; set best practices for experiment design and AI product evaluation.

🎯 Requirements

• ~8-12+ years of experience in data science / ML roles, ideally with experiment design/product analytics. • Proven track record in both startup-style and large-scale product experimentation. • Experience leading teams, setting strategy, and driving execution in cross-functional environments. • Strong background with statistical methods, causal inference, and rigorous measurement. • Experience using LLMs / NLP / AI / prompt engineering or a closely related field. • Excellent coding skills in Python (or similar), strong SQL, experience building and deploying models or analytic pipelines. • Ability to work in cross-functional teams, translate technical results into business or product changes. • Strong communication skills; ability to explain complex analyses to non-technical stakeholders. • Preferred: Experience fine-tuning or working with multiple LLM providers / APIs. • Preferred: Experience with experiment platforms or building internal tooling for experimentation & model evaluation. • Preferred: Experience in voice / ASR or other multi-modal data.

🏖️ Benefits

• Health Care Plan (Medical, Dental & Vision) • Retirement Plan (401k) • Life Insurance (Basic, Voluntary & AD&D) • Flexible Paid Time Off • Family Leave (Maternity, Paternity) • Short Term & Long Term Disability • Training & Development • Work From Home • Stock Option Plan