Senior Data Scientist – AI Evaluation

🔥 6 minutes ago

🏈 North America – Remote

⏰ Full Time

🟠 Senior

📊 Data Scientist

👻 Ghost score 25%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Alpaca

WebsiteLinkedInAll Job Openings

201 - 500 employees

🔌 API

💳 Fintech

₿ Crypto

API • Fintech • Crypto

Alpaca is a fintech company that provides a comprehensive brokerage and trading platform through a suite of APIs. These APIs enable developers and businesses to integrate algorithmic trading, app development, and embedded investing into their services. Alpaca offers services like trading in US stocks, ETFs, and cryptocurrency with options for local currency transactions. The company is recognized for its cyber security practices and is a member of FINRA and SIPC. Alpaca is ideal for fintech startups, broker-dealers, hedge funds, and other financial services looking to build sophisticated trading applications and platforms with minimal friction through their well-documented Broker API.

📋 Description

• Design AI evaluations by defining ground truth, metrics, and scoring methods for models and agents • Build repeatable evaluation loops to track quality over time and catch regressions before release • Partner with engineering and analytics engineering to operationalize evaluation harnesses • Translate evaluation results into actionable recommendations for system improvements • Establish evaluation guidelines, documentation, and review practices • Mentor and align teams on evaluation best practices and measurable AI quality • Partner with Product, Engineering, Analytics Engineering, and business stakeholders to define quality standards and drive iteration • Own the quality bar independently of the teams that build and optimize the systems

🎯 Requirements

• Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation) • Strong statistical and ML foundation, including sample sizing, confidence intervals, significance, handling non-determinism, and validating automated graders against human ground truth • Proficiency in Python and SQL, with experience evaluating models in production environments • Strong judgment in defining quality metrics and ground truth for ambiguous outputs • Excellent communication and cross-functional collaboration skills • Strong problem-solving ability in fast-paced, greenfield environments • 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation • Quantitative degree is a plus; equivalent industry experience is equally welcome • Nice to have: hands-on LLM/agent evaluation in production, eval harnesses, LLM-as-judge calibration, and CI regression gates • Nice to have: experience evaluating text-to-SQL, analytics agents, or systems where correctness is verifiable against data • Nice to have: background in fintech, brokerage, or domains with business or risk consequences • Nice to have: fluency with AI tools in research and engineering workflows

🏖️ Benefits

• Competitive Salary & Stock Options • Health Benefits • New Hire Home-Office Setup: One-time USD $500 • Monthly Stipend: USD $150 per month via a Brex Card