Senior Data Scientist – AI Evaluation

πŸ•’ October 7

🏈 North America – Remote

⏰ Full Time

🟠 Senior

πŸ“Š Data Scientist

Where this job comes from

Posted on Alpaca's careers page β†—Alpaca's careers page, not through a job board

First seen by Remote Rocketship October 7

Still open on Alpaca's site. Checked 2 hours ago.

Apply Now
Logo of Alpaca

Alpaca

201 - 500 employees

πŸ”Œ API

πŸ’³ Fintech

β‚Ώ Crypto

API β€’ Fintech β€’ Crypto

Alpaca is a fintech company that provides a comprehensive brokerage and trading platform through a suite of APIs. These APIs enable developers and businesses to integrate algorithmic trading, app development, and embedded investing into their services. Alpaca offers services like trading in US stocks, ETFs, and cryptocurrency with options for local currency transactions. The company is recognized for its cyber security practices and is a member of FINRA and SIPC. Alpaca is ideal for fintech startups, broker-dealers, hedge funds, and other financial services looking to build sophisticated trading applications and platforms with minimal friction through their well-documented Broker API.

πŸ“‹ Description

β€’ Design AI evaluations by defining ground truth, metrics, and scoring methods for models and agents β€’ Build repeatable evaluation loops to track quality over time and catch regressions before release β€’ Partner with engineering and analytics engineering to operationalize evaluation harnesses β€’ Translate evaluation results into actionable recommendations for system improvements β€’ Establish evaluation guidelines, documentation, and review practices β€’ Mentor and align teams on evaluation best practices and measurable AI quality β€’ Partner with Product, Engineering, Analytics Engineering, and business stakeholders to define quality standards and drive iteration β€’ Own the quality bar independently of the teams that build and optimize the systems

🎯 Requirements

β€’ Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation) β€’ Strong statistical and ML foundation, including sample sizing, confidence intervals, significance, handling non-determinism, and validating automated graders against human ground truth β€’ Proficiency in Python and SQL, with experience evaluating models in production environments β€’ Strong judgment in defining quality metrics and ground truth for ambiguous outputs β€’ Excellent communication and cross-functional collaboration skills β€’ Strong problem-solving ability in fast-paced, greenfield environments β€’ 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation β€’ Quantitative degree is a plus; equivalent industry experience is equally welcome β€’ Nice to have: hands-on LLM/agent evaluation in production, eval harnesses, LLM-as-judge calibration, and CI regression gates β€’ Nice to have: experience evaluating text-to-SQL, analytics agents, or systems where correctness is verifiable against data β€’ Nice to have: background in fintech, brokerage, or domains with business or risk consequences β€’ Nice to have: fluency with AI tools in research and engineering workflows

πŸ–οΈ Benefits

β€’ Competitive Salary & Stock Options β€’ Health Benefits β€’ New Hire Home-Office Setup: One-time USD $500 β€’ Monthly Stipend: USD $150 per month via a Brex Card

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Built byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok