Applied AI Evaluation Scientist

🕒 February 25

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Jump - Advisor AI

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded 2023

🤖 Artificial Intelligence

💳 Fintech

☁️ SaaS

💰 $24.6M Series A - Jump on 2025-02

Artificial Intelligence • Fintech • SaaS

Jump - Advisor AI is an AI-powered SaaS platform that automates administrative and client-facing workflows for financial advisors and wealth teams. It provides meeting preparation, automated note-taking, follow-up email generation, client engagement tools, and data-driven insights, while integrating with CRMs, financial planning tools, conferencing systems, and calendars. Built with enterprise-grade security and configurable compliance controls, Jump is designed for RIAs, broker-dealers, and other advisory firms to save time, increase meeting capacity, and simplify supervision and auditability.

📋 Description

• Design and curate evaluation datasets for retrieval quality • Measure retrieval quality using metrics like Recall@k, Precision@k, MRR, and NDCG@k • Recommend data cleaning/normalization strategies • Evaluate and optimize chunking strategies • Assess embedding and re-ranking strategies • Evaluate generation quality in context • Attribute failures across the pipeline • Conduct systematic error analysis on AI/ML system outputs • Design and validate LLM-as-Judge evaluators • Build and maintain golden datasets for CI regression testing of AI pipelines • Lead or facilitate annotation workflows

🎯 Requirements

• BS in Computer Science, Statistics, Data Science, Information Science, Mathematics, Engineering, or a related quantitative field (MS or PhD preferred) • 5+ years of experience in data science, applied ML, information retrieval, or AI evaluation roles • Strong product sense • Statistical rigor • Information retrieval fundamentals • Proficiency with Python and SQL (Experience with Elixir is a plus) • Hands-on experience with LLMs • Data labeling and annotation experience • Strong written and verbal communication

🏖️ Benefits

• Competitive salary • Equity • Health insurance • Dental insurance • Vision insurance