Data Scientist – AI & Agentic Applications & Benchmarking

August 12

🏄 California – Remote

Although this job is listed in California, this company may be open to hiring remote in other states.

⏰ Full Time

🟡 Mid-level

🟠 Senior

📊 Data Scientist

🦅 H1B Visa Sponsor

Apply Now

Receive Emails with Similar Jobs

CloudBees

WebsiteLinkedInAll Job Openings

Software • B2B • DevOps

CloudBees is a leading provider of DevOps solutions, specializing in optimizing and managing the software development lifecycle through its advanced Continuous Integration and Continuous Delivery (CI/CD) capabilities. By leveraging Jenkins, CloudBees enables developers to streamline workflows, automate processes, and enhance software delivery across hybrid and multi-cloud environments. CloudBees empowers organizations to optimize developer experiences through features like security and compliance management, feature management, and smart testing, ensuring seamless integration of tools within the DevOps ecosystem.

501 - 1000 employees

Founded 2010

🤝 B2B

💰 $95M Debt Financing on 2021-12

📋 Description

• CloudBees is the leading software delivery platform for enterprise DevOps teams. • As a high-growth startup, we empower developers to build, deploy, and manage software more efficiently. • Now, we’re bringing agentic intelligence into our platform to supercharge developer workflows—and we need a data scientist who can both drive insights and tell the story behind the metrics. • The Role: CloudBees is seeking a startup-savvy Data Scientist to help define, measure, and evangelize the impact of Agentic Applications across our platform. • You’ll work closely with engineers and product teams to prototype and measure AI and Agentic experiences, using evals, telemetry, and AI benchmarks to help the company drive the conversation in the market and with customers. • Translating performance into clear, compelling narratives to our customers and internal teams. • As a founding member of the team, you will lead the charge as equal parts builder, evaluator, and communicator—with the technical depth to prototype in Python notebooks, Claude Code, and other tools to drive clarity to write about what matters. • Key Responsibilities: Partner with our platform team to develop and prototype telemetry, eval frameworks, and benchmarks for emerging agentic systems. • Partner with product and engineering teams to measure AI outcomes and usage across customers and teams. • Help define KPIs and success metrics for AI and LLM-driven features and workflows. • Use Python notebooks to explore data, visualize insights, and test hypotheses rapidly and share insights. • Tell the story behind the numbers: Write internal documentation, performance summaries, and thought leadership around outcomes. • Enable engineering teams to instrument, log, and evaluate agent performance effectively. • Stay up to date with evolving metrics and evaluation techniques in the LLM and agentic AI ecosystem.

🎯 Requirements

• 3+ years of experience in data science or ML analytics roles, ideally in startup or high-growth environments. • Proficiency in Python , including building and sharing analysis via Jupyter notebooks . • Experience working with evals, telemetry, A/B testing, and evaluating user-facing ML systems. • Experience with AI/ML tools such as MLFlow, Hugging Face, or other Model / LLM tools. • Ability to partner with technical teams to define meaningful metrics and benchmarks. • Clear communication skills—capable of writing about outcomes , sharing learnings, and influencing stakeholders. • Comfort working in fast-paced, ambiguous environments where speed and clarity matter.

🏖️ Benefits

• Competitive salary • startup equity • excellent benefits • remote-first culture built on trust and innovation