Principal Applied Scientist

πŸ•’ November 12

Where this job comes from

Posted on Robots & Pencils's careers page β†—Robots & Pencils's careers page, not through a job board

First seen by Remote Rocketship November 12

Still open on Robots & Pencils's site. Checked 2 hours ago.

Apply Now
Logo of Robots & Pencils

Robots & Pencils

51 - 200 employees

πŸ€– Artificial Intelligence

🀝 B2B

☁️ SaaS

πŸ’° Venture Round on 2022-04

Artificial Intelligence β€’ B2B β€’ SaaS

Robots & Pencils is a digital innovation firm that specializes in developing digital strategies and products utilizing mobile, web, and frontier technologies to drive transformation for businesses. They focus on areas such as mobile development, product design, UX research, artificial intelligence, machine learning, and organizational change management. With a commitment to blending creativity with technology, they aim to unlock data insights and enhance product innovation, ultimately delivering excellent customer experiences and helping brands gain a competitive edge.

πŸ“‹ Description

β€’ Design the end-to-end evaluation workflow for a planner-executor agent β€’ Define scenario schemas, grading specifications and validators with the client’s team β€’ Build an LLM-based user simulator for realistic multi-turn conversations β€’ Build isolated, parallel experiment execution with configuration files and harness adapters β€’ Implement optimization levers such as cascade-on-failure, cache-aware model switching and prompt caching β€’ Tag model calls and send traces to attribute tokens, cache usage, latency and cost β€’ Grade runs using expected end states and optional LLM judges β€’ Compute verified success, cost per success, p95 latency and pipeline-stage failures β€’ Compare experiments using confidence intervals, paired methods, win/loss/tie analysis and non-inferiority tests β€’ Project dollar savings at production volume and plot cost-versus-quality frontiers β€’ Build an automated evidence-linked insight engine β€’ Build a locally hosted, API-first dashboard with one-command deployment β€’ Run staged experiments from production baseline through model and lever screening β€’ Analyze generalization by scenario category β€’ Deliver recommendations, runbooks and integration guides β€’ Partner with client engineers and leadership on scenarios, margins and success criteria β€’ Write technical reports and leadership summaries β€’ Set experimental standards and mentor engineers on rigorous agent evaluation

🎯 Requirements

β€’ Experience with LLM and agent evaluation methods, including programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation β€’ Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production β€’ Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis β€’ Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and projected dollar savings β€’ Familiarity with LLM routing, model cascades, model selection and LLM inference cost levers β€’ Experience with observability and tracing for LLM systems, attributing tokens, cache usage, latency and cost per request β€’ Ability to build production-quality research tooling using Python services, containerized and sandboxed execution, and data pipelines β€’ Prior customer-facing or consulting experience scoping and delivering research engagements β€’ Ability to explain technical findings clearly to non-technical audiences and translate them into business impact

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ November 11

Upstart

1001 - 5000

🚘 Automotive

πŸ’Ό Consulting

πŸ₯ Healthcare

Senior Applied Scientist developing production machine learning models for Upstart’s AI lending marketplace. Improving credit prediction accuracy through applied research, experimentation, and statistical analysis.

πŸ•’ November 11

Upstart

1001 - 5000

🚘 Automotive

πŸ’Ό Consulting

πŸ₯ Healthcare

Staff Applied Scientist leading machine learning strategy and modeling for Upstart’s AI lending marketplace. Improving predictive performance, reliability, and scalable credit decisions.

πŸ•’ November 11

Hook

11 - 50

πŸ“± Media

πŸ‘₯ B2C

Research Scientist developing personalized recommendation systems for Hook, a social music platform. Applying machine learning research to improve discovery, fan expression, and creator growth.

πŸ•’ November 10

Zillow

5001 - 10000

🏠 Real Estate

πŸ›οΈ eCommerce

πŸ‘₯ B2C

Senior Applied Scientist developing and evaluating LLM and machine-learning systems for Zillow’s real estate platform. Shipping reliable AI features that improve home-search customer outcomes.

πŸ•’ November 10

American Institutes for Research

1001 - 5000

πŸ₯ Healthcare

πŸ’Ό Consulting

πŸ“š Education

Senior Medicaid researcher leading policy studies, administrative-data analytics, and client projects for AIR, a nonprofit behavioral and social science research organization.

Developed byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok