Staff Applied Scientist

πŸ•’ November 5

πŸ‡¨πŸ‡¦ Canada – Remote

πŸ’΅ $176.6k - $243.7k / year

⏰ Full Time

🟠 Senior

πŸ”΄ Lead

🧬 Research Scientist

Where this job comes from

Posted on Robots & Pencils's careers page β†—Robots & Pencils's careers page, not through a job board

First seen by Remote Rocketship November 5

Still open on Robots & Pencils's site. Checked 59 minutes ago.

Apply Now
Logo of Robots & Pencils

Robots & Pencils

51 - 200 employees

πŸ€– Artificial Intelligence

🀝 B2B

☁️ SaaS

πŸ’° Venture Round on 2022-04

Artificial Intelligence β€’ B2B β€’ SaaS

Robots & Pencils is a digital innovation firm that specializes in developing digital strategies and products utilizing mobile, web, and frontier technologies to drive transformation for businesses. They focus on areas such as mobile development, product design, UX research, artificial intelligence, machine learning, and organizational change management. With a commitment to blending creativity with technology, they aim to unlock data insights and enhance product innovation, ultimately delivering excellent customer experiences and helping brands gain a competitive edge.

πŸ“‹ Description

β€’ Lead the design and build of an evaluation framework for an enterprise planner-based agent β€’ Benchmark planner models, executor LLMs, and inference optimization levers through controlled experiments β€’ Design an end-to-end evaluation workflow covering scenarios, user simulation, configuration, execution, grading, comparison, and insights β€’ Define scenario schemas, grading specifications, and validators with the client's team β€’ Build an LLM-based user simulator for realistic multi-turn conversations β€’ Build isolated, parallel experiment execution and route model traffic through the client's LLM gateway β€’ Implement optimization levers such as cascade-on-failure, cache-aware model switching, and prompt caching β€’ Tag model calls and send traces to attribute tokens, cache usage, latency, and cost β€’ Compute success, cost, latency, and pipeline failure metrics β€’ Compare experiments using confidence intervals, paired methods, win/loss/tie analysis, and non-inferiority testing β€’ Project dollar savings at production volume and plot cost-versus-quality frontiers β€’ Build an automated evidence-linked insight engine β€’ Build a locally hosted, API-first dashboard with one-command local deployment β€’ Run staged experiments, analyze generalization, and deliver recommendations, runbooks, and integration guides β€’ Partner with client engineers and leadership, communicate findings to technical and business stakeholders, and mentor engineers

🎯 Requirements

β€’ Experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation β€’ Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production, not just prototypes β€’ Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis β€’ Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings β€’ Familiarity with LLM routing, model cascades or model selection, and LLM inference internals and cost levers β€’ Experience with observability and tracing for LLM systems, including attributing tokens, cache usage, latency and cost per request β€’ Ability to build production-quality research tooling using Python services, containerized and sandboxed execution, and data pipelines β€’ Prior customer-facing or consulting experience scoping and delivering research engagements β€’ Ability to explain technical findings clearly to non-technical audiences and translate them into business impact β€’ Helpful extras include publications or open-source contributions, internal dashboards or web UIs, LLM gateway or tracing stack experience, regulated or high-stakes enterprise domain experience, and production cost-quality tradeoff experience

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ October 19

Kinaxis

1001 - 5000

πŸ’Ό Consulting

πŸ₯ Healthcare

🏭 Manufacturing

Senior Staff Applied Scientist defining time-series forecasting science for Kinaxis, a global supply-chain orchestration company. Guiding ML research, evaluation, and production scale-up.

πŸ‡¨πŸ‡¦ Canada – Remote

πŸ’° $33M Venture Round on 2000-05

⏰ Full Time

🟠 Senior

🧬 Research Scientist

πŸ•’ October 1

Tiger Analytics

1001 - 5000

πŸ₯ Healthcare

πŸ“¦ Logistics

πŸ“£ Marketing

Senior Data Scientist refactoring Python optimization algorithms and developing pricing models. Solving supply-chain challenges for Tiger Analytics’ retail and CPG analytics clients.

πŸ‡¨πŸ‡¦ Canada – Remote

⏰ Full Time

πŸ”΄ Lead

🧬 Research Scientist

πŸ•’ January 15

FirstPrinciples Holding Company

51 - 200

πŸ’Ό Consulting

πŸ“£ Marketing

πŸ—οΈ Construction

Research Fellow at FirstPrinciples developing cutting-edge AI Physicist for theoretical research. Engaging deeply with advanced AI and physics through innovative methods and collaboration.

πŸ‡¨πŸ‡¦ Canada – Remote

⏰ Full Time

🟠 Senior

πŸ”΄ Lead

🧬 Research Scientist

Developed byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok