
51 - 200 employees
π€ Artificial Intelligence
π€ B2B
βοΈ SaaS
π° Venture Round on 2022-04
Artificial Intelligence β’ B2B β’ SaaS
Robots & Pencils is a digital innovation firm that specializes in developing digital strategies and products utilizing mobile, web, and frontier technologies to drive transformation for businesses. They focus on areas such as mobile development, product design, UX research, artificial intelligence, machine learning, and organizational change management. With a commitment to blending creativity with technology, they aim to unlock data insights and enhance product innovation, ultimately delivering excellent customer experiences and helping brands gain a competitive edge.
π November 5
π¨π¦ Canada β Remote
π΅ $176.6k - $243.7k / year
β° Full Time
π Senior
π΄ Lead
𧬠Research Scientist
π» Ghost score 4%
Where this job comes from
Posted on Robots & Pencils's careers page β, not through a job board
First seen by Remote Rocketship November 5
Still open on Robots & Pencils's site. Checked 59 minutes ago.

51 - 200 employees
π€ Artificial Intelligence
π€ B2B
βοΈ SaaS
π° Venture Round on 2022-04
Artificial Intelligence β’ B2B β’ SaaS
Robots & Pencils is a digital innovation firm that specializes in developing digital strategies and products utilizing mobile, web, and frontier technologies to drive transformation for businesses. They focus on areas such as mobile development, product design, UX research, artificial intelligence, machine learning, and organizational change management. With a commitment to blending creativity with technology, they aim to unlock data insights and enhance product innovation, ultimately delivering excellent customer experiences and helping brands gain a competitive edge.
β’ Lead the design and build of an evaluation framework for an enterprise planner-based agent β’ Benchmark planner models, executor LLMs, and inference optimization levers through controlled experiments β’ Design an end-to-end evaluation workflow covering scenarios, user simulation, configuration, execution, grading, comparison, and insights β’ Define scenario schemas, grading specifications, and validators with the client's team β’ Build an LLM-based user simulator for realistic multi-turn conversations β’ Build isolated, parallel experiment execution and route model traffic through the client's LLM gateway β’ Implement optimization levers such as cascade-on-failure, cache-aware model switching, and prompt caching β’ Tag model calls and send traces to attribute tokens, cache usage, latency, and cost β’ Compute success, cost, latency, and pipeline failure metrics β’ Compare experiments using confidence intervals, paired methods, win/loss/tie analysis, and non-inferiority testing β’ Project dollar savings at production volume and plot cost-versus-quality frontiers β’ Build an automated evidence-linked insight engine β’ Build a locally hosted, API-first dashboard with one-command local deployment β’ Run staged experiments, analyze generalization, and deliver recommendations, runbooks, and integration guides β’ Partner with client engineers and leadership, communicate findings to technical and business stakeholders, and mentor engineers
β’ Experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation β’ Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production, not just prototypes β’ Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis β’ Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings β’ Familiarity with LLM routing, model cascades or model selection, and LLM inference internals and cost levers β’ Experience with observability and tracing for LLM systems, including attributing tokens, cache usage, latency and cost per request β’ Ability to build production-quality research tooling using Python services, containerized and sandboxed execution, and data pipelines β’ Prior customer-facing or consulting experience scoping and delivering research engagements β’ Ability to explain technical findings clearly to non-technical audiences and translate them into business impact β’ Helpful extras include publications or open-source contributions, internal dashboards or web UIs, LLM gateway or tracing stack experience, regulated or high-stakes enterprise domain experience, and production cost-quality tradeoff experience
Apply NowImprove your chances of getting an interview by checking your resume score before you apply.
π October 19
Senior Staff Applied Scientist defining time-series forecasting science for Kinaxis, a global supply-chain orchestration company. Guiding ML research, evaluation, and production scale-up.
π¨π¦ Canada β Remote
π° $33M Venture Round on 2000-05
β° Full Time
π Senior
𧬠Research Scientist
π October 1
Senior Data Scientist refactoring Python optimization algorithms and developing pricing models. Solving supply-chain challenges for Tiger Analyticsβ retail and CPG analytics clients.
π January 15
Research Fellow at FirstPrinciples developing cutting-edge AI Physicist for theoretical research. Engaging deeply with advanced AI and physics through innovative methods and collaboration.