Staff Applied Scientist

🔥 1 hour ago

🇺🇸 United States – Remote

💵 $200k - $300k / year

⏰ Full Time

🔴 Lead

🧬 Research Scientist

👻 Ghost score 5%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 22 days.

Apply Now

Find Similar Remote Jobs

Robots & Pencils

WebsiteLinkedInAll Job Openings

51 - 200 employees

🤖 Artificial Intelligence

🤝 B2B

☁️ SaaS

💰 Venture Round on 2022-04

Artificial Intelligence • B2B • SaaS

Robots & Pencils is a digital innovation firm that specializes in developing digital strategies and products utilizing mobile, web, and frontier technologies to drive transformation for businesses. They focus on areas such as mobile development, product design, UX research, artificial intelligence, machine learning, and organizational change management. With a commitment to blending creativity with technology, they aim to unlock data insights and enhance product innovation, ultimately delivering excellent customer experiences and helping brands gain a competitive edge.

📋 Description

• Lead the design and build of an evaluation framework for an enterprise planner-based agent • Benchmark planner models, executor LLMs and inference optimization levers with controlled, statistically sound experiments • Design the end-to-end evaluation workflow covering scenarios, user simulation, configuration, execution, grading, comparison and insights • Define scenario schemas, grading specifications and validators with the client team • Build an LLM-based user simulator for realistic multi-turn conversations and validate it against production logs • Build isolated, parallel experiment execution with configuration files and harness adapters using the client's LLM gateway • Package and implement optimization levers such as cascade-on-failure, cache-aware model switching and prompt caching • Tag model calls and send traces to attribute tokens, cache usage, latency and cost • Grade runs, compute success/cost/latency/failure metrics, and compare experiments using confidence intervals, paired methods and non-inferiority tests • Project dollar savings at production volume and build an evidence-linked automated insight engine • Build a locally hosted API-first dashboard with one-command deployment • Run the staged experiment campaign and analyze generalization by scenario category • Deliver final recommendations, runbooks and integration guides supporting go/no-go decisions • Partner with client engineers and leadership, communicate findings to technical and business stakeholders, write reports and mentor engineers

🎯 Requirements

• Experience with LLM and agent evaluation methods, including programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation • Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production • Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis • Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and projected dollar savings • Familiarity with LLM routing, model cascades, model selection and LLM inference internals and cost levers • Experience with observability and tracing for LLM systems, including attribution of tokens, cache usage, latency and cost per request • Ability to build production-quality research tooling using Python services, containerized and sandboxed execution, and data pipelines • Prior customer-facing or consulting experience scoping and delivering research engagements • Ability to explain technical findings to non-technical audiences and translate them into business impact • Helpful extras include publications/open-source contributions, dashboard or web UI experience, LLM gateway/tracing stack experience, regulated-domain experience, and production cost-quality tradeoff experience

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume