Senior Applied AI Engineer, Agent Quality & Evaluations

Job not on LinkedIn

🔥 1 minute ago

🏄 California – Remote

🏄 California – Remote

💵 $150k - $250k / year

⏰ Full Time

🟠 Senior

🤖 AI Engineer

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Flodesk

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2017

☁️ SaaS

🛍️ eCommerce

📣 Marketing

SaaS • eCommerce • Marketing

Flodesk is a company that provides simple and intuitive software tools for email marketing, allowing entrepreneurs and small businesses to grow their business. The platform enables users to design and send on-brand marketing emails, create sales pages, accept online payments, and utilize opt-in forms to grow their lists. Featuring beautiful, customizable templates and advanced automation capabilities, Flodesk aims to simplify the marketing process. It also offers robust analytics to understand the performance of campaigns and connections to various third-party tools via API. Flodesk caters to businesses of all sizes with options for custom plans and provides resources through Flodesk University for training on the platform.

📋 Description

• Manage member-facing AI behavior across prompt-to-create, agentic editing and future jobs such as segmentation, scheduling, analytics and recommendations • Create, test, version and document prompts and agent instructions • Define system use of context, member data, tools and product state • Build and prototype agent loops, tools and structured interfaces • Own AI evaluation practice, including representative datasets, behavioral scenarios, scoring rubrics, regression suites and human review • Build and operate evaluation harnesses, monitoring and quality dashboards • Diagnose failures across prompts, context, orchestration, models, tools, data and product code • Set quality baselines and release gates for prompt, model and agent changes • Turn production failures, traces and member feedback into durable regression cases • Partner with product, design, marketing and copy experts to encode Flodesk's point of view into the system

🎯 Requirements

• A track record of building or meaningfully improving production LLM or agentic products, not just prototypes • Strong software engineering skills in Python, TypeScript or a similar language • Understanding of the interaction between prompts, context, tools, agent loops, orchestration and product state • Direct experience with LLM evaluation, observability and experimental design • Ability to combine automated checks with human judgment • Strong product sense for translating ambiguous member feedback into testable quality definitions and concrete system changes • Systems thinking and reusable patterns • Clear communication across technical and nontechnical disciplines • Bonus: experience building content-generation, creative or marketing tools • Bonus: experience building multi-turn agents that use tools and preserve user intent across edits • Bonus: experience with Langfuse, Braintrust or comparable evaluation and observability platforms • Bonus: experience working on products for small businesses, creators or marketers

🏖️ Benefits

• Fully paid health insurance for individual coverage • 16 weeks paid parental leave for non-birthing parents • 22 weeks paid maternity leave for birthing parents • Unlimited flexible time off • 401(k) match (US employees only) • $1,000 annual stipend for learning and development