Staff Applied AI Engineer – Product & Agent Performance

Job not on LinkedIn

🔥 0 minutes ago

🇺🇸 United States – Remote

💵 $175k - $200k / year

⏰ Full Time

🔴 Lead

🤖 AI Engineer

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 1%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Arcadia

WebsiteLinkedInAll Job Openings

201 - 500 employees

🏥 Healthcare

💼 Consulting

💰 $29.5M Venture Round on 2020-01

Healthcare • Consulting

Arcadia is dedicated to happier, healthier days for all. We transform diverse data into a unified fabric for health. Our platform delivers actionable insights for our customers to advance care and research, drive strategic growth, and achieve financial success.

📋 Description

• Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks • Design retrieval and context architecture to keep agents grounded in real data • Design memory and state handling across multi-turn and multi-agent flows • Create context and prompt templates using few-shot examples, structured formatting, and reasoning scaffolding • Improve performance through prompting, tool-use strategy, and context construction • Build and run evaluations against real production conditions • Author evaluation rubrics, quality heuristics, and severity- and cost-weighted thresholds • Design and validate escalation paths to human review based on confidence and uncertainty • Optimize cost-aware performance alongside latency, reliability, and accuracy • Evaluate and sign off on model changes and make go/no-go launch decisions • Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes • Partner with Product and product managers to make agents steerable, trustworthy, and ready to scale • Establish production-grounded baselines, document failure modes, and create measurement plans • Map retrieval, context, memory, and escalation patterns and identify reliability, calibration, and cost improvements • Turn production evidence into actionable recommendations for Product and Engineering

🎯 Requirements

• Equivalent practical experience demonstrating the depth required for this staff-level role • 8+ years of production software engineering experience • 3+ years of hands-on ownership of ML, LLM, or agentic systems in production • Direct experience in healthcare, finance, or another regulated industry • Ability to diagnose why an agent failed and attribute fixes to instruction, retrieval, context, or memory design • Ability to weigh failures by severity and cost rather than frequency alone • Hands-on experience with RAG architecture • Experience with production-grounded evaluation frameworks • Experience with fallback or human-in-the-loop logic for automated systems • Working familiarity with AWS AI/ML services, including Bedrock and SageMaker • Experience applying AI to healthcare data or workflows where safety, transparency, and calibrated uncertainty affect care teams or patients (preferred) • Experience with long-horizon, multi-turn or multi-agent workflows and product-level AI documentation such as model cards (preferred)

🏖️ Benefits

• The opportunity to define how agent performance, safety, and readiness are measured for production healthcare workflows • Meaningful ownership across prompts, context, memory, evaluations, and escalation patterns at product scale • A cross-functional role translating production evidence into AI improvements used across Arcadia’s platform • A mission-driven company working to improve how patients receive care • A flexible, remote-friendly culture with personality and heart • Employee-driven programs and initiatives for personal and professional development • Membership in the talented, energized, diverse, and purpose-driven Arcadian community