Staff Applied AI Engineer – Product & Agent Performance

πŸ•’ August 20

Where this job comes from

Posted on Arcadia's careers page β†—Arcadia's careers page, not through a job board

First seen by Remote Rocketship August 20

Still open on Arcadia's site. Checked 3 hours ago.

Apply Now
Logo of Arcadia

Arcadia

201 - 500 employees

πŸ₯ Healthcare

πŸ’Ό Consulting

πŸ’° $29.5M Venture Round on 2020-01

Healthcare β€’ Consulting

Arcadia is dedicated to happier, healthier days for all. We transform diverse data into a unified fabric for health. Our platform delivers actionable insights for our customers to advance care and research, drive strategic growth, and achieve financial success.

πŸ“‹ Description

β€’ Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks β€’ Design retrieval and context architecture to keep agents grounded in real data β€’ Design memory and state handling across multi-turn and multi-agent flows β€’ Create context and prompt templates using few-shot examples, structured formatting, and reasoning scaffolding β€’ Improve performance through prompting, tool-use strategy, and context construction β€’ Build and run evaluations against real production conditions β€’ Author evaluation rubrics, quality heuristics, and severity- and cost-weighted thresholds β€’ Design and validate escalation paths to human review based on confidence and uncertainty β€’ Optimize cost-aware performance alongside latency, reliability, and accuracy β€’ Evaluate and sign off on model changes and make go/no-go launch decisions β€’ Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes β€’ Partner with Product and product managers to make agents steerable, trustworthy, and ready to scale β€’ Establish production-grounded baselines, document failure modes, and create measurement plans β€’ Map retrieval, context, memory, and escalation patterns and identify reliability, calibration, and cost improvements β€’ Turn production evidence into actionable recommendations for Product and Engineering

🎯 Requirements

β€’ Equivalent practical experience demonstrating the depth required for this staff-level role β€’ 8+ years of production software engineering experience β€’ 3+ years of hands-on ownership of ML, LLM, or agentic systems in production β€’ Direct experience in healthcare, finance, or another regulated industry β€’ Ability to diagnose why an agent failed and attribute fixes to instruction, retrieval, context, or memory design β€’ Ability to weigh failures by severity and cost rather than frequency alone β€’ Hands-on experience with RAG architecture β€’ Experience with production-grounded evaluation frameworks β€’ Experience with fallback or human-in-the-loop logic for automated systems β€’ Working familiarity with AWS AI/ML services, including Bedrock and SageMaker β€’ Experience applying AI to healthcare data or workflows where safety, transparency, and calibrated uncertainty affect care teams or patients (preferred) β€’ Experience with long-horizon, multi-turn or multi-agent workflows and product-level AI documentation such as model cards (preferred)

πŸ–οΈ Benefits

β€’ The opportunity to define how agent performance, safety, and readiness are measured for production healthcare workflows β€’ Meaningful ownership across prompts, context, memory, evaluations, and escalation patterns at product scale β€’ A cross-functional role translating production evidence into AI improvements used across Arcadia’s platform β€’ A mission-driven company working to improve how patients receive care β€’ A flexible, remote-friendly culture with personality and heart β€’ Employee-driven programs and initiatives for personal and professional development β€’ Membership in the talented, energized, diverse, and purpose-driven Arcadian community

Apply Now

πŸ“Š Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Similar Jobs

πŸ•’ August 19

KIS Solutions

51 - 200

πŸ’Ό Consulting

πŸ“£ Marketing

πŸ“¦ Logistics

Tech Lead designing and evolving LLM-powered chatbot architectures for an IT services company. Leading AI systems, delivery, engineering mentorship, and product collaboration.

πŸ•’ August 18

STARLIMS

201 - 500

🍽️ Food & Beverage

πŸ“¦ Logistics

🏭 Manufacturing

Senior AI Engineer building reliable, observable agentic systems for STARLIMS laboratory software. Developing runtimes, production agents, evaluation harnesses, safeguards, and enterprise integrations.

πŸ•’ August 17

Toast

1001 - 5000

🍽️ Food & Beverage

πŸ’Ό Consulting

πŸ“¦ Logistics

Staff/Senior Software Engineer building scalable Voice AI infrastructure for Toast’s restaurant platform. Leading AI integrations, reliability, observability, and cross-team technical initiatives.

πŸ•’ August 14

Arketa

11 - 50

πŸ’Ό Consulting

πŸ“£ Marketing

🧘 Wellness

Staff Software Engineer building customer-facing AI for Arketa, an operating system for modern fitness and wellness businesses. Modernizing architecture, infrastructure, reliability, and developer workflows across the stack.

πŸ•’ August 13

Cornelis Networks

51 - 200

πŸ€– Artificial Intelligence

πŸ”§ Hardware

🏒 Enterprise

Senior Software Engineer optimizing MPI, SHMEM, and collective communication middleware for Cornelis Networks’ AI/HPC interconnects. Improving performance, integration, and open-source enablement across customer workloads.

Built byΒ Lior Neu-ner. I'd love to hear your feedback β€” Get in touch via DM or support@remoterocketship.com
Follow Remote Rocketship on LinkedInFollow Remote Rocketship on InstagramFollow Remote Rocketship on FacebookFollow Remote Rocketship on TikTok