Director, AI Automation Engineering

🔥 13 hours ago

🗽 New York – Remote

🗽 New York – Remote

💵 $185k - $210k / year

⏰ Full Time

🔴 Lead

🤖 QA Automation Engineer

👻 Ghost score 6%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Impiricus

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2020

🏥 Healthcare

💼 Consulting

🍽️ Food & Beverage

💰 $3M Seed Round on 2022-04

Healthcare • Consulting • Food & Beverage

Impiricus is a next-generation, proprietary HCP engagement platform designed to facilitate communication between healthcare providers (HCPs) and the pharmaceutical industry. Founded by physicians, it addresses diminishing access to HCPs by providing relevant industry news and resources directly at their fingertips. With a robust API-centric framework, the platform enables partners to track engagement, analyze data, and deliver high-value content efficiently.

📋 Description

• Own infrastructure that continuously verifies whether autonomous agents take intended actions before and after production • Stage, test, and lifecycle-manage bots and agent workflows • Establish baseline quality metrics and coverage standards for AI initiatives • Build infrastructure to run automated research chains and agent evaluations at scale • Create automated evaluation frameworks for LLM outputs covering accuracy, safety, bias, and regression • Implement RAG evaluation, prompt regression testing, and golden-set benchmarking • Define and enforce quality thresholds before release • Partner with AI Researchers and Engineers on evaluation design for new model and agent patterns • Build release-gating infrastructure, including staged rollouts and rollback authority • Develop AI automation solutions to maintain software quality across the product portfolio • Integrate testing into GitHub Actions or equivalent for AI model and agent releases • Maintain test suites as models, prompts, and tools change • Collaborate with AI Solutions Architects, AI Engineers, Principal AI Product, and product teams on evaluation, launch readiness, QA, and feature validation • Document standards, runbooks, and patterns so non-engineers building in Sage can self-serve basic testing • Set technical standards and grow into leading a small automation team

🎯 Requirements

• 5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety • Direct experience evaluating LLM outputs, including RAG evaluation, prompt regression, safety, and accuracy benchmarking • Experience building CI/CD pipelines with integrated test gates using GitHub Actions or similar • Comfortable working across AI platform infrastructure and product-facing AI features • Ability to operate independently as a senior IC and communicate quality standards to engineers and non-engineers • Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus • Familiarity with LLM observability and evaluation tooling such as LangSmith, Braintrust, or custom harnesses is a strong plus • Background in healthcare, pharma, or regulated environments is a strong plus • Prior experience establishing a QA or automation function from scratch, with appetite to grow into leading a team • Proficiency in Python and standard testing frameworks such as pytest or equivalent

🏖️ Benefits

• Medical, dental, and vision coverage for you and your dependents • On-demand healthcare concierge • Pre-tax HSA, FSA, and DCFSA savings options • Monthly employer HSA contributions if enrolled in a high-deductible plan • 100% paid short- and long-term disability insurance • Life and AD&D insurance • Flexible vacation policy • Paid parental leave after 6 months • Remote work option • 401(k) with company match