Clinical AI Evaluation Specialist

Job not on LinkedIn

🕒 April 30

Apply Now

Find Similar Remote Jobs

Receive Emails For Remote Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

CentralReach

WebsiteLinkedInAll Job Openings

201 - 500 employees

Founded 2010

💰 Private equity on 2018-03

CentralReach provides the leading Autism and IDD Care software and services platform to help children and adults diagnosed with autism and related intellectual and developmental disabilities - and those who serve them - unlock potential, achieve better outcomes, and live more independent lives.

📋 Description

• Contribute to evaluation frameworks for clinical AI products, including defining acceptance criteria, test plans, clinically relevant rubrics, and performance benchmarks. • Conduct structured sampling reviews of AI-generated outputs, assessing across criteria alignment with ABA principles and BACB ethical standards. • Design and implement monitoring automation in collaboration with engineering, including automated quality checks, alerting thresholds, and drift detection for AI systems. • Contribute to the creation and maintenance of reusable governance artifacts including system cards, monitoring playbooks, evaluation of SOP templates, and risk assessment documentation. • Assist with pre-deployment validation testing for new AI products and features, executing defined testing and results of documentation. • Participate in risk tiering assessments for new and evolving AI products, providing clinical and evaluative perspectives. • Support incident investigation for AI-related issues if/as needed and in alignment with clinical best practice and CR Development Policy. • Collaborate with product management and engineering teams to provide input during product planning and development, ensuring governance considerations are integrated early. • Contribute to the development of evaluation methodology and standards for clinical AI use cases, documenting approaches and refining processes based on findings and emerging best practices. • Contribute to AI literacy and training content for internal teams and customers, drawing on hands-on clinical evaluation experience. • Stay current with developments in AI evaluation methodology, LLM behavior, ABA best practices, and relevant regulatory and ethical standards. • Support CentralReach's thought leadership in responsible AI through contributions to conference presentations, professional publications, or applied research as opportunities arise.

🎯 Requirements

• Board Certified Behavior Analyst (BCBA) in good standing • 2+ years of demonstrated experience in quality assurance, evaluation, data analysis, or technical assessment roles, preferably within healthcare or behavioral health settings. • Demonstrated experience with large language models, generative AI, and their applications and limitations in clinical or professional contexts. • Experience with prompt/context engineering, AI output evaluation, or AI workflow design. • Experience with statistical sampling methodologies or healthcare quality assurance frameworks. • Familiarity with AI governance frameworks (NIST AI RMF, EU AI Act risk categories, AMA governance guidance). • Experience developing evaluation rubrics, scoring frameworks, or quality measurement tools. • Background in clinical informatics, research methodology, or systematic quality improvement. • Demonstrated proficiency with data analysis tools and comfort working with quantitative and qualitative evaluation data. • Strong analytical skills with the ability to develop systematic, evidence-based evaluation approaches. • Familiarity with HIPAA compliance, healthcare data standards, and ethical considerations in AI. • Excellent written and verbal communication skills with the ability to document methodology clearly and convey findings to clinical, technical, and leadership audiences. • Demonstrated ability to work collaboratively across clinical, technical, and product teams. • Experience in a remote work environment with strong self-direction and accountability.

🏖️ Benefits

• Health benefits • Generous PTO • 401(k) matching • Paid parental leave • Hybrid work schedules • Career development support • Wellness programs