Senior Machine Learning Engineer, Evaluation, Reliability

Job not on LinkedIn

🔥 4 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟠 Senior

🤖 Machine Learning Engineer

👻 Ghost score 25%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 60 days.
• 54 open roles for a team of 16.
• No salary listed.

Apply Now

Find Similar Remote Jobs

Bask Health

WebsiteLinkedInAll Job Openings

11 - 50 employees

🏥 Healthcare

💼 Consulting

📦 Logistics

💰 Pre Seed Round on 2022-05

Healthcare • Consulting • Logistics

Bask Health is a robust platform providing a comprehensive suite of customizable telehealth solutions. It offers tools such as a questionnaire builder, patient portals, and pharmacy fulfillment services, all secured with strong encryption and HIPAA-compliance. Bask Health seamlessly integrates telehealth and e-commerce, enabling streamlined patient care from consultation to delivery. The platform supports entrepreneurs, doctors, and developers by offering scalable no-code solutions that enhance user experience and business growth. With its nationwide pharmacy network, Bask Health empowers businesses to efficiently manage prescriptions, patient data, and order fulfillment across the entire digital health landscape.

📋 Description

• Design and build evaluation infrastructure for Bask Health’s AI and machine learning applications • Develop automated testing systems for LLM applications and intelligent workflows • Establish measurable quality standards and evaluation pipelines assessing accuracy, relevance, completeness, consistency, and reliability • Build and maintain representative benchmark datasets and regression testing systems • Design evaluation methods for retrieval-augmented generation and knowledge-grounded AI systems • Measure retrieval quality, grounding accuracy, and unsupported model outputs • Build tools to identify hallucinations, incorrect reasoning, and unreliable responses • Develop testing approaches for AI agents, tool calling, and multi-step workflows • Implement automated checks for structured outputs, API interactions, and tool execution • Build production monitoring systems, dashboards, and reporting tools for model quality, latency, errors, and operational reliability • Investigate production AI failures and identify underlying technical causes • Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations • Establish testing procedures for model upgrades and provider changes • Develop benchmarking tools comparing models by quality, performance, and cost • Implement evaluation processes for sensitive healthcare-related AI workflows • Work with product and clinical stakeholders to define acceptance criteria and escalation requirements • Support human review and feedback collection for AI outputs requiring additional validation • Design reproducible experiments and build reusable evaluation libraries, services, and internal tools • Integrate AI evaluation checks into development and deployment workflows • Improve observability through structured logging, tracing, and production diagnostics • Mentor engineers on AI testing methodologies and reliability practices • Contribute to AI infrastructure, testing, and monitoring architecture discussions • Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance

🎯 Requirements

• 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline • Demonstrated experience developing and operating machine learning or AI systems in production • Strong programming skills in Python and experience building production software services • Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications • Strong understanding of machine learning evaluation methodologies and performance measurement • Experience developing reproducible experiments and benchmark datasets • Familiarity with large language models and modern LLM application architectures • Understanding of prompt engineering, model inference, and AI application integration patterns • Experience evaluating model outputs for quality, accuracy, and reliability • Familiarity with retrieval-augmented generation and retrieval evaluation methods • Experience with structured datasets, data pipelines, and analytical workflows • Strong understanding of software testing, regression testing, and continuous integration • Experience building or maintaining backend services and APIs • Familiarity with cloud infrastructure and production deployment environments • Experience with monitoring, logging, tracing, and production troubleshooting • Understanding of model latency, throughput, and operational performance metrics • Ability to investigate complex system failures and identify root causes • Experience designing metrics and reporting systems for technical performance • Familiarity with version control, code review, and collaborative software development • Understanding of data quality and its impact on model evaluation • Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost • Strong analytical skills and attention to detail • Experience collaborating with engineers, product managers, and technical stakeholders • Ability to independently own complex engineering initiatives • Strong communication skills and the ability to explain evaluation results clearly • Nice-to-have qualifications include experience with LLM evaluation frameworks, LangSmith, DeepEval, Ragas, LLM-as-a-judge methods, hallucination detection, agent evaluation, tool-calling evaluation, retrieval metrics, vector databases, embeddings, semantic search, MLflow, Weights & Biases, observability infrastructure, model drift detection, CI/CD evaluation checks, distributed processing, AWS, AI developer tools, healthcare applications, HIPAA, human-in-the-loop evaluation, Cursor, and Claude Code

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume