Machine Learning Engineer, Evaluation, Reliability

Job not on LinkedIn

🔥 6 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🤖 Machine Learning Engineer

👻 Ghost score 25%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 60 days.
• 102 open roles for a team of 16.
• No salary listed.

Apply Now

Find Similar Remote Jobs

Bask Health

WebsiteLinkedInAll Job Openings

11 - 50 employees

🏥 Healthcare

💼 Consulting

📦 Logistics

💰 Pre Seed Round on 2022-05

Healthcare • Consulting • Logistics

Bask Health is a robust platform providing a comprehensive suite of customizable telehealth solutions. It offers tools such as a questionnaire builder, patient portals, and pharmacy fulfillment services, all secured with strong encryption and HIPAA-compliance. Bask Health seamlessly integrates telehealth and e-commerce, enabling streamlined patient care from consultation to delivery. The platform supports entrepreneurs, doctors, and developers by offering scalable no-code solutions that enhance user experience and business growth. With its nationwide pharmacy network, Bask Health empowers businesses to efficiently manage prescriptions, patient data, and order fulfillment across the entire digital health landscape.

📋 Description

• Build and maintain evaluation systems for AI-powered applications across Bask Health. • Develop automated testing frameworks for large language models and AI application workflows. • Create and maintain representative datasets for evaluating model behavior. • Implement benchmarks measuring response accuracy, relevance, completeness, and consistency. • Develop evaluation pipelines for prompts, retrieval systems, and model configurations. • Test AI assistants and agents against realistic user scenarios and workflow requirements. • Build automated regression tests to detect performance changes between releases. • Measure hallucination rates, unsupported claims, and other model quality failures. • Evaluate retrieval-augmented generation systems for document relevance and answer grounding. • Develop tests for structured model outputs and schema compliance. • Evaluate tool-calling accuracy and agent action selection. • Test AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior. • Implement evaluation checks for sensitive patient and operational information. • Build dashboards and reports communicating AI quality metrics to engineering teams. • Analyze evaluation results and identify recurring failure patterns. • Investigate production incidents involving inaccurate or unreliable AI outputs. • Collaborate with applied AI engineers to improve prompts, retrieval logic, and application behavior. • Support model comparisons and controlled experiments. • Evaluate tradeoffs between model quality, latency, reliability, and inference cost. • Develop automated quality checks for CI/CD and AI feature release workflows. • Implement monitoring and alerting for production AI performance. • Build tools for reviewing, labeling, and analyzing model responses. • Help maintain evaluation datasets as products and user behavior evolve. • Document evaluation methodologies, test cases, and known model limitations. • Work with product teams to define measurable acceptance criteria for AI features. • Collaborate with senior engineers on evaluation standards and reliability improvements. • Participate in code reviews, technical discussions, and engineering planning. • Own assigned evaluation and reliability projects from implementation through deployment and ongoing improvement.

🎯 Requirements

• 3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field. • Experience building and maintaining production software or machine learning applications. • Strong programming skills in Python, TypeScript, or comparable languages. • Familiarity with large language models and modern generative AI applications. • Experience testing or evaluating machine learning models or AI-powered software. • Understanding of machine learning evaluation metrics and experimental methodologies. • Experience developing automated tests and validation pipelines. • Familiarity with prompt engineering and model output analysis. • Ability to analyze structured and unstructured datasets. • Experience writing scripts or services for data processing and evaluation. • Understanding of model accuracy, precision, recall, and other performance measures where applicable. • Familiarity with API integrations and backend application development. • Experience working with relational databases or analytical data systems. • Ability to identify patterns in model failures and translate findings into engineering improvements. • Understanding of regression testing and software quality assurance. • Familiarity with logging, monitoring, and production troubleshooting. • Experience using Git and collaborative software development workflows. • Understanding of the limitations and failure modes of large language models. • Ability to design representative test cases and evaluation scenarios. • Familiarity with cloud-based development and deployment environments. • Experience writing maintainable, well-tested production code. • Strong analytical thinking and attention to detail. • Ability to communicate technical evaluation results clearly. • Experience collaborating with engineers, product managers, and cross-functional teams. • Ability to independently own assigned technical projects.

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume