ML Platform Engineer – Evaluation Systems

Job not on LinkedIn

🔥 11 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🏗️ Platform Engineer

👻 Ghost score 25%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 60 days.
• 102 open roles for a team of 16.
• No salary listed.

Apply Now

Find Similar Remote Jobs

Bask Health

WebsiteLinkedInAll Job Openings

11 - 50 employees

🏥 Healthcare

💼 Consulting

📦 Logistics

💰 Pre Seed Round on 2022-05

Healthcare • Consulting • Logistics

Bask Health is a robust platform providing a comprehensive suite of customizable telehealth solutions. It offers tools such as a questionnaire builder, patient portals, and pharmacy fulfillment services, all secured with strong encryption and HIPAA-compliance. Bask Health seamlessly integrates telehealth and e-commerce, enabling streamlined patient care from consultation to delivery. The platform supports entrepreneurs, doctors, and developers by offering scalable no-code solutions that enhance user experience and business growth. With its nationwide pharmacy network, Bask Health empowers businesses to efficiently manage prescriptions, patient data, and order fulfillment across the entire digital health landscape.

📋 Description

• Design, build, and maintain machine learning evaluation infrastructure across Bask Health • Develop platform services that execute and manage automated model evaluation workloads • Build scalable systems for large batches of model and AI application evaluations • Implement job orchestration, scheduling, and execution management for evaluation pipelines • Develop infrastructure for storing and versioning evaluation datasets • Build systems for tracking model configurations, prompts, parameters, and evaluation results • Implement reproducible experiment workflows for consistent model comparisons • Develop automated pipelines for benchmarking machine learning models • Build APIs and services for submitting and monitoring evaluation jobs • Create infrastructure for comparing evaluation results across model versions and experiments • Support evaluation workflows for large language models, predictive models, and retrieval systems • Develop asynchronous processing systems for computationally intensive evaluation tasks • Implement monitoring for job performance, failures, latency, and resource usage • Build logging and tracing capabilities for investigating evaluation problems • Develop systems for evaluation artifacts, reports, and historical results • Integrate evaluation infrastructure with model registries and ML development workflows • Implement automated quality checks for controlled model releases • Build retry, timeout, and failure recovery mechanisms • Optimize evaluation performance, infrastructure utilization, and operational costs • Support distributed evaluation workloads across cloud-based compute resources • Collaborate with ML engineers and applied scientists on infrastructure requirements and reproducible experiments • Develop tools simplifying evaluation configuration and execution • Implement secure access controls for evaluation datasets and results • Ensure tenant boundaries and approved data-handling practices are respected • Investigate infrastructure incidents and improve platform reliability • Document infrastructure, deployment procedures, and operational standards • Own ML evaluation platform projects from technical design through deployment, monitoring, and continuous improvement

🎯 Requirements

• 3+ years of professional experience in software engineering, machine learning engineering, platform engineering, or a closely related technical field • Experience building and maintaining production backend services or infrastructure • Strong programming skills in Python, TypeScript, or comparable languages • Familiarity with machine learning development and evaluation workflows • Experience developing APIs and backend service integrations • Understanding of distributed systems and asynchronous processing • Experience working with cloud-based infrastructure and production deployments • Familiarity with automated testing and continuous integration practices • Understanding of machine learning model versioning and experiment reproducibility • Experience working with relational databases and structured datasets • Familiarity with data pipelines and batch processing workflows • Experience designing reliable background job systems • Understanding of infrastructure scalability, performance, and fault tolerance • Familiarity with logging, monitoring, and production observability • Experience implementing retries, timeouts, and error recovery • Understanding of authentication, authorization, and secure service communication • Familiarity with configuration management and deployment automation • Experience writing maintainable, well-tested production code • Ability to troubleshoot complex backend or infrastructure issues • Understanding of software release processes and operational reliability • Experience using Git and collaborative engineering workflows • Ability to translate machine learning requirements into scalable infrastructure • Strong analytical thinking and problem-solving skills • Ability to collaborate effectively with machine learning engineers, applied scientists, and software engineers • Ability to independently own engineering projects from design through deployment • Nice to have: Experience building infrastructure for machine learning evaluation or experimentation • Nice to have: Familiarity with MLflow, Weights & Biases, or comparable experiment tracking platforms • Nice to have: Experience with model registries and artifact management systems • Nice to have: Familiarity with large language model evaluation frameworks • Nice to have: Experience with LangSmith, DeepEval, Ragas, or comparable AI evaluation tools • Nice to have: Experience building distributed evaluation or benchmarking pipelines • Nice to have: Familiarity with workflow orchestration tools such as Airflow, Dagster, or Temporal • Nice to have: Experience with containerized workloads and distributed compute infrastructure • Nice to have: Familiarity with Docker, Kubernetes, or comparable technologies • Nice to have: Experience working with AWS Lambda, SQS, EventBridge, or related cloud services • Nice to have: Familiarity with AWS SageMaker or comparable ML platform services • Nice to have: Experience with serverless infrastructure and infrastructure-as-code tools • Nice to have: Familiarity with SST or comparable application deployment frameworks • Nice to have: Experience developing data versioning and dataset management systems • Nice to have: Understanding of reproducible machine learning experimentation • Nice to have: Experience implementing model performance monitoring and drift detection • Nice to have: Familiarity with large language model APIs and model comparison workflows • Nice to have: Experience building internal developer platforms for machine learning teams • Nice to have: Understanding of HIPAA and healthcare data privacy considerations • Nice to have: Experience using AI-assisted development tools such as Cursor or Claude Code

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume