Software Engineer, AI Quality – Evaluation

Job not on LinkedIn

🔥 11 minutes ago

🇺🇸 United States – Remote

⏰ Full Time

🟡 Mid-level

🟠 Senior

🤖 AI Engineer

👻 Ghost score 25%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.Why this score:

• Just posted.
• Jobs at this company usually fill in 60 days.
• 102 open roles for a team of 16.
• No salary listed.

Apply Now

Find Similar Remote Jobs

Bask Health

WebsiteLinkedInAll Job Openings

11 - 50 employees

🏥 Healthcare

💼 Consulting

📦 Logistics

💰 Pre Seed Round on 2022-05

Healthcare • Consulting • Logistics

Bask Health is a robust platform providing a comprehensive suite of customizable telehealth solutions. It offers tools such as a questionnaire builder, patient portals, and pharmacy fulfillment services, all secured with strong encryption and HIPAA-compliance. Bask Health seamlessly integrates telehealth and e-commerce, enabling streamlined patient care from consultation to delivery. The platform supports entrepreneurs, doctors, and developers by offering scalable no-code solutions that enhance user experience and business growth. With its nationwide pharmacy network, Bask Health empowers businesses to efficiently manage prescriptions, patient data, and order fulfillment across the entire digital health landscape.

📋 Description

• Design, build, and maintain software infrastructure supporting AI quality and evaluation across Bask Health • Develop backend services that execute and manage automated AI evaluations • Build reusable testing tools for large language model applications • Create APIs for submitting evaluation jobs and retrieving results • Develop systems for evaluation datasets, test cases, and expected outcomes • Build automated workflows comparing prompts, models, and application configurations • Implement AI regression testing infrastructure and CI/CD evaluation checks • Record, organize, store, and compare evaluation results over time • Build tools, dashboards, interfaces, and libraries for reviewing AI outcomes and simplifying testing • Support deterministic checks, model-based evaluations, tool-calling behavior, structured outputs, and retrieval-augmented generation applications • Build asynchronous processing, scheduling, retries, timeouts, and error handling for evaluation workloads • Develop monitoring and logging; optimize speed, reliability, and infrastructure costs • Collaborate with AI evaluation engineers, machine learning engineers, and backend engineers to translate testing requirements into reusable tools • Implement access controls, secure handling of evaluation data, tenant boundaries, and privacy requirements • Investigate infrastructure failures and resolve reliability issues • Improve developer documentation and internal AI testing tools • Participate in code reviews, technical planning, and architecture discussions • Own AI quality engineering projects from design through deployment, monitoring, and continuous improvement

🎯 Requirements

• 3+ years of professional software engineering experience, preferably in backend, platform, infrastructure, or developer tooling roles • Experience building and maintaining production backend services • Strong programming skills in TypeScript, JavaScript, Python, or comparable languages • Experience designing and developing APIs and service integrations • Familiarity with automated testing frameworks and software quality practices • Understanding of continuous integration and deployment workflows • Experience building reusable software components or internal engineering tools • Familiarity with large language models and modern AI application development • Understanding of the challenges involved in testing nondeterministic AI systems • Experience working with relational databases and structured application data • Familiarity with asynchronous processing and background job systems • Understanding of distributed systems fundamentals and service reliability • Experience developing automated testing pipelines • Familiarity with cloud-based application deployment • Understanding of authentication, authorization, and secure service communication • Experience implementing logging, monitoring, and production diagnostics • Ability to troubleshoot backend services and infrastructure issues • Familiarity with configuration management and software release processes • Experience writing maintainable, well-tested production code • Understanding of API error handling, retries, and failure recovery • Experience working with Git and collaborative engineering workflows • Ability to translate technical requirements into reusable engineering systems • Ability to collaborate effectively with software engineers, AI engineers, and machine learning specialists • Ability to independently own engineering projects from technical design through deployment • Nice to have: Experience building infrastructure for AI evaluation or machine learning testing • Nice to have: Familiarity with OpenAI, Anthropic, or comparable model APIs • Nice to have: Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools • Nice to have: Familiarity with LLM-as-a-judge evaluation approaches • Nice to have: Experience building evaluation execution services or automated benchmark systems • Nice to have: Familiarity with AI regression testing and prompt comparison workflows • Nice to have: Experience developing internal developer platforms or SDKs • Nice to have: Experience building testing infrastructure for distributed applications • Nice to have: Familiarity with TypeScript and Node.js backend development • Nice to have: Experience with React or Next.js for internal tooling interfaces • Nice to have: Familiarity with tRPC, Prisma, Drizzle, or comparable development tools • Nice to have: Experience working with AWS Lambda, SQS, EventBridge, or related cloud services • Nice to have: Familiarity with serverless infrastructure and infrastructure-as-code tools • Nice to have: Experience with SST or comparable application deployment frameworks • Nice to have: Experience building scalable asynchronous job processing systems • Nice to have: Familiarity with embeddings, vector databases, and retrieval-augmented generation • Nice to have: Experience implementing model observability or AI request tracing • Nice to have: Understanding of multi-tenant SaaS architecture and data isolation • Nice to have: Familiarity with HIPAA and healthcare data privacy considerations • Nice to have: Experience using AI-assisted development tools such as Cursor or Claude Code

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume