Software Quality Engineer – AI

🕒 March 4

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Boomi

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

Founded 2000

☁️ SaaS

🔌 API

🏢 Enterprise

SaaS • API • Enterprise

Boomi is a leading provider of intelligent integration and automation solutions, empowering organizations to streamline business processes and accelerate digital transformation. With a strong focus on integration platform as a service (iPaaS), Boomi enables the secure deployment and management of APIs, integration of applications and data, and automation of workflows using AI-powered solutions. Trusted by over 20,000 customers, Boomi's platform supports a wide array of industries including manufacturing, healthcare, financial services, and more. Their extensive library of pre-built connectors, user-friendly interface, and robust customer support foster seamless integration and connectivity across cloud and on-premise environments. Recognized for its innovation and excellence, Boomi transforms complex integration challenges into manageable solutions, contributing significantly to efficiency and productivity in business operations.

📋 Description

• Design and execute comprehensive test plans for backend services, APIs, and microservices architectures. • Partner with Backend Engineers, AI Engineers, and Product teams to understand requirements and identify quality risks early in the development cycle. • Develop automated testing frameworks using tools like Pytest, Playwright, unittest, and integration testing libraries. • Build end-to-end test suites that validate the integration between backend systems, AI agents, databases & services. • Advocate for quality-first practices, influencing architectural decisions and embedding testing into the development lifecycle. • Design and implement automated evaluation frameworks for Generative AI features, including LLM / SLM model testing, prompt/ output validation, and behavioral assessment of agentic workflows. • Develop quality metrics and evaluation methodologies for LLM-based applications, assessing accuracy, consistency, reliability, quality of AI/ML models. • Create and maintain curated evaluation datasets and synthetic test data that cover edge cases, adversarial scenarios, and real-world variability. • Leverage AI evaluation tools and frameworks & establish observability and monitoring requirements to quality assessment. • Establish observability and monitoring requirements including structured logging, metrics collection, and tracing for both backend services and AI agent behavior. • Implement quality gates and acceptance criteria and create test documentation including test plans, automation architecture, quality reports, and runbooks. • Build strategies for testing in cloud environments & containerized execution. • Implement performance testing, load testing, and reliability testing for production backend services and AI inference pipelines. • Define and champion quality standards, best practices, and testing methodologies for both traditional backend systems and AI applications. • Conduct code reviews with a focus on testability, quality patterns, and maintainability. • Mentor junior quality engineers, sharing expertise in automation frameworks, AI testing approaches, and quality engineering principles. • Communicate quality insights, risk assessments, and test results effectively to technical and non-technical stakeholders using tools like Jira and Confluence.

🎯 Requirements

• 2.5+ years of experience in software quality engineering, test automation, or software development with strong QA focus. • Deep expertise in test automation frameworks and tools such as pytest, unittest, Selenium, Playwright etc. • Strong programming skills in Python or Java with ability to write production-quality test code and automation frameworks. • Proven track record building automated testing infrastructure for backend services and APIs. • Experience with various testing methodologies: unit testing, integration testing, end-to-end testing, contract testing, performance testing, and test-driven development (TDD). • Strong understanding of CI/CD pipelines and experience integrating automated tests into deployment workflows. • Comfortable working in Linux/Unix environments for test execution and debugging. • Familiarity with testing AI/ML applications or LLM-based systems, with understanding of unique quality challenges in non-deterministic systems. • Knowledge of Generative AI concepts, agentic frameworks (ReAct, chain-of-thought, tool use), and LLM capabilities/limitations. • Exposure to agentic AI patterns and knowledge of how to test tool orchestration, multi-turn conversations, and RAG systems. • Excellent written and verbal communication skills with ability to document findings clearly and comprehensively. • Strong collaboration skills working across cross-functional teams in Agile/Scrum environments. • Experience with collaboration tools like Jira, Confluence, and Git for team workflows.

🏖️ Benefits

• Health insurance • 401(k) matching • Flexible work hours • Paid time off • Remote work options