Senior Software Development Engineer in Test, QA - AI Systems

🔥 0 minutes ago

🇨🇦 Canada – Remote

⏰ Full Time

🟠 Senior

⚙️ Software Development Engineer in Test (SDET)

👻 Ghost score 25%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Nexxa.ai

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2024

🏗️ Construction

🏭 Manufacturing

📦 Logistics

Construction • Manufacturing • Logistics

Nexxa. ai is an industrial AI company building agentic automation for industrial operations. Its platform learns from engineers and employees to take over complex, repetitive workflows, autonomously setting goals and execution criteria to increase productivity and resilience. Delivered as an integrated solution that works with existing technology stacks, Nexxa. ai positions itself as a SaaS-like enterprise tool that generates immediate ROI for operations teams.

📋 Description

• Own quality for Nexxa’s AI agent systems through automated and manual testing • Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM-based agents • Build automated evaluation and test pipelines integrated into CI/CD quality gates • Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs • Implement API, contract, integration, and end-to-end tests for services, tools, and orchestration layers • Develop performance, load, reliability, failure-injection, and resilience tests • Perform exploratory testing of agent behavior across ambiguous requests, multi-turn sessions, and unusual operational scenarios • Review agent traces and outputs for reasoning quality, action appropriateness, and safety • Execute manual regression and release verification in staging and customer-like environments • Conduct manual red-teaming for prompt injection, jailbreaks, tool misuse, and unsafe actions • Support UAT and pilot rollouts with Forward Deployed Engineers • Write test plans, test charters, bug reports, and manual test documentation • Create and manage golden datasets and labeled test sets with versioning and drift detection • Design rubrics and conduct human-in-the-loop evaluations • Define quality metrics such as groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations • Build dashboards to track quality metrics over time • Test full agent action loops, including planning, tool selection, execution, recovery, and final output • Investigate and triage failures across models, prompts, tools/APIs, and orchestration logic • Establish quality bars, release criteria, and sign-off processes • Decide what to automate and continually shift repeatable manual work into automation • Mentor engineers on testing strategies and test-driven practices for probabilistic LLM systems • Advocate for testability, observability, and tracing in agent architecture • Collaborate with ML engineers, backend engineers, and Forward Deployed Engineers

🎯 Requirements

• 5+ years as an SDET, QA engineer, or software engineer with a strong testing focus • Demonstrated ownership of test strategy and automation for complex distributed systems • Proven experience in both manual and automated testing, including exploratory testing, test planning, and high-quality bug reports • Strong programming skills, with Python preferred • Experience building test frameworks and infrastructure from the ground up • Hands-on experience with CI/CD systems such as GitHub Actions, GitLab CI, or Jenkins • Experience with containerized test environments, including Docker and Kubernetes • Experience testing APIs and microservices, including contract, integration, and end-to-end testing • Hands-on experience testing LLM-based products, chatbots, or AI agents • Understanding of generative-system testing and limitations of deterministic assertions • Practical experience with eval frameworks or equivalent self-built tooling • Understanding of LLM agents, including prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks • Experience designing test data and labeled datasets, including sourcing, sampling, and dataset drift management • Familiarity with LLM-specific failure modes, including hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism • Comfort defining correctness where no single right answer exists • Strong written communication skills • Preferred: experience with human-in-the-loop evaluation workflows • Preferred: performance/load testing and chaos/fault-injection testing • Preferred: ML/data science background sufficient to read model evaluations and statistical significance • Preferred: ML systems red-teaming or adversarial/security testing • Preferred: observability and tracing tools for LLM applications and general observability stacks • Preferred: industrial, IoT, or operational technology testing experience • Preferred: UAT, pilot, or customer-facing validation experience • Preferred: startup or fast-moving-team experience setting up test and evaluation infrastructure from scratch

🏖️ Benefits

• Comprehensive salary and equity package • Significant opportunities for career development and advancement • Innovative environment focused on transforming heavy industries through AI and automation • Collaborative culture valuing innovation, discipline, and continuous improvement