Senior AI Quality Engineer

🔥 8 minutes ago

🇨🇦 Canada – Remote

💵 C$100k - C$130k / year

⏰ Full Time

🟠 Senior

🔧 QA Engineer (Quality Assurance)

👻 Ghost score 0%

What it means: how likely this posting is sitting open without active hiring.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Novara

WebsiteLinkedInAll Job Openings

51 - 200 employees

☁️ SaaS

🤝 B2B

📋 Compliance

SaaS • B2B • Compliance

Novara is a provider of environmental health & safety (EHS) and compliance software and services. Its flagship platform, Novara Flex, is an all-in-one SaaS solution that supports incident reporting, SDS (safety data sheet) management, inspections and audits, asset management, contractor risk management, carbon accounting and sustainability programs, and an integrated learning management system with an extensive library of safety and compliance training. Novara serves business customers across industries such as construction, manufacturing, energy, utilities, mining, and insurance, and emphasizes configurable dashboards, mobile reporting, and data-driven analytics to reduce workplace injuries and ensure regulatory compliance.

📋 Description

• Design and maintain evaluation frameworks for LLM outputs and agentic workflows, including regression suites, golden datasets, and scoring rubrics • Build test harnesses that catch hallucinations, tool-calling failures, prompt regressions, and unsafe or off-policy behavior before they reach production • Define measurable quality criteria for agent reliability: task completion, factual grounding, latency, cost, and reasoning quality • Integrate evaluation runs into CI/CD so model, prompt, and agent changes are gated the same way code changes are • Partner with engineers on observability and tracing for agent runs, so failures are diagnosable rather than mysterious • Contribute to conventional API and end-to-end automation where AI features sit inside larger product flows • Help shape the team's shared playbook for testing AI features, and coach other QA engineers as agentic work spreads across scrum teams • Work hands-on as an individual contributor reporting to the QA Manager • Partner with product, engineering, and the broader QA team on LLM-driven and agentic workflows

🎯 Requirements

• 6+ years in QA, SDET, or test automation, with real production automation shipping • Hands-on experience testing LLM-based or agentic systems: building evals, working with LLM-as-judge patterns, prompt regression testing, or agent trajectory analysis • Prior experience in a shift-left, embedded QA model • Comfort with at least one modern automation stack (Playwright, Cypress, or similar) and a typed language, TypeScript preferred, Python fine • Deep experience with test frameworks such as vitest, jest, or pytest, and comfort building custom test harnesses rather than only running off-the-shelf suites • API-first testing mindset, including REST and Postman or equivalent • Fluency in HTTP-level API testing, including recording proxies and observing service-to-service traffic • Working knowledge of CI/CD pipelines, GitHub Actions a plus, and how to plug evals into them • Familiarity with cloud secret managers and disciplined handling of sensitive test data in restore-from-prod environments • Ability to reason clearly about probabilistic systems: variance, sample sizes, confidence, and when a flaky result is signal rather than noise

🏖️ Benefits

• Medical • Dental • Vision • Flexible Spending Accounts • PTO • Paid and Floating Holidays • 401k with Company match and immediate vesting • Company-funded Life Insurance • Employee Assistance Programs • No-cost Mental Health Benefits • Annual Bonus Opportunity of 10%