Lead Software Development Engineer in Test

🕒 September 15

🇺🇸 United States – Remote

💵 $165k - $185k / year

⏰ Full Time

🟠 Senior

⚙️ Software Development Engineer in Test (SDET)

🦅 H1B Visa Sponsor

This company has sponsored H1B visas in the past.

👻 Ghost score 23%

What it means: the chance that no one is really hiring for this job.How to read it: 60% or higher means we think this is a ghost. Lower scores are less of a worry.How we score it: age, reposts, open roles vs team size, how fast jobs close, and whether pay is listed.Fine print: an estimate from public data, not a judgment of the company. Real jobs can score high, so treat it as a hint, not a fact.

Apply Now

Find Similar Remote Jobs

Alteryx

WebsiteLinkedInAll Job Openings

1001 - 5000 employees

💼 Consulting

🏥 Healthcare

🏭 Manufacturing

Consulting • Healthcare • Manufacturing

Alteryx is a leading provider of enterprise analytics solutions that empower organizations to unlock valuable insights from their data. With its AI-driven analytics platform, Alteryx automates the processes of data preparation, analysis, and reporting, making analytics accessible to users at all skill levels. The platform offers features such as data enrichment, predictive modeling, and location intelligence, enabling businesses to improve operational efficiency and drive informed decision-making across various sectors, including financial services, retail, healthcare, and manufacturing.

📋 Description

• Work on the test infrastructure powering agent evaluation and observability for Ask Alteryx • Partner with engineering, product, and data science to measure and improve agent quality • Analyze requirements to produce comprehensive test strategies and detailed test cases • Design and implement manual and automated tests across unit, component, integration, and end-to-end levels • Design, build, and enhance scalable automated test frameworks for feature testing and AI-driven product workflows • Develop and maintain UI and API test automation • Conduct performance and load testing to ensure product scalability • Build test reports • Report regressions to corresponding development teams • Provide timely feedback on design documents and testability to influence design decisions • Collaborate with engineers, product management, and UX in an autonomous, cross-functional agile team

🎯 Requirements

• Experience evaluating AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery • Working knowledge of agent frameworks, preferably LangChain, with the ability to understand and debug context handling, tool execution, and agent state • Hands-on experience with LangSmith or similar platforms for evaluation datasets, custom evaluators, experiments, and comparisons across agent changes • Experience with agent observability, using traces and telemetry to investigate quality issues, latency, token usage, and cost • Strong Python software engineering skills to build and extend evaluation tooling, metrics pipelines, and reporting systems • JavaScript or TypeScript experience is a nice-to-have • Applied statistics and data analysis skills to assess variability, compare experiments, and distinguish meaningful improvements from noise • Modeling experience is a plus • Experience developing evaluation datasets and scoring methods, combining deterministic checks, human feedback, and LLM-based judges • Ability to contribute at a technical lead level, guiding evaluation practices, mentoring engineers, integrating evaluations into CI/CD, and translating production failures into regression coverage

🏖️ Benefits

• Target annual bonus of 15% of base salary, based on individual and company performance • Monthly Connectivity Plus stipend of $150 to support remote work-related expenses • Annual $200 home office reimbursement • Medical, dental, and vision coverage • 401(k) with company match • Paid parental leave, caregiver leave, and flexible time off • Mental health support and wellness reimbursement • Career development and education assistance

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume