QA Test Engineer

Job not on LinkedIn

🔥 3 hours ago

🗽 New York – Remote

🗽 New York – Remote

💵 $55 - $85 / hour

⏰ Full Time

🟢 Junior

⚙️ Software Development Engineer in Test (SDET)

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

24-MAG

WebsiteLinkedInAll Job Openings

2 - 10 employees

🤝 B2B

💼 Consulting

B2B • Consulting

24-MAG is a commercial strategy and execution firm that helps B2B organizations design and implement systems, workflows, and operating rhythms for sales, client management, and cross-functional projects. They focus on transforming scattered processes into aligned, measurable, and scalable commercial functions—covering pipeline structure, account management frameworks, and operational discipline for teams seeking efficient, intentional growth.

📋 Description

• Create comprehensive test cases confirming that benchmark tasks function as intended • Design positive, negative, boundary, and edge-case tests • Validate task requirements, expected outputs, reference solutions, and grading logic • Identify scenarios that may produce incorrect or misleading evaluation results • Ensure tests measure the intended technical capability accurately • Review complex multi-step tasks and reference solutions before finalisation • Identify ambiguous instructions, inconsistent requirements, missing assumptions, and incomplete acceptance criteria • Run tasks independently to confirm reproducibility and expected behaviour • Assess whether grading standards are clear, fair, and technically defensible • Provide actionable feedback to task authors and researchers • Investigate failures across Python scripts, test harnesses, repositories, and task environments • Diagnose unexpected behaviour within unfamiliar codebases • Reproduce reported issues and isolate their underlying causes • Correct or document environment, dependency, logic, and validation problems • Use Git-based workflows to support structured review and collaboration • Develop practical checklists and repeatable review procedures for benchmark quality • Improve consistency across task validation, testing, and approval workflows • Document findings clearly so authors can resolve issues efficiently • Track recurring defects and recommend preventive quality measures • Collaborate closely with researchers, task authors, and other technical reviewers • Examine AI agent runs for unintended shortcuts, loopholes, and grading weaknesses • Identify cases where models can receive credit without completing the intended reasoning or technical work • Test whether benchmark tasks remain robust across alternative approaches • Strengthen evaluation criteria to maintain reliable and meaningful benchmark scores • Distinguish valid solution diversity from unintended task exploitation

🎯 Requirements

• At least 1 year of experience in test engineering, quality assurance, software engineering, research engineering, or a related technical role • Demonstrated experience designing test cases and quality-review processes • Strong end-to-end debugging skills across complex technical systems • Working proficiency in Python and Git • Comfort navigating unfamiliar codebases, repositories, and execution environments • Exceptional attention to detail and strong written documentation habits • Ability to identify ambiguity, edge cases, hidden assumptions, and quality gaps • Capacity to work independently through open-ended technical problems • Reliable availability for approximately 35 hours per week