Senior SQA Engineer – LLM

Job not on LinkedIn

🕒 May 19

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

TechSurge Private Limited

WebsiteLinkedInAll Job Openings

11 - 50 employees

Founded 2011

🏢 Enterprise

🤝 B2B

🔒 Cybersecurity

Enterprise • B2B • Cybersecurity

TechSurge Private Limited is an IT services and solutions company providing a broad range of technology offerings including application development, cloud services (Office 365, Azure), business intelligence (Power BI, IBM analytics), enterprise applications and ERP (Microsoft Dynamics, Oracle EBS), infrastructure and virtualization (VMware, hyperconverged), security analysis (endpoint protection, advanced threat protection, DLP) and big data solutions. The company also provides outsourced support (L1–L3, resident engineers), consultancy, migrations and upgrades, paid digital marketing, web design, graphic and animation services, content writing, domain/hosting and web maintenance, along with training and consultancy through its Tech Center. TechSurge serves corporate customers across multiple regions with offices in Pakistan, Ghana, Australia and the USA, focusing on B2B enterprise technology deployments, managed services and cybersecurity solutions.

📋 Description

• Design and own the end-to-end QA strategy for the Conversational Banking Platform, covering functional, regression, performance, security, and AI-specific evaluation. • Build and maintain golden datasets, eval suites, and LLM-as-judge frameworks to validate conversational quality across intents, languages, and tenants. • Define the tenant onboarding QA gate, the certification checklist every new business unit must pass before going live. • Establish regression strategies for prompt changes, model upgrades, retrieval index updates, and guardrail policy changes. • Use Langfuse traces to drive evaluation: mine production failures, convert them into test cases, and close the loop with engineering. • Test NeMo Guardrails configurations against jailbreaks, prompt injection, off-topic drift, and false-positive over-blocking. • Validate governance and compliance behaviors: data residency, PII handling, regulated-product disclosures, and off-limits topics. • Build automated test harnesses for Spring AI services, including tool-calling validation, RAG groundedness, and integration with Cosmos DB and MongoDB data layers. • Partner with the Platform team on quality metrics, SLOs, and the platform eval scorecard. • Coach feature engineers and tenant teams on writing their own evals, making platform-grade quality self-service over time.

🎯 Requirements

• 6+ years in software QA, with at least 1–2 years testing LLM-based, RAG, or conversational AI systems in production. • Hands-on experience with LLM observability and evaluation tools such as Langfuse, LangSmith, Arize, or Phoenix. • Working knowledge of eval frameworks such as Ragas, DeepEval, Promptfoo, or TruLens — including metrics like faithfulness, groundedness, answer relevance, and context precision. • Practical understanding of how to test non-deterministic systems: golden datasets, semantic similarity, LLM-as-judge, and statistical regression detection. • Experience testing guardrail or policy frameworks (NeMo Guardrails, Guardrails AI, or similar). • Solid foundation in API testing, automation frameworks (e.g., pytest, JUnit, Karate, RestAssured), and CI/CD integration. • Familiarity with Spring and Spring Boot applications and JVM-based services. • Comfortable writing queries against NoSQL stores (MongoDB, Cosmos DB) for test data setup and trace inspection. • Strong written communication : able to produce clear test plans, defect reports, and tenant readiness assessments. • Good to Have Experience in banking, financial services, or another regulated industry. • Exposure to multi-tenant platforms: understanding how shared infrastructure changes the testing problem. • Familiarity with red-teaming, adversarial prompt testing, and prompt injection defense. • Working knowledge of vector databases, embedding models, and retrieval evaluation. • Experience with multi-language conversational systems. • Performance and load testing experience for AI workloads (token throughput, latency percentiles, cost per conversation). • Contributions to open-source eval or AI testing tooling. • Experience working with compliance, risk, or audit teams on AI assurance.