Senior Prompt Engineer – Data Science, Quality Analysis

4 days ago

Apply Now

Receive Emails with Similar Jobs

ItsaCheckmate

WebsiteLinkedInAll Job Openings

SaaS • Hospitality

ItsaCheckmate is a technology company that provides powerful ordering solutions for busy restaurants. Their platform enables restaurants to drive sales and connect with customers through various ordering channels, including web, app, kiosk, and third-party integrations. With its scalable enterprise technology, ItsaCheckmate helps restaurants manage and streamline operations, reduce order errors, and increase productivity. They offer features such as loyalty programs, catering management, QR-code ordering, and unified channel management. Trusted by over 2,400 brands, ItsaCheckmate is dedicated to supporting restaurant growth with innovative digital solutions and a marketplace of integrations.

201 - 500 employees

☁️ SaaS

📋 Description

• Design, test, and optimize LLM prompts for conversational AI, text classification, and structured data extraction tasks. • Build evaluation pipelines to analyze prompt performance using quantitative metrics, human-in-the-loop feedback, and business KPIs. • Conduct prompt experiments and regression testing to ensure stability, accuracy, and safety as models evolve. • Collaborate with Machine Learning, Product, and Operations teams to translate business objectives into scalable, data-driven prompt-engineering strategies that enhance model accuracy, efficiency, and real-world usability. • Use Python/SQL to analyze model outputs, identify anomalies, and automate quality checks. • Document best practices and contribute to internal frameworks for prompt evaluation and continuous improvement. • Communicate findings effectively to technical and non-technical stakeholders, driving measurable business impact through insight-driven decisions.

🎯 Requirements

• B.S. or higher in a quantitative discipline (Data Science, Computer Science, Engineering, or related field) or in a field relevant to language models (Linguistics, Philosophy, Cognitive Science, etc.). • 5+ years of relevant experience with a B.S. degree, or 3+ years of experience with a Master’s degree. • Demonstrated proficiency in Python for automation, evaluation, and experimentation with LLM workflows. • Proven experience in prompt engineering and working with LLMs (GPT-4, Claude, Gemini, and LLaMA) for text generation, reasoning, and structured data extraction. • Proficiency in Python and SQL for data analysis, evaluation scripting, and workflow automation. • Strong background in A/B testing, statistical analysis, and performance metric evaluation, with the ability to design experiments and interpret data-driven insights for continuous model optimization. • Familiarity with prompt-evaluation tools such as LangFuse or Galileo, and Weights and Biases for experiment management and regression testing. • Deep understanding of advanced prompting techniques, including few-shot prompting, reasoning-based prompting, multi-turn dialogue design, agentic orchestration, and DSPy/AdaFlow-style programmatic prompting frameworks. • Experience applying CO-STAR and TIDD-EC! prompting frameworks for structured reasoning, instruction design, and context control in production-grade LLM systems. • Excellent requirement-elicitation and communication skills, with the ability to translate business objectives into prompt-engineering solutions. • Analytical mindset with a process-driven approach to optimizing model behavior, data quality, and operational workflows. • Academic or applied research experience related to language models, prompt engineering, or LLM-based systems is a strong plus. • Familiarity with LLM architectures, embeddings, and fine-tuning techniques preferred. • Experience with LLM red-teaming, adversarial evaluation, or model safety testing is a plus. • Candidates must be flexible and work during US hours at least till 5 pm EST, which is essential for this role & must also have their own system/work setup for remote work.