Software Quality Engineer – Conversational AI / LLM Testing
Experience: 4-9 Years
Location : Noida / Kolkata
Domain : Airlines / Travel Technology
Job Summary
We are seeking a highly skilled Software Quality Engineer with hands-on experience in testing Conversational AI, LLM-based applications, chatbots, virtual assistants, and Agentic AI systems. The ideal candidate will have a strong automation and quality engineering background, deep understanding of how modern conversational systems operate, and solid experience within the Airline or Travel domain, particularly around Booking, Post-Booking, and Check-In workflows.
This role involves validating AI-driven customer conversation journeys, ensuring response quality, evaluating non-deterministic outputs, and establishing robust testing strategies for intelligent systems.
Key Responsibilities
- Own and drive the quality strategy for conversational AI products and customer-facing virtual assistants.
- Design, execute, and automate test scenarios for LLM-powered applications, chatbots, voice assistants, RAG-based systems, and agentic workflows.
- Validate conversational flows across booking, post-booking servicing, and check-in experiences within airline and travel ecosystems.
- Develop and maintain automated test frameworks using Python.
- Create test datasets, golden datasets, and evaluation benchmarks to measure AI system performance.
- Design evaluation mechanisms for non-deterministic AI outputs using:
- Rubric-based scoring
- LLM-as-a-Judge approaches
- Human review workflows
- Statistical quality assessment methods
- Validate prompt behavior, context retention, retrieval accuracy, tool/function calling, and response consistency.
- Identify conversational AI failure modes such as hallucinations, context loss, prompt injections, retrieval issues, and reasoning errors.
- Collaborate with Product Managers, Data Scientists, AI Engineers, and Developers to improve system quality and reliability.
- Participate in release validation, regression testing, performance testing, and production quality monitoring.
Mandatory Skills & Qualifications
Quality Engineering
- 4+ years of experience in Software Quality Engineering, QA Automation, or SDET roles.
- Proven experience owning the quality of a product or platform from testing strategy to production validation.
- Strong understanding of software testing methodologies, automation frameworks, and quality metrics.
Python Automation
- Strong scripting and automation expertise in Python.
- Experience developing automated test suites, validation scripts, and data-driven testing frameworks.
Conversational AI / LLM Testing
Hands-on experience testing one or more of the following:
- Large Language Model (LLM) applications
- Conversational AI platforms
- Chatbots and Virtual Assistants
- Voice Assistants
- Retrieval Augmented Generation (RAG) solutions
- Agentic AI applications
AI/LLM Evaluation Expertise
Practical understanding of:
- Prompt engineering principles
- Context windows and memory handling
- Function/Tool Calling
- Retrieval pipelines
- LLM evaluation methodologies
- AI quality metrics and benchmarking
Experience with:
- Golden datasets
- Human evaluation workflows
- Statistical validation approaches
- Non-deterministic output testing
Domain Knowledge
Strong experience in Airline or Travel domain, including:
- Flight Booking journeys
- Reservation Management
- Post-Booking Services
- Check-In processes
- Passenger servicing workflows
Preferred Skills
- Experience with OpenAI, Azure OpenAI, Anthropic, Gemini, or similar AI platforms.
- Understanding of NLP concepts and conversational design.
- Exposure to API testing and automation tools.
- Experience with CI/CD pipelines and test automation integration.
- Knowledge of cloud platforms (Azure, AWS, GCP).
- Familiarity with observability and monitoring tools for AI applications.
À propos de Cognizant
Cognizant (NASDAQ : CTSH) est un AI Builder et une entreprise de services numériques (ESN) élaborant des solutions complètes d’IA maximisant les investissements pour des résultats concrets. Sa profonde expertise des métiers, des processus et des technologies lui permet d’intégrer dans les systèmes technologiques le contexte unique de chaque organisation de l’ingénierie à la production à l’échelle. Son objectif : améliorer l’efficacité des équipes, créer de la valeur et permettre aux grandes entreprises de rester performantes dans un monde qui évolue rapidement. Pour en savoir plus : cognizant.ai ou @cognizant.
Renseignments suppplémentaires sur l'emploi
Les informations relatives à la rémunération du poste à pourvoir dépendent de la date de publication de l’offre de poste. Cognizant se réserve le droit de modifier ces informations à tout moment, sous réserve des lois applicables.
Cognizant est un employeur soucieux de l'égalité des chances entre candidats. Votre candidature sera étudiée indépendamment de votre race, couleur, sexe, religion, croyances, orientation sexuelle, identité de genre, origine, handicap, informations génétiques, grossesse, statut d'ancien militaire ou de toute autre critère jugé discriminant par les lois européennes ou françaises.
Vous êtes porteur d'un handicap, vous pouvez-nous contacter par courriel [email protected] si vous souhaitez préciser les aménagements nécessaires pour le poste ou les entretiens à venir.
Les candidats peuvent être invités à participer à des entretiens en face à face ou par vidéoconférence. En outre, les candidats peuvent être amenés à présenter une carte d'identité valide lors de chaque entretien.
