Overview
An internal and external productivity tool built on a Crew Rostering System (CRS) platform used by airlines. As Lead AI Product Manager, I took it from concept through MVP launch and into ongoing improvements driven by user feedback and performance data.
The chatbot uses Retrieval-Augmented Generation (RAG) to unlock knowledge across both the Interface System and Optimizer System sides of the platform, serving support teams, developers, QA engineers and airline clients.
The challenge
CRS documentation spans technical manuals, change requests, bug fixes, resolved tickets and client-specific configurations. That complexity slowed teams down, stretched client issue resolution in post-launch support, and made onboarding new team members hard.
AI hypothesis
Applying RAG to our own CRS documentation will make finding information much faster and more accurate than keyword search.
Expected outcome: lower support costs, higher productivity for internal and external users, and better product adoption.
Key assumption: our existing documentation is good enough to ground the AI’s answers, making the chatbot trustworthy for mission-critical airline operations.
Who it’s for
Internal teams were losing hours searching thousands of documents, struggling to connect information across modules, and missing support SLAs.
Airline clients couldn’t easily find guidance on advanced features, so they leaned on formal support even for routine questions.
Personas
- Support Lead / Product Analyst spends 40% of their day digging through old tickets and release notes. “Find all documented bug fixes and resolution steps for this rule-compliance issue in the last three releases.”
- Airline Crew Operations Planner needs quick references on rules and integrations without waiting on a ticket. “Explain the steps and data format for connecting third-party logistics data to the Crew Portal.”
Customer journey
| Stage | Pain point | How the chatbot helps | Metric |
|---|---|---|---|
| Triage | Unclear source of a rule violation | “What is the logic for Alert #45 in the latest release?” | Faster root-cause ID |
| Research | Answers scattered across systems | “Summarize the resolution for CR #1234.” | Less search time |
| Solution | Risk of incomplete answers to clients | “Write the client-facing summary for this fix.” | Higher first-call resolution |
Architecture
- Ingestion: PDFs, Confluence pages, XML docs and API specs.
- Chunking and embeddings: stored in Pinecone for fast meaning-based search.
- LLM: GPT-4, tuned for clear, concise answers.
- Grounding and citations: every answer points to its source to minimize made-up content.
RAG was the right choice because answers must come only from verified documentation. In a heavily regulated industry, trust is everything.
Data pipeline
- Automatic ingestion from SharePoint and internal repositories, with cleaning along the way.
- New release notes and rule updates reach the knowledge base within a 24-hour window.
- Role-based access keeps sensitive client configurations protected.
Validating quality
- Tracked retrieval precision and recall, plus grounding rate: the share of answers fully supported by sources.
- A QA and subject-matter-expert team reviewed daily samples; any wrong or invented answers were logged and used to improve retrieval.
- A “Was this helpful?” button fed real user feedback back into tuning.
Grounding rate started at a 90% MVP target and reached 95% after tuning.
Risks and how we handled them
| Risk | Mitigation |
|---|---|
| People sticking to old habits | Internal champions, demos and onboarding sessions |
| Answers misused | Source links and confidence scores to encourage checking |
| Gaps in documentation | Ongoing content enrichment with expert review |
| Sensitive data exposure | Masking personal data, access controls, audit logs |
| AI provider downtime | Fallback models and response caching |
Roadmap
- MVP: internal chatbot for support and developers. Cut search time by 70%.
- Client access: external chatbot with role-based permissions.
- Integrations: Jira and knowledge portal connections for ticket automation.
- Global scale: multi-language and voice support.
Results
- 70% less time spent searching documentation.
- 50% faster client support responses after implementation.
- 90% of internal users rated answers relevant and time-saving.
The chatbot also laid the groundwork for AI search across the company’s other products, and premium support tiers now use it as a competitive edge.