RAG Support Chatbot
Customer Support Assistant
A support assistant that has handled 30,000+ questions for 8,800+ people since May 2025, with staff corrections written back into its knowledge base so the next person asking gets the corrected answer.
- 30,000+Evidence 1 · 30,000+ questions handledProduction logs: 30,826 questions received between May 2025 and October 2026.Go to Resultsquestions handled
- 8,800+Evidence 2 · 8,800+ people servedProduction logs: 8,888 different people asked the bot at least one question in the same period.Go to Resultspeople served
- 20,000+Evidence 3 · 20,000+ chat sessionsProduction logs: 20,642 chat sessions in the same period.Go to Resultschat sessions
- ~86%Evidence 4 · ~86% answered without a fallbackProduction logs: 26,421 of 30,826 questions got an answer instead of the fallback reply (85.7%). This measures coverage, not correctness.Go to Resultsanswered without a fallback
- ~60%Evidence 5 · ~60% fewer support escalationsAn estimate: the drop in questions escalated to staff after the support bot launched.Go to Resultsfewer support escalations
When does my team play next?
Can we bring our own canopy to the field?
Staff correction, saved to the knowledge baseCanopies are allowed behind the dugout fence, not along the foul lines.
Personal answers
It reads the family's own schedule, not just the general FAQs.
Staff fix a wrong answer once
The correction goes straight into the knowledge base.
Everyone gets the fixed answer
The next person who asks gets the corrected answer, so fewer questions reach staff.
The problem
A US youth-sports organization gets the same questions from families all season: schedules, rules and logistics, and often personal ones like “when does my team play?”. Staff were answering them one by one.
Constraints
- Answers must stay current as information changes during the season.
- Many questions need the person’s own team and schedule, not just general information.
- When staff correct an answer, the next person who asks must get the corrected version.
How it works
- Q&A knowledge base
- Chunk on question-answer boundaries
- Embedtext-embedding-3-large
- Searchfaiss
- Passages + history + the user's team and schedule
- LLM answer
- Staff mark an answer wrong
- Staff correction loop
- Written back to the knowledge base
- Index rebuilt
The bot searches a question-and-answer knowledge base and combines the best passages with the chat history and the person’s own team and schedule, then writes the reply. When staff mark an answer as wrong, their correction is written back into the knowledge base and the index is rebuilt, so the next family gets the corrected answer.
Key decisions
- Chunk on meaning, not size. The knowledge base is written as questions and answers, so chunks follow those boundaries. Oversized answers are split by sentence, with overlap.
- Close the loop. Staff corrections are written back into the knowledge base, so the next person who asks gets the corrected answer.
- Personal answers from real data. Questions about “my team” are answered from the person’s own team and schedule, not guessed from general text.
- Measure what happens in production. Every question is logged, which is where the numbers on this page come from.
Results
From production logs, May 2025 to October 2026:
- 30,000+ questions handled for 8,800+ people, across 20,000+ chat sessions.
- ~86% of questions answered without falling back to the “I can’t help with that” reply.
- An estimated ~60% fewer questions escalated to staff after launch.
What I’d do next
- Build an evaluation set that checks answers for correctness, not just coverage.
- Track which corrections fix the most repeat questions, and surface those to staff first.