Last year, our team decided to create a LINE chatbot to directly answer Thai customer questions. Our goal was to reduce the workload for our customer service team and improve response efficiency. However, what we encountered turned out to be far more complex than our original concept. Now, I'll share the story of building a system that had to be precisely timed and the real mistakes we learned from.
The Beginning: A Chatbot That Needed to Answer Every Question
We started with a Thai vocabulary database of 2,176 words, covering most of the common questions customers typically ask, such as account information, deposit/withdrawal methods, promotional news, and other general inquiries. This chatbot was designed to serve as the first phase in answering questions, attempting to match customer queries with our vocabulary database first. If no match was found, it would forward the question to AI for a more accurate answer. But we quickly learned that what seems easiest is often the most challenging.
Design: Speed is Everything
We fully understood that in the world of trading, speed is the most critical factor. Customers don't want to wait long for answers. Therefore, we set a hard deadline of 50 seconds. But setting such a goal forced us to consider every detail. The classifier we developed had a working time of approximately 1.4-2.0 seconds, which is considered very fast for this type of task. However, we wanted it to be as fast as possible to give the AI more time to formulate answers for customers.
The Challenge: Accurate Time Measurement
Testing the system wasn't as straightforward as it might seem, especially when it came to precise time measurement. We needed a large amount of real-world usage data to get a true p95 (95th percentile), meaning 95% of questions had to be answered within the specified time. From our actual testing, we found that the combined response time p95 was 5,334 milliseconds, or approximately 5.3 seconds—significantly below our hard deadline. However, we weren't complacent because we knew that real-world environments might have other factors that could increase response times.
Improvement: Making the Right Decisions
After obtaining response time data from actual testing, we needed to improve the system. One of the main problems we encountered was that sometimes the classifier would forward questions to AI even when they were already in our vocabulary database, wasting unnecessary time. Therefore, we set a higher standard for the classifier to make the best possible decision, even if it took slightly longer. We found that setting it for higher accuracy reduced the number of cases forwarded to AI by 40%, which resulted in significant time savings.
Results: Balancing Speed and Accuracy
After several improvements, we were able to achieve our goal. The chatbot now answers questions within the specified time, with a combined response time p95 of 5,334 milliseconds—comfortably within our set deadline while maintaining good response accuracy. Most customers are satisfied with the fast and accurate responses, which has genuinely helped reduce the workload for our customer service team.
Lessons: The Time War in Question Answering
This project taught us about the balance between speed and accuracy. Sometimes, trying to be the fastest can compromise accuracy, and conversely, trying to be the most accurate can make things slower. Setting strong time goals and accurate measurement is crucial, but more importantly is understanding the real behavior of users and tailoring the system to meet their needs as they expect—not just making it work as fast as possible.
For those who want to try trading gold for real, open an XM account at: https://clicks.pipaffiliates.com/c?c=72816&l=th&p=6
