The chatbot has been implemented using deep learning techniques. can anyone please suggest an efficient and effective way to measure the performance so that it can be compared with other implementations that has been done in the past. In some researches i saw that many researchers used Bleu and perplexity for the same and some researchers just gave the accuracy in term of percentage.
Please let me know if you have any question so that we can make this discussion fruitful.
check the folowing links:
https://research.aimultiple.com/chatbot-analytics/
https://www.revechat.com/blog/chatbot-analytics-metrics/
https://www.inbenta.com/en/blog/10-key-metrics-to-evaluate-your-ai-chatbot-performance/#:~:text=Chatbot%20user%20experience%20metrics&text=Self%2Dservice%20rate%3A%20percentage%20of,action%20after%20using%20the%20bot.&text=Usage%20rate%20per%20login%3A%20volume,of%20sessions%20on%20your%20website.
User Metrics
- 1- Total Users: This is the most basic metric. It captures the number of people using your chatbot. This matter because its trend shows the change in the number of users and therefore the amount of data your chatbot has been exposed to. Also, this would provide critical information regarding the market size calculations and potentially the effect of your chatbot.
- 2- Active Users: Active users can be defined as the people who read a message in the chatbot in a defined time frame. These are your potential targets. Measuring the potential effects of a promotional activity can be estimated from that number. The number of people who read your message is critical. This is something like the promotional content on social media. Engagement is not guaranteed but, the content is seen by the people
- 3- Engaged Users: Those users are the ones who communicate with the chatbot. They receive and send messages. This is important since your chatbot will be able to provide the conversation statistics based on this sub-sample. They are likely to shape your decision regarding the effectiveness of the chatbot. It doesn’t make any sense if the chatbot is not able to start the conversation with the users.
- 4- New Users: This metric captures the overall success of your chatbot promotion campaign. New users will be necessary to keep an active user number. Customers’ preferences changes over time and the amount of interaction with the chatbot shows an exponential decay. For that reason, new users will keep your customer base strong.
- 5- User sentiment: This metric is captured by performing sentiment analysis so that you can categorize messages as positive, neutral, or negative. You can gain insights into the user experience and where/when the conversation went wrong.
Perhaps you can use the total number of words synthesized by the chatbot in one time interval.
Have a look at Review of State-of-the-Art Design Techniques for Chatbots | SpringerLink
An Overview of Chatbot Technology (nih.gov)
check the folowing links:
https://research.aimultiple.com/chatbot-analytics/
https://www.revechat.com/blog/chatbot-analytics-metrics/
https://www.inbenta.com/en/blog/10-key-metrics-to-evaluate-your-ai-chatbot-performance/#:~:text=Chatbot%20user%20experience%20metrics&text=Self%2Dservice%20rate%3A%20percentage%20of,action%20after%20using%20the%20bot.&text=Usage%20rate%20per%20login%3A%20volume,of%20sessions%20on%20your%20website.
User Metrics
- 1- Total Users: This is the most basic metric. It captures the number of people using your chatbot. This matter because its trend shows the change in the number of users and therefore the amount of data your chatbot has been exposed to. Also, this would provide critical information regarding the market size calculations and potentially the effect of your chatbot.
- 2- Active Users: Active users can be defined as the people who read a message in the chatbot in a defined time frame. These are your potential targets. Measuring the potential effects of a promotional activity can be estimated from that number. The number of people who read your message is critical. This is something like the promotional content on social media. Engagement is not guaranteed but, the content is seen by the people
- 3- Engaged Users: Those users are the ones who communicate with the chatbot. They receive and send messages. This is important since your chatbot will be able to provide the conversation statistics based on this sub-sample. They are likely to shape your decision regarding the effectiveness of the chatbot. It doesn’t make any sense if the chatbot is not able to start the conversation with the users.
- 4- New Users: This metric captures the overall success of your chatbot promotion campaign. New users will be necessary to keep an active user number. Customers’ preferences changes over time and the amount of interaction with the chatbot shows an exponential decay. For that reason, new users will keep your customer base strong.
- 5- User sentiment: This metric is captured by performing sentiment analysis so that you can categorize messages as positive, neutral, or negative. You can gain insights into the user experience and where/when the conversation went wrong.
Thanks You Muhammad Ali and Mohammed Salih Mahdi for sharing these links I will check these links and get back to you for further discussion if needed.
https://www.researchgate.net/deref/https%3A%2F%2Fresearch.aimultiple.com%2Fchatbot-analytics%2F
- Badges
- Science topic