Usability Assessment Of Wowbot: A Chatbot Platform For Healthcare Applications

Supawadee Naorungroj; Kittiwara Pupong; Watcharawalee Tangkuptanon; Samerchit Pithpornchaiyakul; Pissamai Wattanasit; Jaranya Hunsrisakhun

doi:10.31584/jhsmr.20261302

Usability Assessment Of Wowbot: A Chatbot Platform For Healthcare Applications

Supawadee Naorungroj, Kittiwara Pupong, Watcharawalee Tangkuptanon, Samerchit Pithpornchaiyakul, Pissamai Wattanasit, Jaranya Hunsrisakhun

Abstract

Objective: Chatbots have emerged as valuable tools in healthcare; however, existing platforms’ features are not designed to support user engagement and data collection for healthcare research. This study aimed to assess the usability issues of Wowbot, a chatbot platform developed as a plugin to the Botnoi platform, through usability testing with chatbot developers.
Material and Methods: Usability testing was conducted with five chatbot developers, who were recruited to perform six tasks using platform functions: dialogue creation, push messages, application programming interfaces (APIs) in creating photo frames and score comparisons, data export, agent-mode chat, and natural language processing (NLP) for parameter recognition. One expert user served as a control. The think-aloud method, semi-structured questionnaires, and exit interviews were used to collect data. Effectiveness was assessed using performance scores, task completion time, and incomplete task incidence, while efficiency was evaluated by comparing task completion time with the expert.
Results: Four out of five participants completed all tasks. The overall task completion rate within two attempts was 74.5%, with lower rates in photo frame creation (20%), score comparison (40%), and message pushing (40%). Participants required more time to complete tasks than the expert (188 vs. 130 minutes). Nevertheless, users reported high satisfaction (mean 4.2/5) and acknowledged the platform’s potential to enhance engagement in healthcare applications.
Conclusion: The Wowbot platform showed potential usefulness for healthcare applications, with new features that may enhance user engagement. However, task completion challenges highlight the need for further refinement, particularly in API design and documentation, to optimize user experience and platform adoption.

Keywords

chatbot platform; oral health education; think-aloud; usability testing

Full Text:

PDF

References

Aggarwal A, Tam CC, Wu D, Li X, Qiao S. Artificial intelligence-based chatbots for promoting health behavioral changes: systematic review. J Med Internet Res 2023;25:e40789.

Pithpornchaiyakul S, Naorungroj S, Pupong K, Hunsrisakhun J. Using a chatbot as an alternative approach for in-person toothbrushing training during the covid-19 pandemic: comparative study. J Med Internet Res 2022;24:e39218.

Sakib MN, Butt ZA, Morita PP, Oremus M, Fong GT, Hall PA. Considerations for an individual-level population notification system for pandemic response: a review and prototype. J Med Internet Res 2020;22:e19930.

Oakley-Girvan I, Yunis R, Longmire M, Ouillon JS. What works best to engage participants in mobile app interventions and e-health: a scoping review. Telemed J E Health 2022;28:768-80.

Kornfield R, Mohr DC, Ranney R, Lattie EG, Meyerhoff J, Williams JJ, et al. Involving crowdworkers with lived experience in content-development for push-based digital mental health tools: lessons learned from crowdsourcing mental health messages. Proc ACM Hum Comput Interact 2022;6(Cscw1).

Zhang J, Oh YJ, Lange P, Yu Z, Fukuoka Y. Artificial intelligence chatbot behavior change model for designing artificial intelligence chatbots to promote physical activity and a healthy diet: viewpoint. J Med Internet Res 2020;22:e22845.

One-time and recurring notifications 2024 [homepage on the Internet]. San Francisco: Chatfuel; 2024 [cited 2024 May 21]. Available from: https://docs.chatfuel.com/en/articles/3705639-one-time-and-recurring-notifications

Kingchang T, Chatwattana P, Wannapiroom P. Artificial intelligence chatbot platform: ai chatbot platform for educational recommendations in higher education. IJIET 2024;14:34-41.

Botnoi Group [homepage on the Internet]. [cited 2024 May 21]. Available from: https://botnoigroup.com/

Nielsen J. Why you only need to test with 5 users [homepage on the Internet]. 2000 [cited 2024 May 21]. Available from: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Hunsrisakhun J, Naorungroj S, Tangkuptanon W, Wattanasit P, Pupong K, Pithpornchaiyakul S. Impact of oral health chatbot with and without toothbrushing training on childhood caries. Int Dent J 2025;75:1348-59.

Kirakowski J. SUMI [homepage on the Internet]. [cited 2025 Aug 20]. Available from: https://sumi.uxp.ie/index.html

International Organization for Standardization (ISO). Ergonomics of human-system interaction—Part 11: Usability: definitions and concepts. ISO 9241-11:2018 [homepage on Internet]. Geneva: ISO; 2018 [cited 2023 Nov 11]. Available from: https://www.iso.org/standard/63500.html

Turner CW, Lewis JR, Nielsen J. Determining usability test sample size. In: Karwowski W, editor. International Encyclopedia of Ergonomics and Human Factors. 2nd ed. Boca Raton, FL: CRC Press; 2006;p.3076–80.

Hwang W, Salvendy G. Integration of usability evaluation studies via a novel meta-analytic approach: what are significant attributes for effective evaluation? Int J Hum Comput Interaction 2009;25:282-306.

Chen W, Lin T, Li C, Yuan P. Automated comprehensive evaluation approach for user interface satisfaction based on concurrent think-aloud method. Uni Access Inf Soc 2018;17.

Jaspers M, Steen T, Bos C, Geenen M. The think aloud method: A guide to user interface design. Int J Med Inform 2004;73:781-95.

Dwivedi Y, Rana N, Jeyaraj A, Clement M, Williams M. Re-examining the unified theory of acceptance and use of technology (utaut): towards a revised theoretical model. Inf Syst Front 2019;21:1-16.

Alkhaldi G, Hamilton FL, Lau R, Webster R, Michie S, Murray E. The effectiveness of prompts to promote engagement with digital interventions: a systematic review. J Med Internet Res 2016;18:e6.

Szinay D, Jones A, Chadborn T, Brown J, Naughton F. Influences on the uptake of and engagement with health and well-being smartphone apps: systematic review. J Med Internet Res 2020;22:e17572.

Safi Z, Abd-Alrazaq A, Khalifa M, Househ M. Technical aspects of developing chatbots for medical applications: scoping review. J Med Internet Res 2020;22:e19127.

Xu L, Sanders L, Li K, Chow JCL. Chatbot for health care and oncology applications using artificial intelligence and machine learning: systematic review. JMIR Cancer 2021;7:e27850.

DOI: http://dx.doi.org/10.31584/jhsmr.20261302

Refbacks

There are currently no refbacks.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.