Development and Effectiveness of an AI Chatbot-Based Mobile Cognitive Screening and Customized Training Application for Preventing Dementia: Older Adults Living in Rural Areas of South Korea

2. 1. AI Conversational Agent-Based Chatbot

A chatbot is an artificial intelligence-based communication tool designed to provide appropriate responses to a wide range of inquiries through text-based interactions with users (Suh, 2017). In clinical settings, chatbot serves as an effective healthcare tool, delivering text-based interventions through mobile devices. They are particularly beneficial in fostering long-term engagement with applications for older adults (Valtolina, 2021; Ryu, 2020). Moreover, chatbots are an effective tool for increasing the medication adherence, which are essential in the healthcare field (Kim, 2024). Chatbot-based cognitive training programs have proven useful in assessing and training cognitive functions in a more accessible and user-friendly manner (Tan, 2023). Additionally, chatbots can provide customized training and immediate feedback, significantly contributing to the improvement of cognitive function in older adults (Rathnayaka, 2022).

In line with this, our application incorporated an AI-based interactive agent and chatbot system to facilitate sustained user engagement and improve medication adherence over time. The objective is to establish a long-term relationship between the user and the system, with the aim of increasing the app usage frequency and adherence rates. The AI chatbot is designed to engage elderly users through a touch-based (tactile) interface, enhancing interaction and ease of use. Our application’s primary modes of interaction are touch and text input. First, in the touch-based games, users respond by selecting answers from a set of pre-configured buttons corresponding to the presented questions. Second, in the text input-based games, users type their answers to the given questions. The chatbot operates on a logic-based system, meaning that it follows pre-programmed responses and interactions. The user’s interaction with the agent occurs without intervention from researchers during the game. For each input, the chatbot provides feedback, such as “That’s correct!” or “Not quite, would you like to try again?”

2. 2. Gamification: Goal Setting, Financial Incentives, Competition, and Achievement

A substantial body of research has explored strategies to enhance engagement with mobile health (mHealth) apps among older adults, with gamification emerging as a particularly effective approach. We conducted a search on Google Scholar for studies from the past five years focused on improving engagement among older adults, and as a result, several studies have identified four key gamification elements that are most effective: goal setting, financial incentives, competition, and achievement (Dewar, 2017; Lynch, 2023; Park, 2021; Shameli, 2017; Kullgren, 2014; McGill, 2018; Lau & Agius, 2021). Clear objectives and progress tracking help maintain adherence (Dewar, 2017; Lynch, 2023), while modest financial rewards reinforce habit formation (Park, 2021). Competition, including self-competition, has been shown to improve engagement (Shameli, 20170, Kullgren, 2014; McGill, 2018). Lastly, visual symbols of achievement, such as badges, foster a sense of accomplishment, encouraging long-term app use (Lau & Agius, 2021).

In line with this, our application incorporated four motivational elements to encourage sustained participation among older adults. These included daily mission reminders sent at 8 AM, weekly reports with usage data and points redeemable for physical rewards, and a leaderboard displaying rankings based on usage, mission success, and cognitive improvement. At the end of the program, the top five participants from each group received certificates.

4. 1. Cognitive assessment and customized cognitive training Saemi-rang

Digital cognitive tools are vital for providing efficient, accessible interventions in aging societies facing increasing rates of cognitive decline. Our tool, Saemi-rang, offers customized cognitive training following a brief assessment using speech and cognitive biomarkers, optimized for smartphones to enhance accessibility. The app’s content was developed in collaboration with a neurology professor and based on existing literature demonstrating the effectiveness of cognitive interventions. The cognitive enhancement games in the Saemi-rang program, outlined in Table 1, target six cognitive domains: calculation, language, attention, executive function, memory, and visuospatial abilities, with eight games in total. Each game adjusts its difficulty level according to the user’s accuracy, enabling a gradual increase in challenge.

Table 1

Customized cognitive enhancement training games

Additionally, the AI chatbot plays a key role in providing personalized training by leveraging a reinforcement learning model that analyzes the user’s cognitive assessment results. Based on these results, the system recommends a tailored cognitive enhancement curriculum to meet individual needs. During the training process, the chatbot guides users through the modules, making the interaction intuitive and accessible for older adults. By delivering training in a friendly manner, offering feedback, encouragement, and guidance, the chatbot simplifies tasks and keeps users motivated, making it an essential component of the cognitive training program.

4. 2. Participants

123 participants from the Suncheon City Dementia Relief Center in South Korea were randomly assigned to intervention (n = 84) or control (n = 39) groups in Table 2. Eligibility criteria were confirmed, and demographic and clinical data were collected. Baseline cognitive assessments were conducted, followed by a reassessment after six weeks to evaluate cognitive function and determine the intervention’s effectiveness. Participants were divided into cognitively normal (n = 36) and cognitively impaired (n = 48) groups. Inclusion criteria for the cognitively normal group were: (1) aged 65 or older, (2) able to read and write Korean, (3) able to use a smartphone, and (4) no difficulty typing. Exclusion criteria included illiteracy, impaired vision, or CDR >0.5. The cognitively impaired group included those diagnosed with CDR 0.5 to CDR 1 within the past 6 months. Other inclusion/exclusion criteria were consistent between groups. Participants could withdraw at any time without penalty.

Table 2

General characteristics of the participants

4. 3. Flow diagram of the study

The number of subjects was set at a significance level of .05, power of .84, and effect size of .25 for a two-tailed t-test using the G*Power 3.17 program. The effect size was based on the calculation of the effect size in Lim’s previous research (Lim, 2017). The requisite sample size for this study was 132 participants. Anticipated dropout rate of 6% (equivalent to 7.9 participants), the sample size was adjusted to be 140 participants. Additionally, the Dementia Relief Centre in Suncheon City requested that participants be recruited in the ratio of 60 in cognitive impairment intervention group, 20 in cognitive impairment control group, 40 in cognitive normal intervention group, and 20 in cognitive normal control group. Consequently, a total of 140 participants were recruited according to this ratio. A total of 12 participants in the cognitive impairment intervention group, 4 participants in the cognitive normal intervention group, and 1 participant in the cognitive normal control group withdrew from the study due to illness or other reasons, leaving 123 participants for analysis in Figure 1.

Figure 1

Flow diagram of the study

4. 4. Primary Outcome: Cognitive Function

The MMSE-KC(Mini-Mental State Examination in the Korean Elderly) is currently the most widely used test for measuring cognitive function and is used as a criterion for determining cognitive decline in this study. The MMSE-KC is a questionnaire instrument that determines the presence or absence of cognitive impairment. The score obtained from the test is used to determine the degree of dementia. It consists of the following scales: time span (5 points), place span (5 points), memory registration (3 points), attention and calculation (5 points), memory association (3 points), and language and spatial organization (9 points) (Folstein, 1975). The MMSE-KC is the most commonly used instrument in clinical practice, with interrater reliability of 0.96 and test-retest agreement of 0.86, indicating good item validity (Folstein, 1975). In this study, the MMSE-KC was used to assess cognitive changes in both the intervention and control groups.

4. 5. Secondary Outcome: User Engagement

Secondary outcomes were measured to assess user engagement through log data collected from the Saemi-rang admin page. Data included game completion rates, app usage frequency, and changes in cognitive test scores (measured monthly). Key metrics were adherence rate (mission success rate) and app usage, with data collected daily, weekly, and monthly. We assume success in a mission if the user finishes at least 3 games, total 15 minutes a day.

4. 6. Study Procedure

Both the intervention and control groups underwent a 6-week study period, with pre- and post-intervention assessments in Figure 2. The study duration was selected based on prior research, which suggests that a 6-week program is effective for cognitive enhancement (Chu, 2007; Youn, 2014). The intervention group received Saemi-rang , an app-based cognitive decline screening and customized cognitive enhancement training program, at home. The control group participated in a six-week offline dementia prevention program provided by local welfare centers and dementia care centers. Each weekly 120-minute session included activities such as art therapy, group percussion, harmonica lessons, trot singing, horticultural therapy, and dementia prevention exercises. The program was conducted once a week on Wednesdays and focused on both cognitive and physical stimulation to prevent dementia. The study was approved by the Institutional Review Board of Yonsei University (IRB No. 202401-HR-2560-06) and conducted in accordance with ethical guidelines. Participants were informed about the study’s purpose, procedures, benefits, and risks before providing written informed consent.

Figure 2

Study flow chart

4. 7. Study Analysis

Data analysis was conducted using SPSS 29.02, with descriptive statistics used to assess participants’ characteristics, and repeated measures ANOVA and paired t-tests employed to compare pre- and post-test results and evaluate the Saemi-rang app’s effectiveness. Post-study interviews were thematically analyzed and categorized for qualitative insights through constant comparative analysis.

5. 1. Hypothesis 1.

As a result of testing the hypothesis “The intervention group will show greater improvement in cognitive function than the control group.” A repeated measures ANOVA was conducted to assess the interaction between time and group (intervention(n= 84) vs. control(n=39)) on MMSE-KC, which is measuring cognitive function. The results showed a significant main effect of time (F=9.144, p<.05) and a significant interaction effect (F=38.709, p<.001). Pre-test scores did not differ significantly between the intervention (M=24.91) and control groups (M=22.64), but post-test scores demonstrated a significant improvement in the intervention group (M=26.54) compared to the control group (M=22.08), with a highly significant difference (p<.001). This confirms that cognitive improvement was significantly greater in the intervention group, supporting Hypothesis 1.

5. 2. Hypothesis 2.

As a result of testing the hypothesis. “Within the intervention group, lower baseline cognitive function will result in greater improvement in cognitive function.” A repeated measures ANOVA assessed the interaction between time and group (intervention vs. control) on MMSE-KC, which is measuring cognitive function, in both cognitively normal and cognitively impaired participants. For the cognitively normal participants, the results showed a significant main effect of time (F=5.720, p<.01) and a significant interaction effect between time and group (F=13.720, p<.001). In the pre-test, no significant difference was observed between the intervention group (M=28.03) and the control group (M=25.63), but a significant difference was found in the post-test, with the intervention group (M=29.25) outperforming the control group (M=25.37) (p<.001), though the effect size was moderate (Hedge’s g = 1.149). For the cognitively impaired participants, the results similarly showed a significant interaction effect between time and group (F=22.958, p<.001), though the main effect of time was not significant (F=3.494, p=.066). In the pre-test, there was no significant difference between the intervention group (M=22.56) and the control group (M=19.80), but the post-test revealed a significant difference, with the intervention group (M=24.50) showing significantly higher scores than the control group (M=18.95) (p<.001). The effect size was larger in the cognitively impaired group (Hedge’s g = 2.320) compared to the cognitively normal group. Thus, the improvement in cognitive function was greater in participants with cognitive impairment, supporting Hypothesis 2. Specifically, as shown in Figure 3, the lower the cognitive function at baseline, the greater the improvement in cognitive scores following the intervention.

Figure 3

MMSE-KC changes for cognitive normal participants over time(left), MMSE-KC changes for cognitive impaired participants over time(right)

5. 3. Hypothesis 3.

As a result of testing the hypothesis. “Higher app usage will link to greater improvement in cognitive function.” A paired t-test analysis was conducted to compare the cognitive function improvements between the top 50% and bottom 50% of Saemi-rang users within the intervention group (n=84, cognitively impaired n=48, cognitively normal n=36). The results indicated a significant improvement in MMSE-KC scores for both the top 50% group (t=-4.018, p<.001, Hedge’s g=2.175) and the bottom 50% group (t=-7.018, p<.001, Hedge’s g=1.783). While both groups showed statistically significant improvements in cognitive function, the effect size for the top 50% group was larger than that for the bottom 50% group. These results are illustrated in Figure 4. For the cognitively impaired participants within the intervention group, a similar analysis was conducted. The top 50% group demonstrated significant cognitive improvement (t=-1.861, p<.05, Hedge’s g=2.021), as did the bottom 50% group (t=-6.713, p<.001, Hedge’s g=1.033). Once again, the effect size was larger for the group with higher usage of Saemi-rang, as seen in Figure 4.

Figure 4

MMSE-KC changes for Saemi-rang usage in intervention group regardless of cognitive status(top), MMSE-KC changes for Saemi-rang usage in intervention group among cognitive impaired group(bottom)

Thus, the results support Hypothesis 3, indicating that higher engagement with the Saemi-rang intervention leads to greater cognitive improvement, particularly for cognitively impaired participants. As shown in Figure 4, the more frequently Saemi-rang was used, the greater the improvement in cognitive scores.

5. 4. Hypothesis 4.

As a result of testing the hypothesis. “Participants with lower cognitive function and higher participation will experience the greatest improvement.” A paired t-test analysis was conducted to test this hypothesis, a comparison was made between frequent Saemi-rang users with mild cognitive impairment (MCI, n=13) and mild dementia (MD, n=22). Both groups showed significant cognitive improvement; however, when cognitive changes were analyzed per unit of time, the MD group demonstrated 5.6 times greater improvement than the MCI group (t=-2.270, p<.001, Hedge’s g = 2.741). These results suggest that frequent Saemi-rang users with lower cognitive function experience greater cognitive gains, supporting Hypothesis 4.

Figure 5

MMSE-KC per time unit for high-frequency users in the MCI and Mild Dementia groups

5. 5. Secondary Outcome

A repeated measures ANOVA was conducted to analyze the weekly trend of daily app usage frequency across cognitive states. Significant differences between cognitively normal and cognitively impaired groups were observed during Weeks 1 (p<.01), 2 (p<.01), 4 (p<.01), 5 (p<.05), and 6 (p<.001), but not in Week 3. The main effect of usage duration on daily app usage was significant (F=7.785, p<.001), as was the interaction between cognitive state and usage duration (F=5.513, p<.01). The cognitively normal group showed a general increase in daily app usage time, with Week 6 having the highest usage (55.0 minutes), while the cognitively impaired group peaked in Week 3 (22.3 minutes).

For adherence rates (measured as completing at least three games per day), a significant difference between cognitive states was found across all weeks (p<.001). The main effect of time on adherence rate was not significant (F=2.161, p=.077), nor was the interaction between cognitive state and usage duration (F=1.426, p=.227). The cognitively normal group had the highest adherence in Week 4 (81.3%), while the cognitively impaired group peaked in Week 3 (43.5%).

Figure 6

Changes in participant engagement rates following intervention implementation

5. 6. Post Program Interview

Interviews were conducted with 14 participants from the intervention group, with an average age of 71 years. The majority were female (71.4%), and the highest level of education was high school (42.9%).

Motivational Factors Influencing Saemi-rang Participation: Participants ranked five motivational factors influencing their engagement with Saemi-rang. The results, weighted by priority, showed that the top motivators were: 1) daily mission reminder messages, 2) chat-based games, 3) weekly reports and incentives, 4) overall weekly rankings, and 5) award certificates. Positive feedback was provided for the daily mission reminders and chat-based games, while the award certificates received negative feedback. Weekly reports and rankings received mixed reactions.

Usability and User Requests: Participants noted the app’s usefulness, reporting improvements in cognitive function and lifestyle habits. Many mentioned that using Saemirang encouraged mental sharpness and reduced time spent watching TV.

“Saemi-rang made me feel sharper, and I now enjoy watching educational programs more than entertainment. (P6)”

However, users requested more positive feedback, encouragement, and hints for challenging questions. They expressed that negative emojis and feedback were discouraging.

“I think it would be better to have smiling emojis instead of frowning ones; it makes me feel like I did something wrong. (P14)”