BACKGROUND
The worldwide introduction of ChatGPT in November 2022 may have changed how its users perceive and interact with other chatbots. This possibility may confound the comparison of responses to pre-ChatGPT and post-ChatGPT iterations of pre-existing chatbots, in turn affecting the direction of their evolution. Before the release of ChatGPT, we created a therapeutic chatbot, MIBot, whose goal is to use Motivational Interviewing (MI) to guide smokers towards making the decision to quit smoking. We were concerned that measurements going forward would not be comparable to those in the past, impacting the evaluation of future changes to the chatbot.
OBJECTIVE
Explore changes in how users interact with MIBot after the release of ChatGPT, and quantify the impact as a function of users’ familiarity with ChatGPT.
METHODS
We compared user interactions with MIBot prior to ChatGPT’s release and 6 months after the release. Participants (n=143) were recruited through an online platform in November of 2022, prior to the release of ChatGPT, to converse with MIBot, in an experiment we refer to as MI v5.2. In May of 2023, a set of (n=129) different participants were recruited to interact with the same version of MIBot, and asked additional questions about their familiarity with ChatGPT, in the experiment called MI v5.2A. We fit a logistic regression model for “ChatGPT Exposure”. We also fit two linear regression models for “Familiarity with ChatGPT”, one with MI v5.2A only and another with both cohorts, on relevant effect variables.
RESULTS
Fully 64% of the MI v5.2A cohort had used ChatGPT, with 51% using it on a regular basis. In the logistic regression model, ChatGPT exposure is negatively correlated to satisfaction with MIBot (β = -1.8, P = .005). In both linear regression models, familiarity with ChatGPT is positively correlated to change in confidence to quit smoking (MI v5.2A only: β = 4.1, P < .001; both cohorts: β = 1.7, P = .002) and average response length (MI v5.2A only: β = 1.0, P = .02; both cohorts: β = 0.73, P = .010), and negatively correlated with satisfaction with MIBot (MI v5.2A only: β = -16, P = .03; both cohorts: β = -14, P < .001).
CONCLUSIONS
The widespread reach of ChatGPT has changed how users interact with MIBot. Users exposed to ChatGPT are less satisfied with MIBot in general. Moreover, users with a higher familiarity with ChatGPT offer longer responses (especially to Yes/No questions), are more strongly influenced by MIBot, and are even less satisfied with MIBot.
CLINICALTRIAL
N/A