消化医护人员与新型聊天机器人在消化内镜癌症筛查相关知识的调查表现

An investigation of digestive healthcare workers and new chatbots on knowledge of endoscopic screening of gastrointestinal cancers

摘要:

目的 探讨并比较消化医护人员以及新型聊天机器人（ChatGPT和新必应）对消化内镜癌症筛查相关知识的知晓情况。方法 问卷共设置23个消化内镜癌症筛查相关问题，涉及消化道癌症的筛查年龄、高危因素、随访时间和消化内镜的优势和风险等。邀请消化医护人员通过电子问卷形式回答问题，应用ChatGPT和新必应对每个问题分别进行10轮回答。以所有问题答案的正确率作为主要研究终点。采用方差分析比较消化医护人员与新型聊天机器人回答消化内镜筛查癌症相关知识的正确率，采用单因素和多因素线性回归模型分析影响消化医护人员答题正确率的因素。结果 共76名消化医护人员（规培医师21名、消化专科护士28名、消化专科医师27名）答题结果被纳入分析。规培医师、消化专科护士、消化专科医师、ChatGPT和新必应的回答正确率分别为36.4%± 10.9%、34.5%±10.2%、52.2%±12.6%、46.3%±9.8%和67.1%±9.3%，差异有统计学意义（F=22.6，P<0.001）。新必应回答正确率最高（P<0.001）；ChatGPT正确率和消化专科医师相当（LSD⁃t=-1.398，P=0.166），均高于消化专科护士（LSD⁃t=2.956，P=0.004；LSD⁃t=5.955，P<0.001）和规培医师（LSD⁃t=2.402，P=0.018；LSD⁃t=4.951，P<0.001）；消化专科护士和规培医师正确率相当（LSD⁃t=-0.574，P=0.567）。与新必应相比，消化专科医师回答消化道癌症筛查不良事件、肠化生随访、结肠癌高危因素和筛查方法等问题的正确率低（P<0.05），回答内镜不良事件和食管癌筛查方法问题的正确率高（P<0.05）。多因素线性回归模型分析显示消化专科医师（β=11.7，t=3.054，P=0.003） 和问卷回答时间（≥7.6 min）（β=7.8，t=2.894，P=0.005）是消化医护人员回答正确率的独立影响因素。结论 与消化医护人员相比，新型聊天机器人——新必应回答癌症筛查相关问题的更准确，但在内镜不良事件问题和食管癌筛查方法问题回答方面表现不佳。

Abstract:

Objective To compare the knowledge of endoscopic screening of gastrointestinal cancers between digestive healthcare workers and new chatbots (chatGPT and new Bing). Methods A test with twenty-three questions of endoscopic screening of gastrointestinal cancers was conducted, focusing on the appropriate age of screening, high-risk factors, the follow-up time, and the advantages and risks of digestive endoscopy. Digestive healthcare workers were invited to complete the test through electronic questionnaires. New Bing and chatGPT were used to answer each question for 10 rounds. The primary endpoint was the correct rate of all answers. The answer accuracy between digestive healthcare workers and new chatbots were compared using variance analysis, and the factors that affected the accuracy of the answers in digestive healthcare workers were explored using univariate and multivariable liner regression analysis. Results The results of the test completed by 76 digestive healthcare workers (21 residents, 28 digestive nurses, and 27 digestive doctors) were analyzed. The accuracies were 36.4%±10.9%, 34.5%±10.2%, 52.2%± 12.6%, 46.3%±9.8% and 67.1%±9.3% in residents, digestive nurses, digestive doctors, chatGPT, and new Bing, respectively, with significant difference (F=22.6, P<0.001). The accuracy was highest in new Bing (P<0.001). The accuracy was comparable between chatGPT and digestive doctors (LSD-t=-1.398, P=0.166), and both higher than that of digestive nurses (LSD-t=2.956, P=0.004; LSD-t=5.955, P<0.001) and residents (LSD-t=2.402, P=0.018; LSD-t=4.951, P<0.001). Furthermore, the accuracy was comparable between digestive nurses and residents (LSD-t=-0.574, P=0.567). Compared with new Bing, digestive doctors had lower accuracy in answering questions related to adverse events of screening, follow-up recommendation of intestinal metaplasia, high risk factors and screening methods for colon cancer (P<0.05), but higher accuracy in answering questions related to endoscopic adverse events and screening methods for esophageal cancer (P<0.05). Multivariable liner regression analysis showed that being digestive doctors (β=11.7, t=3.054, P=0.003) and questionnaire response time (≥7.6 min) (β=7.8, t=2.894, P=0.005) were independent factors for the answer accuracy of digestive healthcare workers. Conclusion Compared with digestive healthcare workers, New chatbots—new Bing has higher accuracy in answering gastrointestinal cancer screening-related questions, but performs poorly in answering questions such as adverse events of endoscopy and screening methods for esophageal cancer.