人工智能系统预测早期胃癌浸润深度和分化状态的能力——在单中心和多中心视频集中的表现

Ability of artificial intelligence system to predict invasion depth and differentiation status of early gastric cancer: performance in single‑center and multi‑center videos

摘要:

目的 评价ENDOANGEL人工智能系统在多样性测试集（多中心来源视频集）中预测早期胃癌浸润深度和分化状态的能力，并在完成系统升级后，对新系统的上述能力进行评价。方法 在已完成的2020年开展的单中心视频集早期胃癌诊断人机比赛基础上，于2022年开展来自国内10个省份30家医院30名内镜医师参与的人机比赛，采用国内8个省、市12家医院回顾性收集的视频作为多中心视频集。研究共分成三个阶段：第一阶段，用多中心视频集重新测试ENDOANGEL，比较ENDOANGEL在单中心视频集、多中心视频集中的测试结果，并完成系统升级（ENDOANGEL‑2022）；第二阶段，使用多中心视频集，开展ENDOANGEL‑2022与30名内镜医师参与的人机比赛，比较ENDOANGEL‑2022、ENDOANGEL以及内镜医师在多中心视频集中的表现；第三阶段，使用2020年人机比赛的单中心视频集测试ENDOANGEL‑2022，比较新系统在单中心、多中心视频集中的表现。结果 相比单中心视频集中的表现，在多中心视频集中，ENDOANGEL预测早期胃癌黏膜下浸润的敏感度明显下降［18.18%（2/11）比70.00%（7/10），P=0.030］，预测未分化型早期胃癌方面表现相似（P>0.05）。在多中心视频集中，ENDOANGEL‑2022预测早期胃癌黏膜下浸润的敏感度高于ENDOANGEL［40.00%（4/10）比18.18%（2/11），P=0.361］、低于30名内镜医师的平均水平［40.00%比52.04%（95%CI：43.70%~60.38%），P<0.001］，特异度低于ENDOANGEL［82.86%（29/35）比100.00%（34/34），χ2=4.41，P=0.036］、高于30名内镜医师的平均水平［82.86%比68.97%（95%CI：60.83%~77.11%），P=0.018］，准确率低于ENDOANGEL［73.33%（33/45）比80.00%（36/45），χ2=0.56，P=0.455］、高于30名内镜医师的平均水平［73.33%比65.30%（95%CI：60.61%~69.99%），P=0.018］；ENDOANGEL‑2022预测未分化型早期胃癌的敏感度高于ENDOANGEL［71.43%（5/7）比57.14%（4/7），P>0.999］和30名内镜医师的平均水平［71.43%比63.11%（95%CI：55.58%~70.64%），P=0.031］，特异度低于ENDOANGEL［76.32%（29/38）比78.95%（30/38），χ2=0.08，P=0.783］、高于30名内镜医师的平均水平［76.32%比65.27%（95%CI：59.10%~71.44%），P=0.004］，准确率与ENDOANGEL相同［75.56%（34/45）比75.56%（34/45），χ2=0.00，P>0.999］、高于30名内镜医师的平均水平［75.56%比65.10%（95%CI：59.96%~70.24%），P<0.001］。相比单中心视频集中的表现，ENDAOANGEL‑2022在多中心视频集中预测早期胃癌黏膜下浸润的敏感度［40.00%比60.00%（6/10），P=0.656］、特异度［82.86%比93.75%（15/16），χ2=0.37，P=0.542］和准确率［73.33%比80.77%（21/26），χ2=0.50，P=0.479］均下降；预测未分化型早期胃癌的敏感度上升［71.43%比37.50%（3/8），P=0.315］，特异度［76.32%比100.00%（18/18），χ2=3.48，P=0.062］和准确率［75.56%比80.77%（21/26），χ2=0.26，P=0.612］下降。结论 多中心数据导致病例的异质性更大，人工智能系统容易给出错误预测，但表现优于内镜医师平均水平。

Abstract:

Objective To evaluate the ability of ENDOANGEL artificial intelligence system to predict invasion depth and differentiation status of early gastric cancer using more diverse multi-center videos, and to test the performance of the new system upgraded from ENDOANGEL. Methods Based on the completed 2020 man-machine competition for early gastric cancer diagnosis using single-center videos, the second man-machine competition was conducted in 2022, involving 30 endoscopists from 30 hospitals across 10 Chinese provinces. A multi-center video cohort was retrospectively collected from 12 institutions in 8 provinces/municipalities in China. The study proceeded in 3 stages. First, the ENDOANGEL was re-tested on multi-center videos, its performance on single and multi-center videos was compared, then the ENDOANGEL was upgraded to ENDOANGEL-2022. Second, the second man‑machine competition was conducted between ENDOANGEL-2022 and 30 endoscopists using multi-center videos, and the performance between ENDOANGEL-2022, ENDOANGEL and endoscopists on multi-center videos were compared. Third, the ENDOANGEL-2022 was re-tested on the single-center videos previously collected in 2020, its performance on single and multi-center videos was also compared. Results Compared with the performance on single-center videos, the sensitivity of ENDOANGEL for predicting submucosal invasion of early gastric cancer decreased significantly [18.18% (2/11) VS 70.00% (7/10), P=0.030], but demonstrated comparable ability to predict undifferentiated type of early gastric cancer (P>0.05). On multi-center videos, in the respect of predicting submucosal invasion of early gastric cancer, the sensitivity of ENDOANGEL-2022 was higher than that of ENDOANGEL [40.00% (4/10) VS 18.18% (2/11), P=0.361], but inferior to that of 30 endoscopists [40.00% VS 52.04% (95%CI: 43.70%-60.38%), P<0.001]. The specificity of ENDOANGEL-2022 was lower than that of ENDOANGEL [82.86% (29/35) VS 100.00% (34/34), χ2=4.41, P=0.036] and higher than that of 30 endoscopists [82.86% VS 68.97% (95%CI: 60.83%-77.11%), P=0.018], the accuracy of ENDOANGEL-2022 was lower than that of ENDOANGEL [73.33% (33/45) VS 80.00% (36/45),χ2=0.56, P=0.455] and higher than that of 30 endoscopists [73.33% VS 65.30% (95%CI: 60.61%-69.99%), P=0.018]. In the respect of predicting undifferentiated type of early gastric cancer, the sensitivity of ENDOANGEL-2022 was higher than that of ENDOANGEL [71.43% (5/7) VS 57.14% (4/7), P>0.999] and 30 endoscopists [71.43% VS 63.11% (95%CI: 55.58%-70.64%), P=0.031], the specificity of ENDOANGEL-2022 was lower than that of ENDOANGEL [76.32% (29/38) VS 78.95% (30/38), χ2=0.08, P=0.783] and higher than that of 30 endoscopists [76.32% VS 65.27% (95%CI: 59.10%-71.44%), P=0.004],the accuracy of ENDOANGEL-2022 was similar to that of ENDOANGEL [75.56% (34/45) VS 75.56% (34/45), χ2=0.00, P>0.999] and higher than that of 30 endoscopists [75.56% VS 65.10% (95%CI: 59.96%- 70.24%), P<0.001]. Compared with performance in single center videos, the sensitivity [40.00% VS 60.00%(6/10), P=0.656], specificity [82.86% VS 93.75% (15/16), χ2=0.37, P=0.542] and accuracy [73.33% VS 80.77% (21/26), χ2=0.50, P=0.479] of ENDOANGEL-2022 for predicting submucosal invasion of early gastric cancer decreased; in predicting undifferentiated type of early gastric cancer, the sensitivity of ENDOANGEL-2022 increased [71.43% VS 37.50% (3/8), P=0.315], while the specificity [76.32% VS 100.00% (18/18), χ2=3.48, P=0.062] and accuracy [75.56% VS 80.77% (21/26), χ2=0.26, P=0.612] decreased. Conclusion Multi-center cases introduce greater heterogeneity that may reduce artificial intelligence prediction accuracy, but the artificial intelligence system still outperforms endoscopists.