Clinical evaluation of artificial intelligence system based on fundus photograph in diabetic retinopathy screening

Authors:Li Meng,  Wang Gengyuan,  Xia Honghui,  Tang Xiaoying,  Feng Ziqing,  Yao Yongyu,  Huang Yijin,  Fan Wei,  Yuan Zhe,  Yuan Jin
DOI: 10.3760/cma.j.issn.2095-0160.2019.08.015
Published 2019-08-10
Cite as Chin J Exp Ophthalmol, 2019,37(8): 663-668.

Abstract

Objective

To study the efficiency and accuracy of artificial intelligence (AI) system based on fundus photograph in diabetic retinopathy(DR)screening, and evaluate the clinical application value of AI system.

Methods

A diagnostic trial was adopted.Total of 13 683 color fundus photos were collected in Zhaoqing Gaoyao People’s Hospital from March, 2017 to November, 2018.The AI system for DR (ZOC-DR-V1) was established, based on transfer learning + NASNet algorithm, by training 4 465 precisely labeled fundus images (2 510 normal, and 1 955 with any stage of DR). One thousand confirmed fundus images (300 normal and 700 with any stage of DR), diagnosed by AI (AI group) and doctors (3 ophthalmologist doctors and 3 endocrinologist doctors) (doctor group), respectively.Ophthalmologist group and endocrinologist group were both composed of primary, intermediate and senior physicians.The mean reading time of each image and the total time of 1 000 images were recorded.The accuracy and efficiency of AI system and doctor groups were compared.The reading process was divided into two stages.The diagnostic coincidence rate and the average reading time of each group between the two parts were calculated and compared.This study protocol was approved by Ethic Committee of Zhongshan Ophthalmic Center, Sun Yat-sen University (No.2017KYPJ104).

Results

After training, the diagnostic coincidence rate of AI system (ZOC-DR-V1) in test set was 94.7%, AUC was 0.994.In this “man-machine to war” , the diagnostic coincidence rate of primary, intermediate and senior endocrinologist was 94.0%, 91.4% and 93.4%; the diagnostic coincidence rate of primary, intermediate and senior ophthalmologist was 92.7%, 94.4% and 95.6%; the diagnostic coincidence rate of AI system was 95.2%.There was no difference in the diagnostic coincidence rate between AI system and senior ophthalmologist (P=0.749). The mean reading time of each image of primary, intermediate and senior endocrinologists was (4.63±1.87), (3.74±3.47) and (5.71±3.47) seconds, and the total time of 1 000 images of primary, intermediate and senior endocrinologists was 1.29, 1.04 and 1.58 hours; the mean reading time of each image of primary, intermediate and senior ophthalmologists was (7.25±6.58), (5.18±5.01) and (5.18±3.47) seconds, and the total time of 1 000 images of primary, intermediate and senior endocrinologists was 2.02, 1.44 and 1.44 hours; the mean and total time of AI system was (1.62±0.67) seconds and 0.45 hours, and the reading time of AI system was significantly shorter than that of the doctor groups (all at P=0.000). The diagnostic coincidence rates between previous and posterior part of primary endocrinologist, primary and intermediate ophthalmologist were significantly different (χ2=11.986, 6.517, 10.896; all at P<0.05), and the mean reading time in the posterior part was significantly shorter than that in the previous part of intermediate and senior endocrinologist and primary ophthalmologist (t=4.175, 8.189, 5.160; all at P<0.01). While the reading time of AI system remained stable throughout the process(χ2=3.151, P=0.103; t=0.038, P=0.970).

Conclusions

The ophthalmic AI system based on fundus images has a good diagnostic efficiency, and its diagnostic coincidence rate can compare with senior ophthalmologist, providing a new method and platform for large-scale DR screening.

Key words:

Artificial intelligence; Diabetic retinopathy/diagnosis; Screening; Reading

Contributor Information

Li Meng
Zhongshan Ophthalmic Center, Sun Yat-sen University, State Key Laboratory of Ophthalmology, Guangzhou 510060, China
Wang Gengyuan
Zhongshan Ophthalmic Center, Sun Yat-sen University, State Key Laboratory of Ophthalmology, Guangzhou 510060, China
Xia Honghui
Department of Ophthalmology, Zhaoqing Gaoyao People’s Hospital, Zhaoqing 526040, China
Tang Xiaoying
Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Feng Ziqing
Department of Ophthalmology, Zhaoqing Gaoyao People’s Hospital, Zhaoqing 526040, China
Yao Yongyu
Department of Ophthalmology, Zhaoqing Gaoyao People’s Hospital, Zhaoqing 526040, China
Huang Yijin
Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Fan Wei
Department of Ophthalmology, Zhaoqing Gaoyao People’s Hospital, Zhaoqing 526040, China
Yuan Zhe
Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Yuan Jin
Zhongshan Ophthalmic Center, Sun Yat-sen University, State Key Laboratory of Ophthalmology, Guangzhou 510060, China
(Read 61 times, 1 visits today)
Updated: October 9, 2019 — 6:39 am