Application of standardized manual labeling on identification of retinopathy of prematurity images in deep learning

Authors: Wang Ji,  Zhang Guihua,  Lin Jianwei,  Ji Jie,  Qiu Kunliang,  Zhang Mingzhi

DOI: 10.3760/cma.j.issn.2095-0160.2019.08.013
Published 2019-08-10
Cite as Chin J Exp Ophthalmol, 2019,37(8): 653-657.

Abstract

Objective

To evaluate the application of the standard manual labeling on identification of retinopathy of prematurity (ROP) images in deep learning.

Methods

According to the International Classification of ROP, different periods of ROP were classified into stage disease and plus disease in this study.From Joint Shantou International Eye Center from August 2009 to July 2018, a total of 1 464 labeled fundus retinal photographs were divided randomly by stratified sampling into 3 groups: stage disease group(subgroup 1∶173, subgroup 2∶117) was used to train for labeling stage disease, whereas plus disease group(subgroup 1∶163, subgroup 2∶116) was used to train for labeling plus disease, and consistent labels group consisted of 895 consistent labeled images on both disease.Graders consisted of senior experts, 3 senior ophthalmologists and 2 interns, and received training for classification and labeling on ROP fundus images.The results were compared among the doctors and doctors with deep learning, and the agreement between non-experts doctors and the reference standards, and deep learning and the reference standards were tested.

Results

After the first training, the overall agreement rate of the senior ophthalmologist group and the intern group were lower than 90% for both two disease labeling.After two to three times of training, in image of consistent labels group, overall agreement rates of senior ophthalmologists and intern doctor’s were 98.99% (Kappa=0.979), 99.22% (Kappa=0.984) on stage disease, and 97.43% (Kappa=0.914), 98.11% (Kappa=0.935) on plus disease, respectively.The agreement on stage disease using deep learning based on human-machine combination was 94.08%, Kappa value was 0.880, which achieved good degree.

Conclusions

Standardized manual labeling can improve the intelligentization of deep learning on identification of ROP images, and be considered as an innovative method of homogenization and standardized training for doctors in ophthalmology.

Key words:

Artificial intelligence; Standardized label; Deep learning; Retinopathy of prematurity; Stage disease; Plus disease

Contributor Information

Wang Ji
Joint Shantou International Eye Center, Shantou University & the Chinese University of Hong Kong, Shantou 515041, China
Zhang Guihua
Joint Shantou International Eye Center, Shantou University & the Chinese University of Hong Kong, Shantou 515041, China
Lin Jianwei
Joint Shantou International Eye Center, Shantou University & the Chinese University of Hong Kong, Shantou 515041, China
Ji Jie
Network & Information Center, Shantou University, Shantou 515063, China
Qiu Kunliang
Joint Shantou International Eye Center, Shantou University & the Chinese University of Hong Kong, Shantou 515041, China
Zhang Mingzhi
Joint Shantou International Eye Center, Shantou University & the Chinese University of Hong Kong, Shantou 515041, China
(Read 51 times, 1 visits today)
Updated: September 25, 2019 — 10:57 am