Application of standardized manual labeling on identification of retinopathy of prematurity images in deep learning

Objective

To evaluate the application of the standard manual labeling on identification of retinopathy of prematurity (ROP) images in deep learning.

Methods

According to the International Classification of ROP, different periods of ROP were classified into stage disease and plus disease in this study.From Joint Shantou International Eye Center from August 2009 to July 2018, a total of 1 464 labeled fundus retinal photographs were divided randomly by stratified sampling into 3 groups: stage disease group(subgroup 1∶173, subgroup 2∶117) was used to train for labeling stage disease, whereas plus disease group(subgroup 1∶163, subgroup 2∶116) was used to train for labeling plus disease, and consistent labels group consisted of 895 consistent labeled images on both disease.Graders consisted of senior experts, 3 senior ophthalmologists and 2 interns, and received training for classification and labeling on ROP fundus images.The results were compared among the doctors and doctors with deep learning, and the agreement between non-experts doctors and the reference standards, and deep learning and the reference standards were tested.

Results

After the first training, the overall agreement rate of the senior ophthalmologist group and the intern group were lower than 90% for both two disease labeling.After two to three times of training, in image of consistent labels group, overall agreement rates of senior ophthalmologists and intern doctor’s were 98.99% (Kappa=0.979), 99.22% (Kappa=0.984) on stage disease, and 97.43% (Kappa=0.914), 98.11% (Kappa=0.935) on plus disease, respectively.The agreement on stage disease using deep learning based on human-machine combination was 94.08%, Kappa value was 0.880, which achieved good degree.

Conclusions

Standardized manual labeling can improve the intelligentization of deep learning on identification of ROP images, and be considered as an innovative method of homogenization and standardized training for doctors in ophthalmology.