SORA

Advancing, promoting and sharing knowledge of health through excellence in teaching, clinical practice and research into the prevention and treatment of illness

An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost-effectiveness.

Tufail, A; Kapetanakis, VV; Salas-Vega, S; Egan, C; Rudisill, C; Owen, CG; Lee, A; Louw, V; Anderson, J; Liew, G; et al. Tufail, A; Kapetanakis, VV; Salas-Vega, S; Egan, C; Rudisill, C; Owen, CG; Lee, A; Louw, V; Anderson, J; Liew, G; Bolter, L; Bailey, C; Sadda, S; Taylor, P; Rudnicka, AR (2016) An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost-effectiveness. Health Technol Assess, 20 (92). pp. 1-72. ISSN 2046-4924 https://doi.org/10.3310/hta20920
SGUL Authors: Owen, Christopher Grant Rudnicka, Alicja Regina

[img]
Preview
PDF Published Version
Available under License ["licenses_description_publisher" not defined].

Download (1MB) | Preview
[img] Microsoft Word (.docx) Accepted Version
Available under License ["licenses_description_publisher" not defined].

Download (962kB)

Abstract

BACKGROUND: Diabetic retinopathy screening in England involves labour-intensive manual grading of retinal images. Automated retinal image analysis systems (ARIASs) may offer an alternative to manual grading. OBJECTIVES: To determine the screening performance and cost-effectiveness of ARIASs to replace level 1 human graders or pre-screen with ARIASs in the NHS diabetic eye screening programme (DESP). To examine technical issues associated with implementation. DESIGN: Observational retrospective measurement comparison study with a real-time evaluation of technical issues and a decision-analytic model to evaluate cost-effectiveness. SETTING: A NHS DESP. PARTICIPANTS: Consecutive diabetic patients who attended a routine annual NHS DESP visit. INTERVENTIONS: Retinal images were manually graded and processed by three ARIASs: iGradingM (version 1.1; originally Medalytix Group Ltd, Manchester, UK, but purchased by Digital Healthcare, Cambridge, UK, at the initiation of the study, purchased in turn by EMIS Health, Leeds, UK, after conclusion of the study), Retmarker (version 0.8.2, Retmarker Ltd, Coimbra, Portugal) and EyeArt (Eyenuk Inc., Woodland Hills, CA, USA). The final manual grade was used as the reference standard. Arbitration on a subset of discrepancies between manual grading and the use of an ARIAS by a reading centre masked to all grading was used to create a reference standard manual grade modified by arbitration. MAIN OUTCOME MEASURES: Screening performance (sensitivity, specificity, false-positive rate and likelihood ratios) and diagnostic accuracy [95% confidence intervals (CIs)] of ARIASs. A secondary analysis explored the influence of camera type and patients' ethnicity, age and sex on screening performance. Economic analysis estimated the cost per appropriate screening outcome identified. RESULTS: A total of 20,258 patients with 102,856 images were entered into the study. The sensitivity point estimates of the ARIASs were as follows: EyeArt 94.7% (95% CI 94.2% to 95.2%) for any retinopathy, 93.8% (95% CI 92.9% to 94.6%) for referable retinopathy and 99.6% (95% CI 97.0% to 99.9%) for proliferative retinopathy; and Retmarker 73.0% (95% CI 72.0% to 74.0%) for any retinopathy, 85.0% (95% CI 83.6% to 86.2%) for referable retinopathy and 97.9% (95% CI 94.9 to 99.1%) for proliferative retinopathy. iGradingM classified all images as either 'disease' or 'ungradable', limiting further iGradingM analysis. The sensitivity and false-positive rates for EyeArt were not affected by ethnicity, sex or camera type but sensitivity declined marginally with increasing patient age. The screening performance of Retmarker appeared to vary with patient's age, ethnicity and camera type. Both EyeArt and Retmarker were cost saving relative to manual grading either as a replacement for level 1 human grading or used prior to level 1 human grading, although the latter was less cost-effective. A threshold analysis testing the highest ARIAS cost per patient before which ARIASs became more expensive per appropriate outcome than human grading, when used to replace level 1 grader, was Retmarker £3.82 and EyeArt £2.71 per patient. LIMITATIONS: The non-randomised study design limited the health economic analysis but the same retinal images were processed by all ARIASs in this measurement comparison study. CONCLUSIONS: Retmarker and EyeArt achieved acceptable sensitivity for referable retinopathy and false-positive rates (compared with human graders as reference standard) and appear to be cost-effective alternatives to a purely manual grading approach. Future work is required to develop technical specifications to optimise deployment and address potential governance issues. FUNDING: The National Institute for Health Research (NIHR) Health Technology Assessment programme, a Fight for Sight Grant (Hirsch grant award) and the Department of Health's NIHR Biomedical Research Centre for Ophthalmology at Moorfields Eye Hospital and the University College London Institute of Ophthalmology.

Item Type: Article
Additional Information: © Queen’s Printer and Controller of HMSO 2016. This work was produced by Tufail et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
Keywords: Health Policy & Services, 1117 Public Health And Health Services, 0807 Library And Information Studies, 0806 Information Systems
SGUL Research Institute / Research Centre: Academic Structure > Population Health Research Institute (INPH)
Journal or Publication Title: Health Technol Assess
ISSN: 2046-4924
Language: eng
Dates:
DateEvent
16 December 2016Published
1 July 2016Accepted
Publisher License: Publisher's own licence
PubMed ID: 27981917
Web of Science ID: WOS:000391378200003
Go to PubMed abstract
URI: https://openaccess.sgul.ac.uk/id/eprint/108481
Publisher's version: https://doi.org/10.3310/hta20920

Actions (login required)

Edit Item Edit Item