Most cases of ROP are mild to moderate and resolve without serious sequelae. Some cases of ROP are severe, and they are characterized by abnormal dilation and tortuosity of the arterioles and venules (plus disease) and growth of abnormal neovascular tissue on the surface of the retina (stage 3). In the Cryotherapy for Retinopathy of Prematurity (CRYO-ROP) clinical trial, 245 of 4099 infants (6%) developed severe, sight-threatening (threshold) ROP, and 2699 of 4099 (66%) infants developed less severe forms of the disease.⁵ In cases of severe ROP, peripheral retinal ablation is done using laser or, less commonly today, using cryotherapy. These treatments reduce the secretion of vasoactive cytokines that stimulate proliferation of neovascular tissue, resulting in regression of disease in most cases. Despite timely treatment, some infants still experience severe disease that progresses to retinal detachment and blindness. Of those eyes treated with cryotherapy and followed for 15 years in the CRYO-ROP study, 85 of 190 (45%) had a visual acuity of 20/200 or worse, and 54 of 180 (30%) had an unfavorable anatomic outcome, defined as a posterior retinal fold or worse.⁶

FIGURE 1 Typical appearance of severe (threshold) retinopathy of prematurity, showing plus disease (abnormal dilation and tortuosity of posterior retinal vessels), stage 3 (neovascular tissue) at the border of vascular and avascular retina, and the locations of (more ...)

Plus disease is severely abnormal dilation and tortuosity of posterior retinal blood vessels, and it exists when an eye is judged to have at least as much dilation and tortuosity of its posterior retinal vessels as a standard photograph that was used in the CRYO-ROP study (Figure 2).⁵ A revised classification of ROP was published in 2005,⁸ and it included an intermediate grade of posterior retinal vascular abnormality known as “pre-plus” disease, which is believed in most cases to be a precursor to plus disease. Whereas plus disease refers to severely abnormal vascular changes, the designation of pre-plus disease applies to mildly to moderately abnormal dilation and tortuosity of the central, posterior retinal blood vessels (Figure 3).

FIGURE 2 Standard photograph of plus disease in retinopathy of prematurity representing the minimum degree of dilation and tortuosity necessary for plus disease. (Reprinted with permission from Archives of Ophthalmology.5)

FIGURE 3 Spectrum of posterior pole vascular changes in infants with retinopathy of prematurity. 0 = normal, 1 = mild venous dilation, 2 = mild venous dilation and mild arteriolar tortuosity (pre-plus disease), 3 = moderate vascular dilation and tortuosity (pre-plus, (more ...)

Investigators in the Early Treatment for ROP (ETROP) study hypothesized that treatment at high-risk prethreshold would reduce the incidence of unfavorable visual outcomes compared to standard treatment at threshold.¹¹ Prethreshold was defined as any of the following: (1) any ROP in zone I, (2) ROP in zone II with plus disease, (3) zone II with stage 3 ROP and no plus disease, or (4) zone II with plus disease and stage 3 ROP but with less than the 5 contiguous or 8 cumulative clock hours of stage 3 required for the diagnosis of threshold ROP. “High-risk” was determined using a computerized model based on risk factors for progression identified in the CRYO-ROP natural history cohort. Three hundred seventeen bilateral cases and 84 asymmetric cases reached high-risk prethreshold, and eyes were randomly assigned either to laser treatment within 48 hours or to standard treatment at threshold if it developed. After 9 months, an unfavorable visual outcome had occurred in 14.5% of eyes in the early treated group compared to 19.5% of eyes receiving standard treatment. (P = .01) An unfavorable anatomic outcome had occurred in 9.1% of eyes in the early treated group compared to 15.6% of eyes receiving standard treatment. (P < .001). Ocular complication rates were similar in the 2 groups, although systemic complication rates were higher in the early treated group. After 9 months, myopia of 5 D or greater was present in 26% of early treated eyes and in 28% of eyes treated conventionally (P = .2). Based on the results of this study, laser treatment is now strongly considered for any eye with type 1 ROP, defined as (1) zone I, any stage with plus disease, (2) zone I, stage 3 without plus disease, or (3) zone II, stage 2 or 3 with plus disease.¹²

It has been hypothesized that the severely abnormal dilation and tortuosity of blood vessels characteristic of plus disease occur as a result of reduced capillary resistance and increased retinal blood flow through developing arteriovenous shunts.^14–16 This hypothesis is supported by a strong correlation between the appearance of posterior arterioles and venules and the severity of ROP in the retinal periphery.^15,17 However, Holland and associates¹⁸ used color Doppler imaging to show that blood flow (peak systolic velocity) through the optic nerve was not increased in eyes with plus disease. An alternative explanation for the appearance of plus disease is that vascular endothelial growth factor, which is produced in response to retinal ischemia, may be present in sufficient amounts in the vitreous of the eye to act directly on central retinal blood vessels and alter their morphology (Ho M, ARVO Meeting, 2000. Abstract). This proposed mechanism may also help to explain the observation that plus disease occurs before obvious peripheral shunting and neovascularization in some cases of severe, posterior ROP.

In the first study,²⁵ video clips of posterior pole images captured at the examination closest to 33 weeks postmenstrual age were analyzed. Images of 64 eyes of 32 infants screened during an 18-month time period were randomly ordered and graded by 2 masked examiners. The examiners graded the images from 0 to 4 in comparison to standard photographs of 5 representative grades of retinal vascular dilation and tortuosity (Figure 3). The standard photographs were digital images captured from video clips taken during indirect ophthalmoscopy. The examiners disagreed on the precise grade in 16 of 64 images (25%), and in these cases, the grades of the 2 examiners were averaged. The incidences of progression to stage 3, development of plus disease, and requirement of laser treatment were compared between infants with normal posterior retinal blood vessels and those judged to have early dilation and tortuosity insufficient for plus disease. This analysis included only one eye per infant, which was the right eye unless the left eye had more severe disease. Of 8 eyes with mild vascular dilation and tortuosity insufficient for plus disease, 5 (63%) progressed to plus disease, 5 (63%) eventually required laser treatment, and 4 (50%) developed stage 3. (These outcomes were not mutually exclusive, and some eyes were treated prior to threshold.) Of the 24 eyes with normal posterior retinal vessels, none required laser treatment, 2 (8%) developed stage 3, and none progressed to plus disease. The group with vascular dilation and tortuosity insufficient for plus disease had a significantly higher incidence of progression to laser treatment (P = .0003), stage 3 (P = .027), and plus disease (P = .0003).²⁵

The results of this study suggested that pre-plus disease had prognostic significance at the examination performed closest to 33 weeks postmenstrual age, but whether it was applicable to the entire screening period remained unknown. Therefore, a second pilot study was performed using serial RetCam photographs that had been collected as part of a separate study (Wallace DK, AAPOS Meeting, 2003. Abstract). RetCam photographs were taken at every examination of 44 infants for an entire year, for a total of 371 serial images over the entire screening period.²⁶ In order to assess the prognostic significance of pre-plus disease at various ages, a masked examiner graded the posterior retinal blood vessels of these images in comparison to the same standard photographs used in the first pilot study. The incidences of progression to stage 3, development of plus disease, and requirement of laser treatment were compared between eyes with normal posterior retinal vessels and those with pre-plus vascular changes at various ages throughout the screening period.

Eighteen of the infants had serial photographs taken throughout their entire screening period and were included in the data analysis. In comparison to eyes (one per patient) with normal posterior retinal vessels on every examination, those with pre-plus changes on at least one examination had a higher incidence of stage 3 (5/11, 45% vs 1/4, 25%), plus disease (4/11, 36% vs 0/4, 0%), and laser treatment (3/11, 27% vs 0/4, 0%), but these differences were not statistically significant. In 7 eyes that required laser treatment, plus disease was the first ominous sign in 3 cases and pre-plus was the first sign (appearing prior to stage 3 or plus disease) in 4 cases. Pre-plus disease was most predictive of progression to stage 3 when it was present at 34 to 39 weeks postmenstrual age. It was predictive of development of plus disease when present at 33 to 34 and 36 weeks, and it predicted need for laser treatment when present at 33 to 38 weeks.

In the second pilot study, the natural history of posterior retinal blood vessel changes was also studied (Wallace DK, AAPOS Meeting, 2003. Abstract). It was found that grade 2 (mild dilation and tortuosity) was used much more commonly by the masked examiners than either grade 1 (mild dilation) or grade 3 (almost plus disease) to describe the images in the study. In addition, of 8 patients with grade 2 vascular changes, only 2 had prior examinations with grade 1 findings, suggesting that many eyes develop mild vascular dilation and tortuosity simultaneously or so close in time as to escape detection with examinations every 1 to 2 weeks.

Consequently, there remains controversy regarding the usual progression of central retinal blood vessel changes in ROP. Some examiners believe that, contrary to popular opinion, vascular tortuosity precedes dilation in some cases. It may be that the natural history of vascular changes, as well as the timing of events, depends on the location of the disease; that is, more posterior disease tends to progress more rapidly and may develop dilation and tortuosity simultaneously. However, no published study has reported on the natural progression of vascular changes in a large number of infants, and such a study is necessary to verify or refute the accuracy of current classification systems of posterior retinal vascular changes in ROP.

An accurate, automated measure of plus disease is needed for several additional reasons. It has become more difficult for many neonatal nurseries in the United States to find ophthalmologists willing to perform ROP examinations. This problem is due in part to fewer graduates from pediatric ophthalmology fellowships in recent years as well as the substantial medicolegal risk of screening for ROP. Also, the number of infants requiring ROP examinations has increased owing to improved survival of very low-birth-weight infants and recent changes in screening guidelines to now include infants born at 29 or 30 weeks gestational age with birth weights over 1500 g.²⁷ Many developing countries are facing an increasing number of surviving premature infants without an adequate number of ophthalmologists trained to screen and treat ROP. Telemedicine offers a potential solution for these problems, and computer-assisted analysis of images would greatly enhance a telemedicine approach to screening.

In response to these needs, interest has grown in recent years in measuring as accurately as possible the degree of blood vessel dilation and tortuosity. In collaboration with computer scientists, a computer program was developed that automatically traced retinal blood vessels and quantified the degree of tortuosity and dilation. This technology was first applied to the precise measurement of cerebral vessel dilation and tortuosity in angiograms.^28–31 Three pilot studies have been performed to assess the accuracy of this computer program at various stages of its evolution.³² In the first study, a representative sample of posterior retinal images from 20 premature infants, 10 normal and 10 representing various degrees of dilation and tortuosity, was extracted from a video database and analyzed by both the computer program and by 2 masked examiners. The standard photograph from the CRYO-ROP study (Figure 2) representing the minimum degree of dilation and tortuosity required for plus disease was also digitized, analyzed, and used as a numeric comparison for the automated determination of plus disease. Of the 5 images determined to have plus disease by both examiners, 4 were calculated to have plus disease by the computer program (80% sensitivity). Of the 11 images without plus disease, 10 were calculated to not have plus disease by the computer program (91% specificity).³² The examiners disagreed on the presence or absence of plus disease for 3 images, and 1 poor-quality image was excluded.

A second study utilized RetCam images from 22 premature infants collected as part of the Photographic Screening for Retinopathy of Prematurity (PHOTO-ROP) study.³³ Of 6 images determined to have plus disease by 2 masked examiners, 5 were calculated to have plus disease by the computer program (83% sensitivity). Of 13 images without plus disease, 9 were calculated to not have plus disease by the computer program (69% specificity) (Wallace DK, ARVO Meeting, 2003. Abstract). Although promising, these 2 pilot studies had several limitations. First, the sample sizes were small, and larger samples are required to obtain more accurate estimates of sensitivity and specificity. Second, accurate determination of vessel dilation was problematic owing to variables that can result in variations of image size, such as the power of condensing lens used to capture the image. Third, the blood vessels were often not traced by the computer for their entire length. In particular, sharp turns in a vessel were sometimes missed, and this problem affected the measurement of tortuosity. One of the most important variables in success of the tracing program was the quality of the digitized image. Those blood vessels that were out of focus or exhibited poor contrast against the fundus background were more difficult for the computer to identify, trace, and analyze.

Since completion of these pilot studies, the computer program has been refined and named “ROPtool.” The program was previously unable to consistently identify on its own the major vessels from each image, so a feature was added that allowed the operator to click on individual vessels. The program now traces the selected vessel from the border of the optic nerve to the extent of the view that an examiner would have with a 28 D lens centered on the optic nerve. This view subtends an angle of approximately 30 degrees, and it includes those segments of arterioles and venules that are generally used to determine the presence or absence of plus disease. Another problem in the pilot studies was that the tortuosity measure was calculated as the ratio of the total vessel length to the distance of a straight line drawn between the beginning and ending points of the vessel. However, this simple formula had the effect of artificially increasing the tortuosity of a nontortuous vessel that simply makes a large, gradual curve (Figure 4). The program was refined to choose points along the vessel that are 200 points (about 40 pixels) apart and to draw a smooth curve connecting these points. The tortuosity measure used by ROPtool is now the ratio of the total length of the vessel to the length of this smooth curve, which likely gives a more accurate measure of tortuosity. Also, features have been added to allow the operator to mark the borders of the optic nerve and the center of the macula. Finally, the interface was modified to make it “user friendly” and to include functions such as cutting, merging, deleting, and undoing vessel tracings. The result is the current working version of ROPtool shown in Figure 5.

FIGURE 4 Two methods of measuring tortuosity of retinal blood vessels: A, Using the simplest method, tortuosity is calculated as the ratio of the total vessel length (purple line) to a straight line drawn between points 1 and 2 (red line). This method will overestimate (more ...)

FIGURE 5 User interface for ROPtool, a computer program that traces retinal blood vessels of premature infants and measures their tortuosity. The operator clicks on any of the functions on the right and then executes this function by clicking on the image on the (more ...)

Recently, this new version of ROPtool was piloted in order to troubleshoot for problems, determine its reliability (repeatability), assess its accuracy as a tool to measure dilation and tortuosity, and calculate appropriate numerical thresholds for tortuosity sufficient for plus disease and pre-plus disease.³⁴ During the pilot study, it was noted that the same vessels were sometimes traced twice and that tracing occasionally continued around the optic nerve border, and ROPtool was modified to correct these problems. Reliability and accuracy of the tortuosity measure of ROPtool were equal to or better than determination of tortuosity sufficient for plus disease by investigator judgment. Tortuosity of both arterioles and venules were included in the overall tortuosity measure, primarily because it is not always possible to accurately differentiate between them, and it is common for arterioles and venules to become tortuous as plus disease develops. The measurement of dilation was disappointing in that ROPtool did not consistently discriminate between dilation that was or was not sufficient for plus disease as judged by 2 experts.

In summary, a computer program (ROPtool) has been developed that automatically traces retinal blood vessels, measures the tortuosity of each vessel in comparison to the standard photograph of plus disease, and calculates whether an eye has sufficient tortuosity for plus disease or not. Several pilot studies have been completed, and results have been used to refine the computer program. The purpose of this study was to assess the accuracy of measurement of vascular tortuosity by ROPtool. It was hypothesized that determination of the presence or absence of tortuosity sufficient for plus disease by ROPtool would be superior to judgment by individual pediatric ophthalmologists.

Six pediatric ophthalmology colleagues of the author were identified and asked to participate in the study. Three of them were considered by the author to be ROP experts, based on their extensive ROP experience and certification as investigators in the ETROP study. The other 3 ophthalmologists (the “examiners”) had different amounts of experience in ROP diagnosis and treatment. The study was designed to challenge the accuracy of the examiners individually vs ROPtool, and judgment of the experts was considered to be the reference standard for comparison.

RetCam (Clarity Medical Systems, Pleasanton, California) photographs of the posterior retina of 190 different eyes of premature infants were collected. This number was based on a priori sample size calculations. None of the infants could be identified from the retinal images, so an institutional review board exemption was granted for this study. The sample was enhanced to include a larger proportion of images with plus disease and pre-plus disease than would normally be encountered during routine screening examinations, including as many images as possible with “borderline” plus disease. The ages of the infants at the time of the photographs were not known, so age was not considered when selecting images. Images from these 190 eyes were not included in any of the pilot studies that were used to train ROPtool. Some images came from photographic databases of previously published studies done elsewhere,^26,33 and some images were from teaching files. Most images were very high quality and in sharp focus, with only a few images slightly blurred or decentered.

Adobe Photoshop (Adobe, San Jose, California) was used to crop each image in the shape of a circle centered on the optic nerve that approximated the view seen with a 28 D lens (Figure 6). This view is the typical one of the posterior retinal vessels that is initially seen on examination and used to assess the presence or absence of plus disease. Cropping the RetCam images in this manner assured that all ophthalmologists and ROPtool used the same view and extent of the retina to judge tortuosity sufficient for plus disease and those more peripheral findings, such as stage 3 disease, did not influence this judgment. The cropped images were randomly ordered and distributed to the 6 ophthalmologists for grading. The same images were analyzed using ROPtool by the author (the “operator”), who was masked to the grades of the other ophthalmologists.

FIGURE 6 Cropped version of RetCam image of a premature infant’s retina approximating the view obtained by a 28 D lens with the optic nerve centered.

All 6 pediatric ophthalmologists were informed about the general purpose of the study and sent a copy of the standard photograph of plus disease (Figure 2) as well as examples of plus disease and pre-plus disease from the ICROP Revisited publication.⁸ They were also reminded of the definition of plus disease (“the standard photograph defines the minimum amount of vascular dilation and tortuosity required to make the diagnosis of plus disease”) and pre-plus disease (“vascular abnormalities of the posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arterial tortuosity and more venous dilation than normal”). Each ophthalmologist then independently scored each quadrant of each image by grading tortuosity and dilation separately (8 total grades per eye) as plus, pre-plus or normal. These scores were used to generate both quadrant-level and eye-level data. Quadrant-level data were based on individual quadrant grades, and eye-level grades were based on a combination of the quadrant-level grades for each image. For example, an eye-level grade of tortuosity sufficient for plus disease was present if at least 2 of the 4 quadrants in a single eye had tortuosity sufficient for plus disease.

The combined grades of the three experts were used to establish the reference judgment of tortuosity sufficient for plus disease (“plus tortuosity”) for each image. Expert consensus of eye-level tortuosity sufficient for plus disease versus not plus disease (the reference standard) was determined as follows. If at least 2 quadrants had tortuosity sufficient for plus disease, then the eye was determined to have tortuosity sufficient for plus disease. When there was disagreement as to whether there was or was not eye-level tortuosity sufficient for plus disease, the expert who was the outlier was asked to reconsider the image. Those images for which disagreement still existed were discussed on a conference call between the experts. If consensus was still not reached, then each expert recast a vote, and consensus was determined by majority. Expert grades for eye-level determination of tortuosity sufficient for pre-plus disease (“pre-plus tortuosity”) or for quadrant-level determination of tortuosity sufficient for plus or pre-plus disease were not adjudicated, since they were not primary outcome measures. An eye was considered to have tortuosity sufficient for pre-plus disease if at least 2 quadrants had pre-plus tortuosity, or if 1 quadrant had plus tortuosity and 1 or more quadrants had pre-plus tortuosity.

The 3 examiners had different amounts of experience in ROP diagnosis and treatment in order to represent pediatric ophthalmologists in practice. The total number of ROP examinations performed by the each of them was approximately 1000, 7000, and 10,000, respectively. Their grades for the images were used to test the primary hypothesis that the computer program (ROPtool) is more accurate than an individual examiner in determining the presence or absence of tortuosity sufficient for plus disease. No examiner grades were adjudicated, since each examiner’s grades were compared separately to expert consensus.

FIGURE 7 Appearance of a retinal image after analysis by the computer program ROPtool. Solid blue circle is optic nerve border; dotted blue circle is 28 D circle; green lines are borders of quadrants; red lines are traced vessels whose tortuosity has been calculated. (more ...)

Next, the operator clicks on the major blood vessels (arterioles or venules) one at a time, and ROPtool automatically traces the selected blood vessel in 2 directions. The tracing of the vessel ends when (1) it encounters the optic nerve border or the 28D circle, (2) it begins to follow the course of a vessel that has already been traced, or (3) it no longer finds a vessel to trace. The vessel extraction will continue through an intersection with another traced vessel. As the vessel is traced, ROPtool automatically calculates vessel tortuosity, which is a ratio of the total length of the vessel divided by a smooth curve generated from points spaced 200 points (about 40 pixels) apart on the vessel (Figure 4). Tortuosity index is defined as the vessel tortuosity divided by the average tortuosity of the standard photograph of plus disease⁵ multiplied by 10. Therefore, a vessel with a tortuosity index of 15 has 50% more tortuosity that the average tortuosity of the major blood vessels in the standard photograph. The program also calculates dilation and dilation index, but this feature of the program has not been fully developed and is not the subject of this investigation. If ROPtool traces the wrong vessel, then the operator can click on “Undo Extract” to cancel the most recent tracing. Sometimes, 2 segments of the same vessel are traced separately, because ROPtool does not recognize them as part of the same vessel. In these cases, the “Merge” function can be used to fuse the 2 vessels into one, generating a new tortuosity value for the merged vessel that is an average weighted by the length of each segment. If ROPtool partially traces a vessel but then inaccurately continues to trace away from it, then the “Cut” function can be used to splice a vessel into 2 parts, allowing the inaccurate segment to be deleted. (In this study, the operator attempted to identify and click on the major arteriole and venule in each of the 4 quadrants, tracing a total of 8 vessels for each image. Tortuosity of both arterioles and venules was usually included, primarily because it was not always possible to accurately differentiate between arterioles and venules. Sometimes, fewer or more vessels were included, although at least 1 major vessel was always traced in each quadrant.)

Once all of the major vessels are identified, the operator clicks on “Compute,” and ROPtool calculates the mean tortuosity of each quadrant and determines whether eye-level tortuosity is sufficient for plus disease, based on two quadrants meeting or exceeding a numeric threshold for tortuosity sufficient for plus disease. (For this study, the plus disease threshold was set a priori at a tortuosity index of 9, because it was more accurate in pilot studies than the default value of 10 that would correspond with the average tortuosity of the vessels in the standard photograph of plus disease.) Values for each quadrant and for each vessel can then be reviewed individually. Finally, all of the values can be transferred to a database such as Microsoft Excel by clicking on “Save DB”.

Sensitivity, specificity, and concordance (overall accuracy) were calculated in comparison to the reference standard (1) for the assessment of tortuosity sufficient for plus disease by ROPtool and (2) for judgment of each of the examiners individually. Sensitivity was calculated by dividing the number of true positives by the number of positives by the reference standard. Specificity was calculated by dividing the number of true negatives by the number of negatives by the reference standard. Concordance was calculated by dividing the total number of agreements (true positives plus true negatives) by the total number of images. To test the primary hypothesis, the concordance of each examiner was compared to the concordance of ROPtool. This hypothesis was tested 3 times, once for each examiner in comparison to ROPtool. Sensitivities and specificities of ROPtool were also compared to that of the examiners individually.

Data were collected and analyzed in part using Microsoft Office Excel 2003, and some statistical analyses were done using SAS 9.1 (Cary, North Carolina). Concordance, sensitivity, and specificity were estimated by assigning an equal weight to each eye. To account for possible dependency between eyes of each patient, a generalized estimating equation method with a working independent correlation model was used.³⁵ Mean tortuosities were compared using a t test. All P values were 2-sided.

Initially, the 3 experts disagreed on the presence or absence of tortuosity sufficient for plus disease for 27 of the 185 analyzable images. After the outlier expert for each image reconsidered it, the number of disagreements decreased to 17. On the conference call, agreement of all 3 experts was ultimately reached on all images except one. For this single image, the judgment of tortuosity sufficient for plus disease was determined by majority opinion (2 of 3).

At the quadrant level, there were 190 images with 4 quadrants each, or 760 total quadrants. Of these, 17 quadrants (2%) were excluded because of insufficient image quality, leaving a total of 743 units for the quadrant-level analyses. Three experts independently graded a total of 2259 quadrants, and of these, 313 quadrants (14%) were graded as tortuous enough for plus disease, 474 quadrants (21%) were graded as tortuous enough for pre-plus disease but not tortuous enough for plus disease, and 1472 quadrants (65%) were graded as normal.

The operator was able to successfully trace all of the images except one, and this image was one of the three excluded from the primary analysis because of insufficient image quality. The analysis using ROPtool took approximately 5 minutes per image. The better the image quality, the simpler and faster the analysis could be performed. Images of poorer quality due to image blur and/or those with less contrast between vessels and retina were more difficult and time-consuming to analyze because the “cut” and “delete” edit functions had to be used more frequently.

TABLE 1 NUMBER OF AGREEMENTS AND DISAGREEMENTS WITH REGARD TO THE PRESENCE OR ABSENCE OF PLUS DISEASE BETWEEN CONSENSUS OF 3 RETINOPATHY OF PREMATURITY EXPERTS AND THE COMPUTER PROGRAM (ROPtool) AND 3 PEDIATRIC OPHTHALMOLOGISTS

TABLE 2 CONCORDANCE, SENSITIVITY, AND SPECIFICITY OF THE COMPUTER PROGRAM (ROPtool) AND 3 INDIVIDUAL EXAMINERS IN DETERMINING THE PRESENCE OR ABSENCE OF PLUS DISEASE IN COMPARISON TO CONSENSUS OF 3 EXPERTS*

TABLE 3 EFFECT OF USING DIFFERENT NUMERIC THRESHOLDS (“CUTOFF POINTS”) ON THE AGREEMENT OR DISAGREEMENT OF THE COMPUTER PROGRAM (ROPtool) WITH EXPERT CONSENSUS*

FIGURE 8 Receiver operating characteristic curve showing sensitivity vs 1-specificity (false-positive rate) for the detection of tortuosity sufficient for plus disease in retinopathy of prematurity at the eye level (unit = eye) by the computer program (ROPtool) (more ...)

The numeric threshold for tortuosity sufficient for pre-plus disease of 7 tortuosity units was also determined a priori based on pilot data. In comparison to expert consensus of eye-level tortuosity sufficient for plus or pre-plus disease, ROPtool had 161 agreements (87%) and 24 disagreements (13%) (Table 4). The concordance with expert consensus of tortuosity sufficient for plus or pre-plus disease for the 3 examiners was 93%, 91%, and 90%, respectively. Figure 9 is an ROC curve showing the effect of changing the numeric threshold for tortuosity sufficient for pre-plus disease on the sensitivity and specificity of ROPtool. It shows that 7 tortuosity units is a reasonably good threshold for tortuosity sufficient for pre-plus disease. Although sensitivity increases slightly at values below 7 tortuosity units, specificity drops off sharply (ie, there are many more false-positive results).

TABLE 4 NUMBER OF AGREEMENTS AND DISAGREEMENTS WITH REGARD TO THE PRESENCE OR ABSENCE OF PLUS DISEASE OR PRE-PLUS DISEASE BETWEEN CONSENSUS OF 3 EXPERTS AND THE COMPUTER PROGRAM (ROPtool) AND 3 PEDIATRIC OPHTHALMOLOGISTS

FIGURE 9 Receiver operating characteristic curve showing sensitivity vs 1-specificity (false-positive rate) for the detection of tortuosity sufficient for plus or pre-plus disease in retinopathy of prematurity at the eye level (unit = eye) by the computer program (more ...)

TABLE 5 MEAN QUADRANT TORTUOSITY MEASURED BY COMPUTER PROGRAM (ROPtool) FOR EACH LEVEL OF MEAN EXPERT GRADE

FIGURE 10 Mean quadrant tortuosity (and 95% confidence intervals [CI]) measured by the computer program (ROPtool) for 3 levels of mean expert grade. Three experts graded the tortuosity of each quadrant of images of premature infants’ retinal blood vessels (more ...)

Quadrant-level data were also used to generate ROC curves showing the effect of changing the numeric threshold for tortuosity sufficient for plus disease and for pre-plus disease on the sensitivity and specificity of ROPtool. The quadrant-level ROC curve for tortuosity sufficient for plus disease confirms that a tortuosity index of 9 is a good numeric threshold for plus tortuosity, with a sensitivity of 96% and specificity of 87% (Figure 11). The quadrant-level ROC curve for tortuosity sufficient for pre-plus disease confirms that a tortuosity index of 7 is a good numeric threshold for pre-plus tortuosity, with a sensitivity of 96% and specificity of 87% (Figure 12).

FIGURE 11 Receiver operating characteristic (ROC) curve showing sensitivity vs 1-specificity (false-positive rate) for the detection of tortuosity sufficient for plus disease in retinopathy of prematurity at the quadrant level (unit = quadrant) by the computer (more ...)

FIGURE 12 Receiver operating characteristic (ROC) curve showing sensitivity versus 1-specificity (false-positive rate) for the detection of tortuosity sufficient for plus or pre-plus disease in retinopathy of prematurity at the quadrant level (unit = quadrant) (more ...)

TABLE 6 AGREEMENT AND DISAGREEMENT BETWEEN 3 EXPERTS WITH REGARD TO EYE-LEVEL TORTUOSITY SUFFICIENT FOR PLUS DISEASE

TABLE 7 AGREEMENT AND DISAGREEMENT BETWEEN 3 EXPERTS WITH REGARD TO EYE-LEVEL TORTUOSITY SUFFICIENT FOR PLUS DISEASE IN THOSE EYES WHERE EXPERT CONSENSUS WAS PLUS DISEASE OR PRE-PLUS DISEASE (EXCLUDING NORMAL EYES)

TABLE 8 AGREEMENT AND DISAGREEMENT BETWEEN 3 EXPERTS WITH REGARD TO EYE-LEVEL TORTUOSITY SUFFICIENT FOR PRE-PLUS DISEASE

ROPtool is a computer program that measures retinal vascular tortuosity and calculates whether this tortuosity is sufficient for plus disease or pre-plus disease. In this study, ROPtool showed excellent sensitivity (97%) and specificity (94%) in the detection of tortuosity sufficient for plus disease when consensus of 3 ROP experts was considered to be the reference standard for comparison. Individual examiners also performed well in comparison to expert consensus, so there was not a statistically significant difference in overall concordance with experts between ROPtool (95%) and any of the individual examiners (93%, 93%, and 91%). Therefore, the null hypothesis that ROPtool is no more accurate than individual examiners in assessing the presence or absence of tortuosity sufficient for plus disease could not be rejected. However, 97% sensitivity by ROPtool in detecting tortuosity sufficient for plus disease was superior to that of the individual examiners, who had lower sensitivities of 65%, 70%, and 81%. Specificities were clinically similar among ROPtool (94%) and the individual examiners (100%, 99%, and 93%), although there was a statistically significant difference between specificity of ROPtool and the 2 examiners who had specificities of 100% and 99%, respectively. ROPtool also demonstrated the ability to discriminate between plus tortuosity, pre-plus tortuosity, and normal as shown by the significant difference in mean tortuosities between each of these categories (4.8, 9.9, and 19.1 tortuosity units, respectively).

The timely and accurate diagnosis of plus disease is extremely important. The ETROP study demonstrated a significant benefit to treatment of eyes with high-risk prethreshold disease.¹² Consequently, the recently revised guidelines endorsed by the American Academy of Ophthalmology, the American Association for Pediatric Ophthalmology and Strabismus, and the American Academy of Pediatrics recommended that laser treatment be performed for eyes with type 1 ROP, defined as (1) zone I ROP with plus disease, (2) zone II, stage 2 or 3 with plus disease, or (3) zone I, stage 3 without plus disease.²⁷ Since the third criterion (zone I, stage 3 without plus disease) is relatively uncommon, it is clear that the primary indication for ROP laser treatment is now the presence of plus disease. It is recommended that treatment be performed within 72 hours of determination of treatable disease, because delay in treatment may increase the risk of an unfavorable outcome. Therefore, failure to accurately detect plus disease may place an infant at increased risk for retinal detachment and blindness.

The finding in this study that the sensitivity of ROPtool was superior to that of individual examiners is important because high sensitivity is arguably the most desirable aspect of a diagnostic test for plus disease in ROP. High sensitivity means that there are very few false-negative results, or very few cases in which plus disease exists and the diagnostic test fails to detect it. Plus disease has become the primary criterion for laser treatment, so it is important to miss as few cases of plus disease as possible in order to reduce the risk of delayed treatment and retinal detachment. High specificity, although desirable, is not as critical as high sensitivity in diagnosing plus disease. It is true that false-positive results could lead to unnecessary laser treatment. However, if one assumes that false-positive results are most likely to occur in cases that are “almost plus disease,” many of these eyes will progress to plus disease and need treatment over the next few days or weeks. Although unnecessary treatment is not desirable and can result in reduced peripheral vision, the incidence of severe side effects from laser treatment is low when it is done by an experienced ophthalmologist. In the ETROP study, serious side effects of laser treatment, such as closure of the central retinal artery or inadvertent burns outside of the target zone, occurred in only 3 of 597 infants (0.5%).¹²

The relative value of sensitivity vs specificity has implications for choosing the appropriate numeric thresholds (cutoff points) for tortuosity sufficient for plus disease and for pre-plus disease. ROC curves are particularly valuable for visualizing this trade-off between sensitivity and specificity (Figures 8, 9, 11 and 12). When ROPtool was first developed, the threshold for tortuosity sufficient for plus disease was 10 tortuosity units, because it was the average amount of tortuosity present in the standard photograph of plus disease.⁵ After the pilot study, the threshold was adjusted to 9 tortuosity units and a pre-plus threshold of 7 tortuosity units was established. A threshold of 9 tortuosity units for tortuosity sufficient for plus disease was chosen because it had the best sensitivity without significantly compromising specificity. The results presented herein confirm that 9 tortuosity units is an appropriate threshold for tortuosity sufficient for plus disease. Although using a threshold of 10 or 11 tortuosity units would have improved the overall concordance, it would have done so at the expense of decreasing sensitivity to an unacceptable level. An important part of the continuing “training” of ROPtool will be to adjust these numeric thresholds as results are obtained from analyses of additional image sets.

At first glance, it is counterintuitive that the mean tortuosity of quadrants with pre-plus tortuosity was 9.9 tortuosity units, even though the numeric thresholds were 7 for pre-plus tortuosity and 9 for plus tortuosity. This occurred because the range of tortuosity measurements in the study was between 1.7 and 45.3 tortuosity units. Therefore, very low tortuosity measurements could be no more than 5.3 (7 minus 1.7) tortuosity units below the threshold for pre-plus tortuosity of 7 tortuosity units, whereas some very high tortuosity measurements were more than 15 tortuosity units above the threshold, skewing the mean value upward. In addition, 2 quadrants measuring 7 tortuosity units or greater were required for pre-plus tortuosity at the eye level. As a result, 1 quadrant with a very high tortuosity was insufficient for eye-level pre-plus tortuosity. For example, an eye with tortuosities in the 4 quadrants of 7.5, 8, 8.5, and 16 tortuosity units would have pre-plus tortuosity, because only 1 quadrant exceeds the plus tortuosity cutoff of 9 tortuosity units, even though the overall mean tortuosity of 10 tortuosity units exceeds this cutoff point.

ROPtool performed well in this study; however, the individual examiners also performed quite well, so the null hypothesis that the overall accuracy of ROPtool is no better than that of an examiner could not be rejected. Overall accuracy, or concordance, is greatly influenced by the prevalence of disease in the study population, whereas sensitivity and specificity depend much less on disease prevalence. This fact explains why sensitivity of ROPtool was superior to examiner’s sensitivity, but concordance was so similar. Sixty-two percent of the images in the study were judged by experts to be normal, 18% had pre-plus tortuosity, and only 20% had plus tortuosity. If the study were to be repeated using a sample enhanced with a greater proportion of images with plus disease, and if similar sensitivities and specificities were found, then the overall concordance of ROPtool would likely be better than that of individual examiners. In addition, the proportion of disagreements in a study such as this one depends heavily on the number of “borderline” images included in the sample. If one wanted to make a new diagnostic test appear to be very accurate, using a sample enhanced with normals and obvious abnormals would likely give the best results. This is one reason why it was important to include comparison of the accuracy of ROPtool with that of individual examiners. In this study, the sample consisted of as many borderline images as possible. However, it is difficult to obtain a very large number of high-quality RetCam photos from infants with ROP, especially with this study’s rigid criterion of using only one image from each eye.

For most calculations, the unit of analysis was a single image or eye (eye-level data). For these analyses, the overall grade of the eye was determined by the second most tortuous quadrant, because 2 quadrants of sufficient tortuosity were necessary in order for an eye to be given a grade of plus or pre-plus tortuosity. For example, if the grades of quadrants of an eye were 0, 0, 1, and 2, then the eye-level grade was 1 (pre-plus tortuosity), because 2 quadrants had a grade of 1 or more (and the second most tortuous quadrant had a grade of 1). Additional analyses were done using each quadrant as the unit of analysis, or 4 units from each image. This quadrant-level approach could be used because the experts also graded each quadrant individually. This method has the advantage of increasing the number of observations by a factor of 4. It is particularly useful for analysis techniques such as ROC curves when establishment of the best numeric threshold for plus disease or for pre-plus disease is limited by the relatively small number of individual observations with values between 2 consecutive cutoff points. One limitation to this approach is that there is correlation between tortuosity of vessels in the same eye or image, so quadrant-level observations are not completely independent of each other.

In addition to the primary analysis involving plus disease, this study looked at the ability of ROPtool to distinguish eye-level tortuosity sufficient for pre-plus disease from normal. ROPtool did not perform quite as well when assessing tortuosity sufficient for pre-plus disease, as there were 24 disagreements with expert consensus (compared to only 10 disagreements for plus tortuosity). There are several possible reasons for this observation. First, expert consensus for tortuosity sufficient for pre-plus was not defined as rigorously as for plus disease, because it was not the primary analysis. Consensus for pre-plus was based on majority opinion, and disagreements were not reconsidered by experts or adjudicated on a conference call. Second, there is no “standard photo” defining the minimum amount of tortuosity for pre-plus disease, so experts may not be well calibrated for diagnosing pre-plus disease. For the above reasons, expert consensus for tortuosity sufficient for pre-plus may not have been as accurate as that for plus disease. Third, a few images were slightly blurry or relatively magnified, which can lead to an abnormally high tortuosity measure by ROPtool as its vessel-tracing program follows a choppy course through large or slightly blurry blood vessels, leading to overestimation of tortuosity and false-positive assessments of tortuosity sufficient for pre-plus disease.

In addition to the advantages of quantifying plus disease, it is desirable to accurately measure pre-plus disease for several reasons. First, the presence of pre-plus disease could influence the timing of the next examination. Current ROP screening guidelines state that infants without pre-threshold disease can be examined at 2-week intervals.²⁷ Type 2 ROP is defined as any zone I, stage 1 or 2 without plus disease, or zone II, stage 3 without plus disease, and it is generally an indication for close follow-up.¹² However, there are some cases in which pre-plus is present without type 2 ROP, and in these cases, pre-plus may be the first sign of impending disease progression. If pre-plus disease is present but not noted by the examiner, then a routine follow-up examination of 2 weeks would likely be recommended. If the eye progresses rapidly, then an opportunity to treat at the optimal time could be missed, resulting in an increased risk of an unfavorable outcome. Second, ophthalmologists are often asked by neonatologists to predict the likelihood of progression to severe ROP before an infant is transferred to another hospital or discharged home. The presence of pre-plus could alert physicians and other caretakers that an infant should not be transferred to another hospital unless the receiving hospital staff is able to provide follow-up and treatment. Third, the presence or absence of pre-plus disease may aid in counseling parents of premature infants regarding the risk of disease progression. Frequently, parents are understandably anxious to know whether their infants’ eyes are “getting worse” and whether laser treatment will be necessary.

Even among experts, there is a tremendous amount of interobserver variability in the assessment of tortuosity sufficient for plus disease. When excluding normal images, there was disagreement among the 3 experts in this study for 26 (37%) of the 70 images. This observation agrees with the findings of Freedman and associates³⁶ that 3 CRYO-ROP certified experts disagreed on the presence or absence of plus disease in 29 of 72 cases (40%). Why does so much variability exist? Heretofore, there has been little attention paid to identifying the factors that influence an individual’s assessment of plus disease, so there are numerous unanswered questions. For example, is the assessment of tortuosity driven by the most tortuous vessel in each quadrant or by some composite of tortuosity of all of the major vessels in a quadrant? How much weight is given to extremely tortuous short vessel segments (“wiggles”) vs more gradual curves in longer vessel segments? How does the amount and degree of dilation influence the assessment of tortuosity? How does the total number of vessels influence this judgment? Interestingly, status as an ROP expert or as an experienced examiner of infants with ROP does not necessarily improve one’s ability to accurately assess tortuosity or to discriminate between various degrees of dilation and tortuosity. Freedman and associates³⁶ found in a separate study that naïve and expert subjects performed indistinguishably when ranking groups of computer-generated retinal vascular tracings in order of increasing mean diameter and tortuosity. The large amount of interobserver variability in the assessment of plus disease suggests that even ROP experts are frequently inaccurate in their assessment of plus disease at the bedside. Therefore, a computer-automated system such as ROPtool has the potential to improve diagnostic accuracy, particularly in cases of “borderline” plus disease.

Various methods have been used to measure the tortuosity of retinal blood vessels. The original version of ROPtool used the simplest approach, dividing the total length of the vessels by the straight-line distance between its starting and ending points. A major problem with this approach was that a nontortuous vessel that gradually curved in one direction was falsely measured as tortuous. In order to avoid this problem, ROPtool generates a smooth curve using several points along the vessel, and it compares the total length of the vessel to the length of this smooth curve (Figure 4). Capowski and associates³⁷ developed a novel numeric index based on spatial frequency, and they compared its accuracy in assessing tortuosity of retinal blood vessels to the standard ratio of vessel length to straight-line distance. Their numeric index was calculated by selecting all possible segments of a vessel with lengths that varied from 780 to 1200 micrometers. The tortuosity of each segment was determined using the standard ratio of vessel length to straight-line distance. The overall vessel tortuosity was then calculated as the average of the 5 highest values whose segments did not overlap. They found that this numeric index was a more sensitive indicator of tortuosity than the standard ratio of vessel length to straight-line distance. Gelman and associates³⁸ quantified tortuosity using a different method, which incorporated the concept of “integrated curvature.” They converted segmented vessels into a skeleton of 1-pixel-width trees, and they calculated integrated curvature as the sum of the angles along the “skeleton,” normalized by the length of the vessel.

Other investigators have reported early success with computer-assisted or computer-automated quantification of dilation and tortuosity of posterior retinal vessels in infants with ROP^36–40 (Capowski, ARVO Meeting, 1994, Abstract; Kylstra JA, ARVO Meeting, 1995, Abstract). Freedman, Kylstra, and Capowski^36,37 were the first to use computer technology to quantify vascular dilation and tortuosity in ROP. Heneghan and associates³⁹ utilized a general technique for segmenting out vascular structures in retinal images and characterizing the segmented blood vessels. Using images from 23 subjects, they reported a sensitivity of 82% and a specificity of 75% in the detection of plus disease. Swanson and associates⁴⁰ used Retinal Image multiScale Analysis (RISA), a semiautomated image analysis software package, to quantify the size and tortuosity of retinal blood vessels. They found that arteriolar tortuosity was associated with ROP severity, but no significant association was observed between ROP severity and venular and arteriolar diameters. Gelman and associates³⁸ applied an enhanced version of RISA that utilized their concept of integrated curvature to 32 eyes of 16 infants. They found that eyes with plus disease had vessels with significantly higher integrated curvature, diameter, and tortuosity than those without plus disease. The best diagnostic accuracy was observed with integrated curvature, as curvature had an area under the ROC curve that was greater than both dilation and tortuosity for both arterioles and venules.

In this study, the ability of ROPtool to measure tortuosity sufficient for plus disease was assessed, even though the diagnosis of plus disease depends on a minimum amount of tortuosity and dilation. The dilation measurement used by ROPtool is still in development, and dilation is more difficult than tortuosity to accurately quantify. This is primarily because the degree of change from a normal to an abnormally dilated vessel is relatively small, as opposed to the relatively large change from a straight to a tortuous vessel. In addition, detection of a small change in vessel width is limited by the number of pixels in a digital image. Even if these challenges are overcome and an automated measure of dilation is perfected, it is possible that it will add very little to the overall assessment of plus disease. Kylstra and associates noted that computer-assisted determination of plus disease based on a minimum degree of tortuosity alone showed both excellent sensitivity (85%) and specificity (91%) when compared to evaluation by an expert (Kylstra JA, ARVO Meeting, 1995. Abstract). Their computer-assisted model differed from ROPtool in that they used the computer’s mouse to mark a series of points along each vessel, and they calculated vessel tortuosity as the length of the vessel divided by the straight-line distance from its origin to its termination. Another reason why dilation may not add much to the assessment of plus disease is that abnormal dilation probably precedes tortuosity in most cases; therefore, it is the development of a threshold level of tortuosity that usually triggers the diagnosis of plus disease.^15,25 Once tortuosity is known, inclusion of dilation will change the overall assessment of plus disease only in cases when the vessels are tortuous enough for plus disease but not dilated enough for plus disease. In the author’s experience, this scenario is relatively uncommon.

Plus disease was defined by ICROP as being present when the vascular changes are so marked that the posterior veins are enlarged and the arterioles tortuous. In this study, ROPtool was used to measure the tortuosity of both arterioles and venules for several reasons. First, it is not always possible to reliably differentiate between arterioles and venules in an image of a premature infant’s retina. Second, as plus disease progresses from mild to severe, both the arterioles and the venules become dilated and tortuous. In the standard photograph of plus disease (Figure 2) that was first used to calibrate ROPtool, both the arterioles and the venules are tortuous. Third, both arteriolar and venular tortuosities were quantified in the pilot study of ROPtool, and these data were used to determine cutoff points for tortuosity sufficient for plus disease and for pre-plus disease for this study. Even when both arterioles and venules are tortuous, the arterioles are usually more tortuous than the venules, so using arterioles only would increase the tortuosity of each quadrant and would change these cutoff points. Fourth, it is not yet understood precisely which vessels are used by examiners when determining if plus disease is present or not, but it seems likely that both dilation and tortuosity of all vessels are considered. In future studies, it would be interesting to measure examiners’ abilities to differentiate arterioles from venules. If these blood vessels can be differentiated with reasonable accuracy, then new cutoff points could be calculated and the accuracy of ROPtool in determining tortuosity sufficient for plus disease when considering arterioles only could be determined.

This study must be viewed in light of some limitations. First, there is no ideal method for determining the “truth” or reference standard with regard to the presence or absence of plus tortuosity. This aspect of the study was designed as rigorously as possible by including an initial assessment by experts, reconsideration by the expert outlier when disagreement existed, and a conference call to reach consensus when disagreements persisted. However, it remains possible that in cases where ROPtool or the individual examiners were considered to be incorrect, they were actually correct and expert consensus was wrong. A second limitation of this study is that an accurate assessment of tortuosity does not necessarily translate into an accurate assessment of the presence or absence of plus disease. More work needs to be done to develop the dilation measurement of ROPtool or to show that dilation is unimportant or less meaningful than tortuosity in the overall assessment of plus disease. Third, it would have been preferable to include a larger number of “borderline” images that were not obviously plus disease or normal. This would have made it more difficult for ROPtool and for the examiners, possibly amplifying differences between them. Fourth, a few images were difficult for the graders and/or ROPtool to assess owing to decentering or slight blurring of the image. Although this may have affected the results, it may make them more generalizable, since any future clinical application of ROPtool will likely include excellent as well as suboptimal images.

In addition to improving the assessment of dilation, much work remains to improve and refine ROPtool before it can augment or replace human judgment of plus disease. Analysis of a large number of images with different vessel shapes and patterns will allow the program to “learn” and will allow adjustment of the appropriate numeric thresholds for tortuosity sufficient for plus disease and pre-plus disease. Other techniques for measuring dilation or tortuosity, such as the “spatial frequency” method of Capowski and associates³⁷ or the “integrated curvature” measurement of Gelman and associates,³⁸ may be shown to improve accuracy, and if so, they could be incorporated into future versions of ROPtool. Also, ROPtool can be used to analyze high-quality RetCam images, but it is unknown how it will perform with other types of images that may be easier and less expensive to capture, such as those from a Nidek camera or still images captured during video indirect ophthalmoscopy.

How might computer-assisted or computer-automated analysis be used in the management of infants with ROP in the future? Its greatest potential impact in the near future is probably in the areas of telemedicine and research. It has become increasingly difficult to provide ROP screening for infants at risk because of the lack of availability of qualified examiners, particularly in developing countries. One possible solution is telemedicine screening, utilizing the internet to transmit images to ROP reading centers. Several studies have established the feasibility of remote screening for ROP.^26,41–45 Analysis of transmitted images could include a computer-assisted calculation of its tortuosity with or without dilation. One approach could be to use expert opinion and ROPtool, 2 diagnostic tests with different degrees of sensitivity and specificity, as either simultaneous or sequential tests. The precise degree of tortuosity with or without dilation could aid in decisions to treat with laser or to transfer to a center with an ophthalmologist for closer follow-up. Even if plus disease is not present, pre-plus vascular changes may be associated with the eventual need for laser treatment,²⁵ and the ability to quantify the precise amount of vascular abnormality could influence follow-up intervals. For example, assuming a threshold for tortuosity sufficient for plus disease of 9 tortuosity units, an examiner would likely follow very closely an infant whose tortuosity increases from 5 to 7 tortuosity units in 1 week. Computer-assisted quantification of plus disease also offers potential benefits to research in ROP. Greater accuracy in measuring the components of plus disease is likely to lead to better understanding of its evolution. This knowledge may facilitate studies of causes of plus disease and the effect of new or existing treatments for ROP. One can envision a clinical trial in which an image of an infant with severe ROP is transmitted to a reading center where experts confirm that treatment is needed, assisted by computerized assessment of plus disease. Finally, computer-assisted analysis of plus disease may eventually have a role at the bedside. Still images captured from video indirect ophthalmoscopy could potentially be analyzed in real time, and a measurement of posterior retinal vascular tortuosity with or without dilation could be included in the record of every examination.

In conclusion, ROPtool is a promising technology that has the potential to remove subjectivity in the assessment of tortuosity sufficient for plus disease, thereby increasing accuracy and optimizing the timing of laser treatment. Its greatest potential application in the near future is in research and telemedicine. Results from this study demonstrate that ROPtool can accurately discriminate between tortuosity sufficient or insufficient for plus disease, but more work remains before it will augment or replace the human observer.

Funding/Support: Supported by grant K23 EY015806 from the National Eye Institute.

Financial Disclosures: None.

Other Acknowledgments: I thank Michael Chiang, MD, of Columbia University, New York, NY; David Coats, MD, of Baylor College of Medicine, Houston, Texas; Graham Quinn, MD MSCE, of Children’s Hospital of Philadelphia, Pennsylvania; and Laura Enyedi, MD, Sharon Freedman, MD, and Terri Young, MD, of Duke University, Durham, North Carolina, for serving as experts and examiners. I thank Zheen Zhao, PhD, of Duke University, and Julien Jomier, PhD, and Stephen Aylward, PhD, of the University of North Carolina, Chapel Hill, for ROPtool programming. I thank Sin-Ho Jung, PhD, and Sandra Stinnett, DrPH, of Duke University for statistical assistance, and Sarah Jones of Duke University for assistance with data analysis. I thank Michael Chiang, MD, Anna Ells, MD, of Alberta Children’s Hospital, Calgary, Alberta, Canada; Antonio Capone, MD, of William Beaumont Hospital, Royal Oak, Michigan; and the PHOTO-ROP study group for sharing RetCam photographs. I thank David Coats and Dan Jones of Baylor College of Medicine, Houston, Texas, for reviewing this thesis and providing helpful suggestions.