Appendix VIII

Criteria for Assessing External Validity (Generalizability) of Individual Studies

Each study that is identified as one that provides evidence to answer a KQ is assessed by according to its external validity (generalizability) using the following criteria.

Criteria:

Study Population:
The degree to which the people who were involved as subjects in the study constitute a special population because they were selected from a larger eligible population or were for other reasons unrepresentative of people who are likely to seek or be candidates for the preventive service. The selection has the potential to affect the following:

• Absolute risk: The background rate of outcomes in the study could be greater or less than what might be expected in asymptomatic people because of the inclusion/exclusion criteria, because of non-participation, or for other reasons.
• Harms: The harms observed in the study could be greater or less than what might be expected in asymptomatic people.

The following are features of the study population and the study design that may cause experience in the study to be different from what would be observed in the US primary care population:

• Demographics (age, gender, ethnicity, education, income): The criteria for inclusion/exclusion or non-participation do not encompass the range of people likely to be candidates for the preventive services in the US primary care population.
• Co-morbidities: the frequency of co-morbid conditions in the study population does not represent of the frequency likely to be encountered in people who seek the preventive service in the U.S. primary care population.
• Special inclusion/exclusion criteria: There are other special inclusion/exclusion criteria that make the study population unrepresentative.
• Refusal rate (ratio of included to not-included but eligible participants): The refusal rate among eligible study subjects is high, making the enrollees in the study unrepresentative even of the people eligible for the study.
• Adherence (run-in phase, frequent contact to monitor adherence): The design of the study has features that may make the effect of the intervention in the study greater than it would be in a clinically observed population.
• Stage in natural history of disease; severity of disease: the selection of subjects for the study includes people with at a stage that is earlier or later than would be found in people who are candidates for the preventive service.
• Source, intensity of recruitment: The sources for recruiting subjects for the study and/or the effort and intensity of recruitment may distort the characteristics of the study subjects in ways that could increase the effect of the intervention as it is observed in the study.

Situation:
The degree to which the clinical experience in the situation in which the study was conducted is likely to be reproduced in other settings

• Healthcare system: The clinical experience in the system in which the study was conducted is not likely to be the same as experience in other systems because, for example, the system provides essential services for free when these services are only available at a high cost in other systems.
• Country: The clinical experience in the country in which the study was conducted is not likely to be the same as in the U.S. because, for example, services available in the U.S. are not widely available in the other country of study conduct or vice versa.
• Selection of participating centers: The clinical experience in which the study was conducted is not likely to be same as in offices/hospitals/settings in which the service will be delivered to the U.S. primary care population because, for example, the centers have ancillary services not available generally.
• Time, effort, and system cost for the intervention: The time, effort, and cost to develop the service in the study is more than would be available outside the study setting.

Providers:
The degree to which the providers in the study have the skills and expertise likely to be available in general settings

• Training to implement the intervention: The intervention in the study was done after giving providers special training not likely to be available or required in U.S. primary care settings.
• Expertise, skill to implement intervention: The providers included in the study had expertise and/or skills at a level that is higher than the level likely to be encountered in typical settings.
• Ancillary providers: The study intervention relied on ancillary providers who are not likely to be available in typical settings.

Global Rating of External Validity (Generalizability):

External validity is rated "good" if:

• The study differs minimally from the US primary care population/situation/providers and only in ways that are unlikely to affect the outcome; it is highly probable (>90%) that the clinical experience with the intervention observed in the study will be attained in the US primary care setting.

External validity is rated "fair" if:

• The study differs from the US primary care population/situation/providers in a few ways that have the potential to affect the outcome in a clinically important way; it is only moderately probable (50%-89%) that the clinical experience with the intervention in the study will be attained in the US primary care setting.

External validity is rated "poor" if:

• The study differs from the US primary care population/ situation/ providers in many way that have a high likelihood of affecting the clinical outcomes; the probability is low (<50%) that the clinical experience with the intervention observed in the study will be attained in the US primary care setting.

Return to Contents
Proceed to Next Section