 |
Meetings & Workshops |
U.S. Food and Drug Administration • Center for Drug Evaluation and Research |
|
Meetings & Workshops |
 |
|
|
Evaluation of Written Prescription Information Provided in Community Pharmacies: An 8-State Study Bonnie L. Svarstad, Ph.D.1 and Dara C. Bultman, Ph.D., R.Ph.2 University of Wisconsin - Madison 1William S. Apple Distinguished Professor of Social Pharmacy, University of Wisconsin-Madison, School of Pharmacy, 425 North Charter St., Madison, WI 53706; Tel: 608-265-2128; Email: blsvarstad@pharmacy.wisc.edu 2Clinical Instructor, University of Wisconsin-Madison, School of Pharmacy, Madison, WI 53706; Tel: 608-262-3312; Email: dcbultman@pharmacy.wisc.edu Acknowledgments: We gratefully acknowledge the many helpful suggestions and support of the expert panelists Heidi Anderson-Harper, Robert Beardsley, Chester A. Bond, Marie Gardner, Carole Kimberlin, Duane Kirking, Sharlea Leatherwood, Helene Lipton, and Betsy Sleath. We also appreciate the support of the National Association of Boards of Pharmacy, state coordinators, and Ellen Tabak and her colleagues at the U.S. Food and Drug Administration. INTRODUCTION This report describes the methods and findings of a study to assess the quality of written prescription information provided voluntarily to individuals receiving new prescription medicines in community pharmacies. The study was requested and funded by the U.S. Food and Drug Administration (FDA) and conducted in collaboration with the National Association of Boards of Pharmacy (NABP) and a national panel of experts in drug information and communications (Appendix A). This study differs from previous studies in several ways. First, it involved the collection of written prescription information materials by state inspectors and other trained observers who acted as patients (hereafter referred to as patient-observers). All patient-observers were instructed to use a standard scenario and script when purchasing three new prescriptions for ibuprofen, amoxicillin, and paroxetine. Second, patient-observers visited selected community pharmacies in eight states representing four geographic areas of the United States. Random sampling procedures were used to select the pharmacies within designated areas. Third, expert panelists evaluated the quality of written prescription information given to the patient-observers using a Patient Information Evaluation Form (PIEF) for each study drug. These forms were developed from various criteria for evaluating the quality of written drug information, as defined by the Steering Committee for the Collaborative Development of a Long-Range Action Plan for the Provision of Useful Prescription Medicine Information (1996). Further details about study methods and criteria for evaluating written information are provided in the methods sectionbelow. The specific research questions addressed in this report were the following: · What percentage of patient-observers were given any written prescription information with the study drugs (in addition to the labels and stickers on their medication containers)? · What were the panelists' overall evaluations of written information given with the study drugs? · What were the panelists' evaluations of written information on the 10 general criteria listed in the Patient Information Evaluation Forms? · What were the panelists' evaluations of written information on the sub-criteria listed in the Patient Information Evaluation Forms? METHODS Sampling procedures NABP selected 10 states for possible participation in the study based on previous cooperative work relations. Directors of state boards of pharmacy were contacted by NABP in the fall 1998. NABP held a conference call with state directors Nov. 3 to share general information about theproject. At this time, the ten states were Arizona, California, Florida, Illinois, Minnesota, North Carolina, New York, Ohio, Texas and Washington. A second conference call was held April 8 with key participants of organizations including NABP, University of Wisconsin School of Pharmacy researchers, FDA and State Board of Pharmacy directors. Representatives from Illinois and Ohio were not available. Several problems were noted during the spring conference. State board directors were concerned about their ability to identify physicians, create patient identities and do random sampling of pharmacies. In addition, some state directors mentioned concern about having enough inspectors to act as patient-observers. Finally, it was suggested that Ohio be replaced, since recent media coverage of pharmacy practice might result in misleading data. In response to state director concerns about potential difficulties in carrying out the project, changes were made in sampling procedures (described below,) and assistance was offered in the identification of additional patient-observers as needed. Wisconsin was selected to replace Ohio, and California and Florida decided not to participate in the project. The eight remaining states were Arizona, Illinois, Minnesota, North Carolina, New York, Texas, Washington and Wisconsin. The sample, therefore, originates from four U.S. geographic regions: east (NY, NC), midwest (IL, MN, WI), south west (TX, AZ) and northwest (WA). Procedures for sampling pharmacies varied by state. Illinois and Washington sampled pharmacies state-wide. The remaining 6 states sampled pharmacies within certain metropolitan or geographic regions within the state. The number of metropolitan or geographic areas varied from 1 to 7 depending on state size and availability of inspectors. Wisconsin sampled from one metropolitan area; Minnesota sampled from 3 areas; North Carolina sampled from 4 areas; Arizona and New York sampled from 5 areas in each state; and Texas sampled from 7 areas. Pharmacies were randomly selected within the selected areas. The pharmacy samples in Illinois and Washington were generated from lists of all licensed pharmacies within the state (provided by each state board of pharmacy). In the remaining 6 states the sampling list of pharmacies was created using zip codes. A list of licensed pharmacies with zip codes of the designated metropolitan or geographic areas was created from state board of pharmacy records. Then a random sample of 40 pharmacies plus 20 replacements was generated for each state except Wisconsin where researchers generated a random sample of 30 pharmacies plus 20 replacements. Random samples were generated using a computerized random number generator for sample selection when lists were available on disk. Pharmacy samples for Minnesota and North Carolina were generated manually using a table of random numbers and a printed list of all licensed pharmacies in the state. Observer Protocol Data were collected by trained observers acting as patients. These "patient-observers" were designated by state board of pharmacy directors in all states except Wisconsin where patient-observers were designated by researchers as part of the pilot study. The majority of patient-observers were state inspectors. If inspectors were unavailable, observations were conducted by selected pharmacy students and other temporary employees who were not known at the pharmacies to be visited. Approximately 71% of pharmacy visits were made by male observers and 29% by female observers. Observer age ranged from 21 to 85 years old. Observers were instructed to use a standard scenario as the script when acting as patient-observers. The standard scenario defined the patient-observer's name, address, reason for being in the area, health and medication history and other information that might be needed while acting as patient-observer. Instructions were written out and provided to state coordinators for distribution to observers. The standard patient - observer scenario follows: You are to assume the following scenario. You are from (assigned by state coordinator-city, state or use your own home address or a friend or relative in the area). You are visiting or driving through (pharmacy site area). You have received three prescriptions from your doctor and forgot to fill them before leaving. You are a healthy person and have no other medical conditions. You take no other medications and have no drug allergies. You have not taken these medications before. You have Blue Cross/ Blue Shield insurance and submit your own receipts for reimbursement and therefore pay with cash. The reason you now have three prescriptions is that you have had a head cold for acouple of weeks that you just haven't been able to shake (like you usually do). In addition, your knees have been bothering you, and you are not sleeping through the night, can not concentrate and have been feeling down. When you saw your doctor the medications were prescribed for a sinus infection, tendinitis of the knee, and depression. The doctor didn't give you other information but said the directions would be on the prescription bottles. You have a follow-up appointment with your doctor in a week. You will use an assigned name as Patient-Observer. The assigned name will be on the three prescriptions. You can use your birth date, address and telephone number. In the situation where you must visit a second pharmacy of a chain or other corporately owned pharmacy, use a different last name, address and telephone number. Observers were trained to make their approach and presentation in each pharmacy as uniform as possible. They were told to be polite, to act interested, to answer questions briefly and to accept any written information offered. In addition, observers were trained to avoid volunteering any information unless asked, to not ask questions, to not initiate "small talk" and to politely leave the pharmacy if the legitimacy of the prescriptions was questioned. If observers recognized or were recognized by any pharmacy personnel, they were instructed to exit the pharmacy prior to presenting the prescriptions. In addition, it was suggested that observers role-play prior to data collection. Role-play would allow each observer to practice his or her script with the state coordinator acting as thepharmacist. Each observer would practice presenting prescriptions and answering commonly asked questions using the standard scenario in order to practice their "patient" role. See Appendix B for questions commonly asked in the pharmacy setting and suggested patient-observer responses. Observers were provided with study prescriptions, money to purchase prescriptions, and a list of assigned pharmacies. Observers traveled to pharmacies, presented study prescriptions to be filled, purchased the prescriptions, answered any questions according to the standard scenario, collected any drug information materials given, and exited the pharmacy. State coordinators collected all drug information materials from observers and forwarded them to the University of Wisconsin-Madison for data analysis. Wisconsin data were collected during February 1999 and data for the remaining states were collected from May to November 1999. Pilot Study A pilot study was conducted to determine whether the study procedures were adequate and allowed for valid and reliable data collection. The University of Wisconsin Health Sciences Human Subjects Committee reviewed and approved the pilot study protocol. A metropolitan area in Wisconsin was chosen for the pharmacy visits because it offered racial and economic diversity. A list of all licensed pharmacies with zip codes of the study area was obtained from the Pharmacy Society of Wisconsin (n=197). A random sample of 30 pharmacies plus 20replacements was selected from the list. Pharmacy managers or owners were contacted by telephone, asked if the pharmacy filled prescriptions for cash paying customers, informed that the project involved two unannounced visits to the pharmacy by simulated patients and a short telephone interview, and then asked for study participation. No details about the study drugs or scenarios were provided. Calls were made until 30 pharmacists agreed to participate. Five pharmacy organizations declined study participation. Four observers were trained according to pilot study protocol. Each observer visited 15 pharmacies so that the 30 pharmacies were each visited by 2 observers (on different days). Double visits were conducted to determine reliability of receipt of written information with the study prescriptions. Observers presented 2 different prescriptions at each pharmacy to reduce the likelihood of the second observer being recognized or suspected. Pharmacists were telephoned by research staff after all pharmacy visits were completed to determine if observers were recognized or their prescriptions questioned. The pilot study was helpful in several respects. First, we learned that the great majority of observed pharmacists did not question the legitimacy of the study prescriptions or patient-observer. Pharmacists later reported to researchers that they may have recognized a patient-observer in only four of 60 visits due to an out-of-town physician or address. We therefore concluded that the standard scenario was sufficient and that pharmacists were following their usual patterns of behavior when distributing written information. Second, the frequency of written information transmission was remarkably similar across the double pharmacy visits,suggesting that one visit was sufficient to assess performance in this area. Third, commonly asked questions from pharmacists were learned and inserted into the final protocol for other states. Fourth, the pilot study was helpful in determining the time required by observers to complete pharmacy visits. Finally, we learned that patient-observers would need additional scenario identities when visiting chain and other corporately owned pharmacies with interconnected computer systems. Because the pilot study did not identify any major flaws in study design or procedures, data from the Wisconsin pilot study were used in the final 8-state analysis. Patient Information Evaluation Forms Patient Information Evaluation Forms were reviewed and approved by a national expert panel of nine individuals. The panelists are experienced behavioral scientists and clinical pharmacists specializing in the areas of professional-patient communication, patient information, and drug therapies (see Appendix A for a list of panelists). Initially, the Wisconsin researchers prepared a draft version of the Patient Information Evaluation Form for each study drug (amoxicillin, ibuprofen, paroxetine). Each form listed ten general criteria for evaluating the quality of written drug information, as suggested in the 1996 Action Plan. The ten general criteria included: · scientifically accurate · unbiased in content and tone · identifies drug and its benefits · identifies contraindications and what to do if applicable · includes specific directions about how to take medication, receive maximum benefit, and interpret benefits · includes specific precautions while using medicine, their significance and how to avoid harm · includes enough detail for proper monitoring, interpretation, and action regarding adverse reactions that are serious or occur frequently · includes proper storage instructions and general information · information is legible and readily comprehensible to most consumers · information is up-to-date and timely To enhance reliability, we also developed a check list of 2-5 sub-criteria or operationalizations for each general criterion. Raters were asked to check whether there was partial or full adherence with each sub-criterion and then to indicate how well the general criterion was met on a scale of 1-9 (9=high adherence). The number of sub-criteria was 29 for ibuprofen, 28 for amoxicillin, and 32 for paroxetine. In April 1999 panelists reviewed the 1996 Action Plan for the Provision of Useful Prescription Medicine Information focusing on Chapter 3 "Guidelines for Useful Prescription Medicine Information" and Appendix G "Specific Language and Format Guidelines, with Samples". Inaddition, panelists received FDA labeling information for the three study drugs, product information from the USP-DI, and draft versions of the Patient Information Evaluation Forms for each study drug. Drafts also were submitted to the contractor and National Association of Boards of Pharmacy (NABP). The panelists and contractor were asked to review the Patient Information Evaluation Forms for consistency with criteria outlined in the 1996 Action Plan and to contribute other suggestions for improving the clarity and reliability of the forms. Their comments were then incorporated into revised drafts of the evaluation forms. This review and feedback process was repeated until all panelists independently approved the Patient Information Evaluation Forms. Copies of the final forms are provided in Appendix C. Analysis of Inter-rater Reliability A test of inter-rater reliability was conducted using the final Patient Information Evaluation Forms and written prescription information sheets obtained by a sub-set of patient-observers in four of the participating states. The nine panelists were randomly assigned into three subgroups with each subgroup including one clinical specialist/practitioner and two drug information specialists. Each panelist received four pieces of written information pertaining to his or her study drug and drafts of the evaluation forms. Panelists independently reviewed and evaluated the four pieces of information using the evaluation forms. Pearson correlation coefficients were used to determine inter-rater reliability for each of the 10 general criteria (score 1-9) and the sum of ratings for the 10 criteria (range 10-90). Pearson rwas 0.95 for the sum of ratings and ranged from 0.49 (Criterion 8) to 0.91 (Criterion 2) for the ten individual criteria. We were very pleased with these results and decided to proceed with the final ratings after providing a few additional suggestions to panelists in an effort to further enhance inter-rater reliability. Data Processing and Analysis After patient-observers completed their visits, all written information materials were sent to the University of Wisconsin-Madison for data processing and analysis. A total of 799 items were received (265 ibuprofen items, 268 amoxicillin items, 266 paroxetine items). Researchers ordered all items by pharmacy identification number and sorted them into folders for the panelist sub-groups described earlier (ibuprofen, amoxicillin, paroxetine). Researchers then divided the items in each sub-group folder and assigned approximately one-third to each panelist based on identification numbers. Staff also removed identifying information before mailing copies to the assigned panelist for his or her independent evaluation using the Patient Information Evaluation Forms (Appendix C). This means that each panel sub-group rated an average of 266 items and that each individual panelist rated an average of 88.8 items. This approach to evaluation was used to maintain good reliability, to reduce study costs, and to complete the evaluation in a timely manner. The panelists' ratings were returned and submitted to the University of Wisconsin Survey Laboratory for data entry. Two methods were used to summarize the panelists' ratings. Thesemethods are referred to as the "9-point method" and the "1-point method". The 9-point method involved rating how well the written information adhered to each of the 10 general criteria on a nine point scale, with higher scores indicating better adherence. We later defined three levels of adherence using this 9-point method: low adherence (ratings 1-3), moderate adherence (ratings 4-6), and high adherence (ratings 7-9). The 1-point method involved assigning one point for each sub-criterion with full or partial adherence and then calculating the percentage of possible points with full or partial adherence. We later defined five levels of information quality using this 1-point method. They included: · Level 0 (no written information given) · Level 1 (written information obtained <20% of possible points) · Level 2 (written information obtained 20-39% of possible points) · Level 3 (written information obtained 40-59% of possible points) · Level 4 (written information obtained 60-79% of possible points) · Level 5 (written information obtained >80% of possible points) RESULTS Frequency of any written information transmission Patient-observers presented new prescriptions in 306 different pharmacies located in eight states. The great majority of these prescriptions were dispensed with some written prescription information in addition to the label and stickers on the medication container. The percent of patient-observers who were given any written information was 86.6% for ibuprofen, 87.6% for amoxicillin, and 86.9% for paroxetine (Table 1). The overwhelming majority of written information items were given in the form of a 1-page information sheet generated by computer software. Panelists' ratings of written information using the 1-point method by drug Table 2 presents panelists' ratings of the quality of written information using the 1-point method of assessment by drug. In the first column, we see the distribution of ratings for ibuprofen information. Approximately 44% of the patient-observers received information items that met the Level 4 threshold and only 32.7% received information that met the Level 5 threshold. The remaining patient-observers either received no information (13.4%) or information that fell below the Level 4 threshold (10.1%). The panelists' ratings of written information for amoxicillin were somewhat higher than their ratings of information for ibuprofen, as shown in the second column. The distribution of ratings using the 1-point method indicates that 9.9% of the patient-observers received amoxicillin information that met the Level 4 threshold and 66% received information that met the Level 5 threshold. The remaining patient-observers either received no written information (14.4%) orinformation that fell below the Level 4 cut-off (9.5%). The ratings of written information for paroxetine were similar to the ratings for amoxicillin information, as shown in the third column. Approximately 10.1% of the patient-observers received information that met the Level 4 threshold and 65.7% received information that met the Level 5 threshold. The remaining patient-observers either received no information (13.1%) or information that fell below the Level 4 cut-off (11.1%). Panelists' ratings of written information using the 9-point method by drug Table 3 shows panelists' ratings of written information using the 9-point method of assessment by drug. First, it is clear that the quality of written information varied considerably across the 10 general criteria, as reflected in the distribution of ratings (columns 1-3) and mean ratings (column 4). For example, only 12-31% of the information sheets received high adherence ratings on Criterion 3 (identifies contraindications and what to do) while 88-94% of the information sheets received high adherence ratings on Criterion 7 (is unbiased in content and tone). Results show that panelists' mean ratings generally exceeded 8.0 points on Criterion 7 (is unbiased in content and tone) and Criterion 9 (is scientifically accurate and includes disclaimer). In contrast, the mean ratings generally fell below 7.0 points on Criterion 3 (identifiescontraindications and what to do), Criterion 4 (includes specific precautions, their significance, and how to avoid), Criterion 6 (includes storage instructions and general information), and Criterion 10 (is up-to-date and includes publication information). The mean ratings for Criterion 1 (identifies drug and its benefits), Criterion 2 (includes specific directions about how to take and achieve maximum benefits), Criterion 5 (includes enough detail for monitoring and interpretation of adverse reactions), and Criterion 8 (is legible and readily comprehensible) generally fell in between these extremes. Second, quality of written information varied by the drug in several areas. For example, the distribution of ratings showed considerable variability by drug on Criterion 2 (directions) and Criterion 4 (precautions), with ibuprofen items receiving lower adherence ratings than other drugs. Panelists gave a high adherence rating Criterion 2 for 77.6% of amoxicillin items, 58.6% of paroxetine items, and only 33.2% of ibuprofen items. Findings also show a high adherence rating on Criterion 4 for 73.5% of amoxicillin items, 15.8% of paroxetine items, and 5.3% of ibuprofen items. Panelists' ratings of written information by sub-criteria Tables 4-6 show the percentage of information sheets with full or partial adherence on each drug-specific sub-criterion, suggesting areas for improvement. For example, ratings on Criterion 1 can be improved by including information about drug class (ibuprofen and amoxicilllin). Ratings on Criterion 2 can be improved by including information about how to achieve and interpret benefits (ibuprofen, amoxicillin, paroxetine). Ratings on Criterion 3 can be improved by including information about certain allergies, drug interactions, or medical problems that may affect use of the drug and what, if anything, should be done during pregnancy or nursing (ibuprofen, amoxicillin, paroxetine). Ratings on Criterion 4 can be improved by including specific precautions about long term use without proper monitoring (ibuprofen), how use may alter the effects of oral contraceptives and laboratory tests (amoxicillin), and overdose information (paroxetine). Ratings on Criterion 6 and 10 can be improved by informing patients about the availability of additional information from providers, the availability of non-child resistant caps, and the publisher and date of publication. DISCUSSION/CONCLUSIONS This study produced several encouraging results. First, we found that nearly 87% of new prescriptions were dispensed with some written prescription information in addition to the label and stickers on the medication container. This suggests that the provision of written prescription information is becoming a routine practice in community pharmacies. Second, expert panelists found that most written information sheets were accurate and unbiased in content and tone and that nearly two-thirds of the information sheets for amoxicillin and paroxetine met the Level 5 threshold for information quality. The majority of information sheets also identified the drug and its benefits, included necessary details for monitoring and interpreting adverse reactions, andappeared legible and comprehensible to consumers. The study raised several concerns. One concern is that the mean panelist ratings generally fell below 7.0 points on Criterion 3 (identifies contraindications and what to do), Criterion 4 (includes specific precautions, their significance, and how to avoid), Criterion 6 (includes storage instructions and general information), and Criterion 10 (is up-to-date and includes publication information). The majority of information sheets for ibuprofen also fell below the Level 5 threshold. As a result, we conclude that the quality of written prescription information provided in the eight states was variable and that there are many areas for improvement. It is important to note several study limitations and issues for further discussion. First, we gave equal weight to each criterion and sub-criterion. However, panelists raised the possibility that certain types of information should be given greater weight than other types of information in future studies. This is a complex issue that requires further discussion and consideration in future studies of this kind. Second, data were collected in eight states that volunteered to help with the project. We must be cautious in generalizing to other states having different practice regulations or conditions that may influence the rate and quality of medication counseling and use of written prescription information. Third, we were not able to apply uniform sampling procedures within the participating states or provide intensive training and supervision of the patient-observers due to limited resources and staff. Finally, the study showed that it is possible to achieve excellent inter-rater reliability using anexpert panel and explicit criteria for evaluating the quality of written prescription information. However, there was some discussion among panelists about the need for consumer input and more detailed examination of Criterion 8 (is legible and readily comprehensible to most consumers). In the future, we suggest adding several consumers to the expert panel or appointing a separate consumer panel to provide additional input with regard to legibility, comprehensibility, and other issues that require further discussion. REFERENCE Steering Committee for the Collaborative Development of a Long-Range Action Plan for the Provision of Useful Prescription Medicine Information, Action Plan for the Provision of Useful Prescription Medicine Information, Unpublished report submitted to The Honorable Donne E. Shalala, Secretary of the U. S. Department of Health and Human Services, December 1996. FDA/Center for Drug Evaluation and Research |
