UNITED STATES OF AMERICA

FOOD AND DRUG ADMINISTRATION

CENTER FOR DRUG EVALUATION AND RESEARCH

* * *

DRUG SAFETY AND RISK MANAGEMENT ADVISORY COMMITTEE

* * *

MEETING

* * *

WEDNESDAY,

JULY 17, 2002

* * *

The Advisory Committee met at 8:00 a.m in the Ballroom of the Holiday Inn Gaithersburg, Two Montgomery Village Avenue, Gaithersburg, Maryland, Dr. Peter A. Gross, Chairman, presiding.

PRESENT:

JACQUELINE S. GARDNER, Ph.D., M.P.H., Member

PRESENT (Continued):

FDA REPRESENTATIVES:

PAUL SELIGMAN, M.D.

ANNE TRONTELL, M.D., M.P.H.

PUBLIC SPEAKERS:

WM. RAY BULLMAN, M.A.M.

GERALD K. McEVOY, Pharm.D.

THOMAS MENIGHAN

TISH PAHL, J.D.

NICHOLAS RATTO, Pharm.D.

LARRY D. SASICH, Pharm.D., M.P.H

DONNA STOREY, Ph.D.

C O N T E N T S

PAGE

Conflict of Interest Statement 4

Welcome and Charge to the Committee, Dr. Paul

Seligman 8

History of Patient Information Efforts, Paul

McGinnis 12

Information Development and Flow to Consumer:

Dr. John Coster 22

Sharlea Leatherwood 43

Report of Evaluation of Written Patient

Information Penetration and Usefulness,

Dr. Bonnie Svarstad

Part 1 65

Part 2 102

Open Public Comment:

Dr. Nicholas Ratto 156

Dr. Gerald McEvoy 159

Dr. Donna Storey 166

Thomas Menighan 172

William Ray Bullman 177

Dr. Larry Sasich 183

Tish Pahl 192

Consumer Comprehension of Educational Materials:

Key Cognitive Principles, Ruth S. Day.

Ph.D. 196

Committee Consideration of Questions 227

P R O C E E D I N G S

(8:11 a.m.)

CHAIRMAN GROSS: Good morning. I'd like to call the meeting to order if everyone would please have a seat.

I'm Dr. Peter Gross. I'm Chair of the Drug Safety and Risk Management Advisory Committee.

And today we're going to address the issue of the components of the consumer medication information sheets. This is a discussion about it, not a regulatory affair.

So I'd like to turn the meeting over now to Tom Perez.

MR. PEREZ: Good morning. The following announcement addresses the issue of conflict of interest with respect to this meeting and is made part of the record to preclude even the appearance of such at this meeting.

The Food and Drug Administration has prepared a general matters waiver for Dr. Brian Strom, special government employee, which permits him to participate in today's discussion. A copy of this waiver statement may be obtained by submitting a written request to the agency's Freedom of Information Office, Room 12A30 of the Parklawn building.

The topic of today's meeting is an issue of broad applicability. Unlike issues before a committee in which a particular product is discussed, issues of broader applicability involve many industrial sponsors and academic institutions.

The committee members and invited guests have been screened for their financial interests as they may apply to the general topic at hand. Because the general topic impacts so many institutions, it is not prudent to recite all potential conflicts of interest as they apply to each participant.

FDA acknowledges that there may be potential conflicts of interest, but because of the general nature of the discussion before the committee, these potential conflicts are mitigated.

In addition, we would like to disclose that Dr. John Sullivan is the nonvoting guess industry representative on the committee. He is not a government employee, and hence, we do not screen him for conflicts of interest and can make no comments on his actual or perceived conflicts of interest.

In the event that the discussions involve any other products or firms not already on the agenda for which FDA participants have a financial interest, the participants' involvement and their exclusion will be noted for the record.

With respect to all other participants, we ask in the interest of fairness that they address any current or previous financial involvement with any firm whose product they may wish to comment upon.

Thank you.

CHAIRMAN GROSS: At this particular point, I'd like to introduce everyone to you. So we're going to start off at the table on the left here. If you would please introduce yourself.

MS. LEATHERWOOD: I'm Sharlea Leatherwood. I'm a pharmacist and owner of pharmacies in Kansas City, Missouri. I'm Chairman of the Executive Committee of the National Community Pharmacists Association.

MS. OSTER: I'm Karen Oster. I'm the Assistant to the Executive Director at the National Association of Boards of Pharmacy.

DR. COSTER: John Coster, Vice President of Federal and State Programs with the National Association of Chain Drug Stores.

DR. SVARSTAD: Bonnie Svarstad, professor of social pharmacy and sociology, University of Wisconsin, Madison.

DR. SULLIVAN: John Sullivan. I'm a physician. I'm currently head of clinical pharmacology at Amgen, and I represent the Pharmaceutical Research and Manufacturers Association.

DR. COHEN: I'm Mike Cohen. I'm President of the Institute for Safe Medication Practices, and I'm a pharmacist.

DR. STROM: I'm Brian Strom. I'm a pharmacoepidemiologist at the University of Pennsylvania.

MR. PEREZ: Tom Perez, Acting Executive Secretary to this meeting.

CHAIRMAN GROSS: Peter Gross. I'm Chair of the Department of Internal Medicine at Hackensack University Medical Center in Northern New Jersey.

DR. CRAWFORD: Stephanie Crawford, University of Illinois at Chicago, College of Pharmacy. I'm a member of the Drug Safety and Risk Management Advisory Committee.

DR. CAMPBELL: Bill Campbell, Dean of the School of Pharmacy at the University of North Carolina, Chapel Hill.

DR. GARDNER: Jacqueline Gardner, Associate Professor, Department of Pharmacy, University of Washington in Seattle.

DR. DAY: Ruth Day, I'm at Duke University and direct the Cognition Laboratory.

MR. LEVIN: Art Levin, Director of the Center for Medical Consumers. I'm the consumer representative on the Advisory Committee, and I was a member of the Keystone Steering Committee.

MR. McGINNIS: Tom McGinnis, Office of Policy, Food and Drug Administration.

DR. TRONTELL: Anne Trontell, Director of the Division of Surveillance, Research and Communications Support in the FDA Office of Drug Safety.

DR. SELIGMAN: Paul Seligman, Director of the Office of Pharmacoepidemiology and Statistical Science and Acting Director of the Office of Drug Safety, FDA.

CHAIRMAN GROSS: At this point I'd like to reintroduce you to Dr. Paul Seligman, Director of the Office of Pharmacoepidemiology and Statistical Science, who will give us a welcome and charge to the committee.

DR. SELIGMAN: Good morning. Welcome and charge.

(Laughter.)

DR. SELIGMAN: First of all, let me thank all of the members of the Advisory Committee this morning for giving their time, effort, and expertise to address this important issue as well as our guest speakers and all of you in the audience who will be participating in today's meeting.

This is the first full, independent meeting of this Drug Safety and Risk Management Advisory Committee, and I can't think of a more interesting, appropriate, and timely topic for this committee to engage in.

I also wanted to take a brief moment to thank the members of the FDA staff who worked so hard to set up and develop this meeting and agenda, including Anne Trontell, Jeanine Best, Ellen Tabak, Kathleen Bongiovanni, Melodi McNeil, Tom McGinnis, Kimberly Topper, as well as others.

As mentioned, the focus of today's meeting is the information that consumer receive with prescription medications. FDA has long held that to achieve the greatest benefit and to maximize the safe use of prescription medications that the consumer needs accurate, clear, comprehensive information, and that communication of this information is vital.

To this end, the FDA regulates and reviews certain communication products, including the medication guide and the patient package insert, which is required to be included in certain hormonal therapies, such as over-the-counter or -- excuse me -- such as oral contraceptives as well as other hormonal preparations that contain estrogens and hormonal replacement therapy.

In addition, the FDA regulates and reviews information produced by manufacturers and sponsors that are used for direct-to-consumer advertising.

By way of background today, we have four presentations. The first presentation will trace the history of the consumer drug information issue which has a long, interesting and stored history, and will trace us to where we are today.

Second, we will hear from representatives from the National Association of Chain Drug Stores, as well as the National Community Pharmacists Association, who will describe how the information that is contained in the FDA approved label is conveyed to consumers and how this information is developed and the flow of the information from the developer to the consumer.

Next we will hear about a study that was done that will review the progress in meeting both the year 2000 and 2006 goals, which are stated on the slide, that by the year 2000, 75 percent of all consumers who received prescriptions receive useful information, and that this number increased to 95 percent by the year 2006.

And finally, to insure that communication with consumers is based on good evidence and scientific principles, we will hear about the sort of key cognitive principles and some of the research done on consumer comprehension of educational materials.

The scope of this meeting then is to focus on how to improve consumer information and how best to achieve the goals set for the year 2006 by Congress. We hope to engage in a discussion this afternoon on next steps that both the FDA and in collaboration with pharmacies and consumers can work to achieve these 2006 goals.

We have posed three classes of questions for the committee to consider. These include what additional analyses of the current data should be conducted. That would be most useful in addressing this question of how to make useful consumer information available patients.

Second, we will ask the committee to look at what additional research needs to be conducted to insure that the best evidence is used to improve patient medication information.

And finally, we will be asking our committee to provide us advice on what actions would improve consumer medication information to meet the 2006 goal.

As you will hear today, substantial progress has been made in many areas towards achieving the 2006 goals, but that there are still some areas where we need, I think, considerable amount of work.

The FDA believes that continues collaboration with consumers and the pharmacy profession and close monitoring of this process should lead to full compliance by 2006 with the goals identified by Congress.

With that, Mr. Chairman, I'd like to turn the meeting over to you and wish us all a successful meeting today.

Thank you very much.

CHAIRMAN GROSS: Thank you very much, Paul.

Next I'd like to introduce Tom McGinnis, who will present some background and the history of patient information efforts.

MR. McGINNIS: thank you, Mr. Chairman.

Over the next ten minutes, I'd like to go through 34 years of history of patient information at the Food and Drug Administration.

FDA started in the area of patient information back in 1968, when the agency believed that consumers needed information on how to use isoproterenol inhalation products properly. Back at this time the only one conveying information to patients was the prescriber. It was unethical for a pharmacist to talk to patients about their medication.

It wasn't until 1969 that the American Pharmaceutical Association changed the code of ethics allowing pharmacists for the first time to communicate to patients about their prescription medications.

In 1970, FDA required for the first time patient package inserts for estrogen containing products. This was because the data and information on estrogens and oral contraceptives was changing very rapidly, and the agency believed consumers, women, needed this information and needed to be assured that this information was getting to women about these products.

This was done by notice and comment rulemaking, a formal procedure that is quite arduous in many cases.

In 1979, FDA embarked on a project with ten classes of drugs where the agency wanted to develop, have the industry develop and the agency approve, patient package inserts. There was a lot of controversy on this project, particularly from medicine and pharmacy. The physician groups, again, wanted to be the only ones conveying information to the patient. There was concern that patients seeing adverse events on these products may actually develop these adverse events. Typical textbook type medicine reactions, and that they may even be concerned enough not to take the medication after seeing what possible side effects could occur.

The pharmacy profession had a problem with the paper that would be generated and pushed through the distribution system in the United States. Many pharmacy departments were very small and compact. It might require putting a file cabinet back there, which was not going to be doable in many pharmacies, and as that patient information changed, which it tends to do fairly rapidly, the pharmacists have to take the new information, get the old information out of that file cabinet to make sure that the patient did get timely and up-to-date information with their prescription drugs.

With that controversy, FDA withdrew that patient package insert proposal rule in 1982. At the same time, FDA set up an internal working group on patient information, and the private sector formed the National Council on Patient Information and Education to foster this private sector initiative, and I believe you'll be hearing more about that later.

In 1991, FDA revisited this issue. We had done telephone surveys of patients beginning in 1982 when we wrote the proposed rule, and we found in 1991 that only 32 percent of patients were telling us that they were getting any type of written information when they picked up a prescription drug.

We took a look at some of these pieces of information that were being picked up at this time, and we found the information to be very variable. In some cases there were only a few bullets of information being given to patients. In other cases, there was a paragraph or information and no more, and yet in other cases, there was a full page or two full pages of information being given to patients that looked very comprehensive and useful.

FDA continued to encourage the voluntary efforts to provide patient information through the private sector initiatives, through articles and speeches by senior FDA officials.

On August 24th of 1995, FDA published the medication guide proposed rule mainly for drugs with serious and significant side effects. As I mentioned, for FDA to mandate consumer information with a prescription drug, we had to go through formal notice and comment rule making, which was a tedious process. This essentially would eliminate that process when the agency believed a drug posed serious and significant side effects and consumers needed that information to possibly avoid those.

The proposed rule also allowed the private sector to continue with their efforts. However, the agency was getting concerned that the progress was not up to par. So the agency set some performance standards for the private sector, those being 75 percent of patients should be getting useful information with their prescription drugs by the year 2000, and that virtually everybody, 95 percent, should be getting this information by the year 2006.

FDA also proposed broad criteria what these things should look like in this proposed rule.

In 1996, FDA convened a public meeting like this one to discuss the private sector initiatives of the proposed rule and to clarify information on what would be required of the drug industry in the formal medication guide process.

There's still some controversy on whether the agency should embark on this process, and on August 29th, Congress passed the law and the President signed it into effect, and that was Public Law 104-180. That public law directed the Secretary of Health and Human Services to facilitate development of a long-range action plan that meets the goals and objectives through private sector initiatives.

It gave the private sector an opportunity to meet distribution and quality standards of the plan that was to be developed, and it codified FDA's distribution and quality goals of 75 percent of patients receiving useful information by the year 2000 and 95 percent, virtually everybody, by 2006.

The secretary, in not wanting to have to review multiple plans and pick one plan that was submitted, immediately contracted with the Keystone Center. The Keystone Center was a nonprofit, consensus building alternative dispute resolution organization that was successful in the past in bringing together stakeholders with varying interests in coming to a consensus process. The statute only allowed 120 days to facilitate development of the action plan by interested stakeholders.

The Keystone Center selected 34 private sector organizations to develop this action plan. The government was not part of this process.

The action plan was collaboratively developed and accepted by the Secretary in January of 1997. The criteria to develop usefulness endorsed the broad criteria in the public law, and described eight specific criteria that should be met. It was consistent with the public law, and the plan called for periodic assessment of the quality of written information.

The eight criteria that were developed by this consensus building process were, first, the consumer should have the drug name and its indications for uses.

The consumer should see the contraindications and what to do if they experience one of those or have that particular condition and are accidentally prescribed this medication.

How to use the drug to get the most benefits out of the medication.

And precautions, how to avoid harmful side effects.

The fifth criteria was serious or frequent adverse reactions to expect and what to do about them.

The sixth was general information, encouragement of the consumer to ask questions of their physician and pharmacist.

The seventh was scientifically accurate and not promotional and up-to-date information should be conveyed to the consumer. This information was not to be promotional in any way.

And finally, the information needed to be comprehensible to the consumer. It needed to be legible and readable to those consumers with a sixth to eighth grade reading comprehension level.

In 1998, FDA contracted with the National Association of Board of Pharmacy to do a pilot study to see where we were, how we were and how were doing at the time. The State Boards of Pharmacy arranged with eight Boards of Pharmacy to collect this information and to provide the materials for review.

The contract called for development of a scoring instrument based on the eight Keystone criteria to assess the usefulness of this information.

In February we held a public meeting, again, like this one to discuss the results of the interim assessment. At that time deficiencies were still noted in the distribution and usefulness of information. Stakeholders were given feedback to the agency on the draft scoring instrument. Changes were made in that instrument to be used in the end of the year 2000 study, and we announced our plans to start that study at the end of the year 2000.

In June, FDA renewed its contract with the National Association of the Boards of Pharmacy and the University of Wisconsin School of Pharmacy for the evaluation phase. The study was implemented in January of 2001, and throughout the year 2000 a professional shopping service was used to present four prescription drugs to over 300 pharmacies in the United States in order to make a national projection on how we were doing, and the results of that study will be presented to you shortly.

Thank you.

I'll take any questions that you might have.

CHAIRMAN GROSS: Thank you, Tom. That was a very nice review.

Does anyone have any questions, in particular, on the Keystone criteria or how we got to where we are today?

Michael?

DR. COHEN: Let me try to understand something because I'm just a little bit confused about some of the background material that I read. To meet the 2000 goal, was it just the written information that had to reach the 75 percent or the individual criteria had to meet the 75 percent level when assessed or what?

And if it was the individual criteria, which ones had to meet the 75 percent?

(No response.)

MR. McGINNIS: All information picked up by the agency was scored against a scoring instrument that we'll hear a little bit about in the next presentation. That information then was evaluated and we'll see extensively how it was evaluated.

And then the final distribution levels -- and I hate to take Dr. Svarstad's thunder away in announcing those numbers right now -- were evaluated and presented to the agency in the final report that you're going to hear soon. So I want to defer that, Mike, until we hear Dr. Svarstad's presentation.

CHAIRMAN GROSS: Any other questions?

(No response.)

CHAIRMAN GROSS: If not, we're staying ahead of schedule here. Nothing wrong with that.

Thank you very much, Tom.

Next we're going to hear about the information development and flow from the developer to the consumer. Dr. John Coster and Sharlea Leatherwood will present.

Yes, Anne?

DR. TRONTELL: We might offer some clarification to the previous question about the criteria. The law itself describes numbers for what is termed "useful information," and that useful information was subsequently specified in the criteria that are going to be discussed and presented.

So the overall description and requirement was for what was termed useful and then operationalized, as we'll hear.

CHAIRMAN GROSS: Okay. Thank you, Anne.

Dr. Coster.

DR. COSTER: Thank you.

Mr. Chairman, members of the committee, we appreciate the opportunity to be here. I understand my role and that of Sharlea is to describe as best we know it the process by which information ultimately reaches the consumer, the written information that you will be discussing here today.

Just as background, NACDS represents approximately 33,000 community retail pharmacies. We have about 200 chain member companies. We fill about 70 percent of all out-patient retail prescriptions, and like Art, I was a member of the original Keystone committee that met what seems like decades ago, but back in 1997 to put together the action plan.

So my goal today is to give you our perspectives, our research on what we know happens in terms of the process by which consumers receive this written information.

First, let me start by saying that NACDS is strongly committed to working with FDA, consumers, and our member pharmacies to continue to make strides. I think we've made significant strides since 1997. FDA's own data indicates that the percent of consumers receiving written information has increased significantly over the last ten years, but we're equally concerned that the so-called quality goals are falling short, and we ourselves want to know why that is happening as well.

I may not have all the answers today for you, but I can tell you that we are working with our members and trying to provide as much quality written information to consumers as possible.

The provision of written information by pharmacies to consumers really began as a value added service. As you know, over '90, the Omnibus Budget Reconciliation Act of 1990 required that pharmacists offer to counsel Medicaid recipients on their prescriptions, and as a result of that, almost every state changed its practice laws to require that pharmacists offer to counsel all recipients or all patients, for that matter, on their prescriptions.

The written information leaflets, I think, really started to generate after that as a leave behind for patients to help reinforce for them the oral counseling that they receive from their physician and their pharmacist.

We all know as consumers ourselves it's often difficult at times to remember everything we're told about a prescription either by our physician or by our pharmacist, and these leaflets really act as references for patients to refer back to something about the particular prescription medication.

We don't believe, however, that these information sheets cannot and should not be viewed as a substitute for the professional advice and counseling of health professionals.

In terms of how the process flows, the nuts and bolts of how information ultimately gets down to consumers, retail pharmacies do not produce this information on their own. We purchase it or, more accurately, we license it from the major database companies that produce it.

And due to recent consolidations in the marketplace, there are really only a few producers of this information left in the marketplace. Our understanding, for example, is that First DataBank and Medi-Span provide the written information to the overwhelming majority of the retail pharmacy marketplace, but there are other providers of written prescription information.

For example, there's a company called Gold Standard, Micromedex, USP of course, and Facts and Comparisons. I can't tell you exactly what percent of the market these other companies have or which particular market they serve, but there are multiple providers, but there are only really a few left that provide written information to the retail marketplace.

They can talk, and I'm sure they will later, in greater detail about how they actually produce the information, but our understanding is that they rely on the FDA approved labeling, peer reviewed literature and other sources.

In talking to some of our members about this, on occasion a pharmacist will note that something is incorrect in the information that's provided and will notify the database company and ask that it be corrected.

So the database companies produce the written information. Then what happens to it?

Almost every pharmacy has an underlying software processing system, a prescription processing system or a software vendor, and that software system helps to manage prescription records, helps to check for adverse reactions, produces labels, and interacts with what's known as a switch to help adjudicate and process claims.

These systems have greatly enhanced the efficiency of the prescription delivery process and have helped to improve the quality of care provided to patients by providing real time information to pharmacies.

So the database companies produce the information. Then it flows down to intermediaries' software vendors, and these software vendors then take this written information and incorporate it into the pharmacy's underlying prescription processing system.

We understand there are probably about 75-plus pharmacy software database vendors. These are companies such as TechRx, QS1, PDX, and there are others. Some of these are very small vendors, and they only serve a limited number of pharmacies. Others are larger.

But our research indicates that pharmacies do not necessarily know whether the information they're receiving from their software vendor, in fact, meets the Keystone criteria, and in some cases the information may have to be updated or modified to fit within the processing system that the pharmacy uses.

Some of our large chains do not utilize software vendors. They have their own database systems that they develop and operate, and in talking to a few of them over the last couple of weeks, they have told us that they make no changes to the information they receive from the database companies.

In fact, I understand that part of the new and renewal licensing agreements, at least two of the major providers are requiring that no changes be made to the information.

I did an unscientific survey of the information being provided by eight of our chains over the last couple of weeks, and whether or not this is a big revelation to anybody, it looks like most of that information is coming from First DataBank as the source of the information.

The information presentation appears pretty consistent. There is some variability in the type face, in the size of the printing font. We understand that there is variability in terms of how often the information is updated in the systems as well.

For those that obtain information directly from the company, like some of our members, the information might be updated more frequently. In other cases, it's only updated quarterly.

Now, whether this has any impact on the usefulness of the information, you know, is a question.

Again, many of our members are probably unaware of the Keystone criteria for written information and probably don't perform their own assessment. I think you'll find that most pharmacies obtain independence, and Sharlea will address this further, rely on what they receive from the companies. We trust that this information is factually correct and, you know, are relying on the database companies to provide the information to us.

We believe that any information presented to patients must not only be useful, but must compel them to read the information. Written information that is two or even three pages long may not be read by patients because of its length, and clearly that's not a desired outcome.

We understand that already about 80 percent of the information produced is greater than two pages in length, the average being about a page and a half. Therefore, any additional mandates on information that's required would obviously spill over into two or three pages.

Alternatively, information that is too short or not specific enough may not be useful. So, for example, information that simply says report any side effects to your doctor clearly doesn't help patients understand what to look for.

Thus, I think the point there is balance is what's necessary.

Let me also describe some logistical information, logistical issues for pharmacies that provide this information. I already said that in terms of the flow of information it comes from the database companies through the software vendor in many cases, and sometimes directly to the chain, and then the information has to be incorporated into the software system and then has to be printed out in some way.

Those of you who have never been behind a pharmacy counter, most pharmacy software systems utilize one printer. There's one printer behind the counter. The printer is printing out in many cases one sheet, and on that sheet is included, for example, the prescription label, refill information, maybe auxiliary labels, as well as the written information that pharmacies are providing.

So there's generally not two systems in the pharmacy, one printing out the labeling information and auxiliary labels and one printing out the written information. It's usually all coming on the same information sheet.

So, you know, just logistically it would obviously slow down the prescription filling process when two and three additional pages of written information is being printed off.

In terms of the marketplace again, up until April 2000, we understand that First DataBank was producing both a short form and a long form of written information. We understand that the short form was discontinued, again, back in April of 2000, but some pharmacies may have continued to use the short form because it remained in their prescription processing system.

Dr. Svarstad might have some additional insight into that.

We're not sure why that was being done, but if those forms were, in fact, collected as part of the 2001 review, that might explain why some of the results fall short.

We also understand that mergers and acquisitions in the database companies, which I referred to before, may have created some issues related to updating written information to meet the usefulness standards.

Having said all of that, I just want to give you a few suggestions on where we think we might go from here in continuing to make improvements in the quality and quantity of written information.

First, without stating the obvious, I think that the database companies producing the information should be providing Keystone compliant information to their customers. There's only a few of them. So it doesn't seem like, you know, a huge task to interact with them.

And for the most part, I think they are producing that information.

Second, in terms of the software vendors, this is where perhaps we need to focus a little bit more of our attention. It's clear that many pharmacies don't really know whether the information being produces is Keystone compliant. They rely on their software vendor to assume it meets the usefulness criteria.

How can pharmacies know that this information is, in fact, compliant? It may be in the interest of the agency to convene a workshop for companies to help educate them about the criteria and what it means to be Keystone compliance.

And we suggest that you might work with the group representing the software vendors, the American Society for Automation and Pharmacy, also known as ASAP. There's a group representing everybody, as you know.

We continue to emphasize to our members the importance of distributing information that is not edited. I am reasonably comfortable that many of our members are not editing the materials, but, you know, again, I think it's incumbent upon us to insure that for our smaller members in particular that they are not editing information provided to them by the database companies.

We at NACDS continue to support discretion to health professionals in developing information, communications to patients as a function and responsibility of physicians, pharmacists, and other health professionals. Every patient is different. So we would have concerns with any additional mandates or prescriptions or prescriptive criteria on what we should distribute.

We suggest that you also look at other out-patient dispensers of information. The study, as I understand it, only looks at independent and chain pharmacies. Clearly there are other entities that distribute out-patient information. Mail order is about 13 percent of the market. Hospitals have out-patient departments. Clinics, and there are even federal facilities that distribute out-patient information.

These patients should get no less useful material than other patients, and I think the term "useful" also needs to be assessed. "Useful" is a subjective term. What's useful to me may not be useful to you, and while there clearly needs to be some minimum standards for usefulness, to my knowledge, the Keystone criteria have never been validated for usefulness.

So we would urge that the committee consider further research into what truly constitutes useful information to consumers, whether those criteria in Keystone are, in fact, the most useful to patients.

So, again, let me reiterate that we are very interested in working with the FDA, consumer groups, this committee in moving forward to improving the usefulness of the information. We look forward to working with you over the course of the discussion about this issue, and we hope you would consider us partners in trying to improve the usefulness of the information.

And we'd be happy to answer any questions about my comments.

CHAIRMAN GROSS: Are there any questions for Dr. Coster?

DR. CAMPBELL: Yes, I have one.

CHAIRMAN GROSS: Yes.

DR. CAMPBELL: Thank you, John. That was very helpful, at least for this member of the committee, to understand there is a significant element of the black box here operating and some of your thoughts about peeling back layers to see what's inside the box I think would be very helpful.

I did want to follow up on your comment about maintaining an approach of not editing the data. It does seem to me that the suppliers of the data are not aware of the specific user of the data in the pharmacy, and when it comes to that point, the pharmacy, the pharmacist is aware that the recipient of that articular information piece may not need to know that it may result in an enlarged prostate because that recipient may not have a prostate.

And there may be cultural differences that require communication via a different language, and gender specific information, and so on and so forth.

So I wonder why you wouldn't want to allow the endpoint provider of information to edit what is very generic information in order to make it more specific to the individual who will be using it.

DR. COSTER: Well, first of all, I agree with your characterization of information that may not be helpful, useful or even relevant to particular patients. We understand that it's the database companies that are going to start requiring as a condition of licensing the information that the pharmacy not change the information.

Now, I don't know what that means in terms of changing it. Is it you can't change it, you know, any word, or can you modify it somewhat?

So that may be a question better to ask the companies in terms of what their new licensing agreement will say.

I agree with you. I think pharmacists, physicians should have the flexibility to tailor information specific to particular patients. Information may not be relevant to an individual patient. On the other hand, you may be taking out information that may not meet these so-called criteria.

So I think this is where you have to try to strike the balance here. What are the minimum criteria for useful information to patients, but at what point do you allow the health professional to modify to make it useful or more useful to the individual patient? Should there be any flexibility in modification?

This was one of the issues back when FDA issued their regulation back in 1996, the concern by the pharmacy and medical profession that it would lead to prescriptive standards that would not allow for flexibility for health professionals to tailor information specific to patients or allow for innovation in the future as new, different ways of delivering ways of delivering information became available.

So I think that's the balance. You have to ask yourself: at what point does the government say, "This is enough in terms of standards," but allow the health professions flexibility to alter the information?

And, again, in terms of what the database companies are going to require, I don't know. I haven't seen the licensing agreements to know how far you can go to edit the information.

CHAIRMAN GROSS: I have a question. How do these medication sheets address, let's say, a compromised renal function in a patient? Do they address it at all?

If the kidney function is not normal and lower doses, let's say, of the medication should be used, are they address at all on these information sheets?

MS. LEATHERWOOD: Would you like for me to?

DR. COSTER: Yeah, please. And, please, maybe you also want to respond to Bill's question.

MS. LEATHERWOOD: On some monographs it would say if you have an issue with a renal problem, then to contact your doctor or your pharmacist. Basically that's what the step is. So there is a line that says if you have a kidney problem and it's actually in the terms that is understood by many people, then you address that with your physician

CHAIRMAN GROSS: Thank you, Sharlea.

Michael?

DR. COHEN: Yeah, I can certainly understand why portions of the information might need to be modified to tailor it for a patient. I guess I would worry if we got into the area of risk management. You know, what side effects would be left off? What adverse reactions or potential for adverse reactions might be left off would be a concern.

Also, even tailoring it for individual patient, I would worry at least at this point that in many cases we don't know in the community pharmacies and in some of the other settings exactly what's wrong with the patient so that we could tailor it. We know what the patient tells us if we speak to them. That's not always done obviously, and that would be a concern as well.

And I worry at least at this point, you know, in thinking this thing out would people modify it to make it shorter or to make it compatible with their computer systems up front rather than tailor it for the individual patient's needs?

It's just a concern that we have to consider.

CHAIRMAN GROSS: Okay. One more question from Stephanie Crawford, and then I think we'll move on.

DR. CRAWFORD: Thank you.

One very quick one, and one comment. John, you had mentioned that most of the information that's distributed, I think you said about a page and a half, and I wondered if that's a standard in an eight and a half inch times 11 inch or smaller page.

The other one is the comment about sometimes the pharmacist noticing errors and contacting the vendors, but at the same time perhaps not timely update.

What happens is the pharmacist on a practical basis still distributes the information even if there are errors in it?

DR. COSTER: The first issue of the length of the information, the length of the information clearly depends upon the font size that the printer is using, and just some examples that I collected from chains over the last couple of weeks, it's clear that the font size varies. There are some that are bigger, which is easier to read, and some that are smaller to fit within.

So when I say page and a half, I guess page and a half based on the normal font size that, you know, you would read from -- not some of the ones that I've seen here.

In terms of your second question, when I was discussing this with some of our members, they did say that oftentimes they will identify something in the information that's either not correct and they will notify the companies. I can't answer for you like if they continue to distribute the information.

They might and they might, you know, either take that particular sentence or whatever out of that information or highlight for the patient that in this particular case this is not relevant to you, but I don't know exactly how they treat that.

DR. CRAWFORD: Peter, may I follow up with a brief comment?

About customization, we've only talked about individualizing material for specific patients. There's another way to go on this. The old USP leaflet is no longer available. That form called MedCoach had a customization procedure whereby you could print out the same information, but customize by gender and perhaps age would be another thing that I would add.

But they definitely had the age -- excuse me -- the gender. So you didn't put in the prostate for females and so forth.

And so if there are a few broad classes of individuals where the information would be different, that could be pre-set and provided by the data providers, and that would be a very useful way to go.

But once you get down to the level of individuation, things do fall off, and we actually looked at these customized leaflets and did find out that even when they were being customized for gender, things tended to fall off, and people would update one but not update the other, and so on.

So your point about being very careful not to lose information along the way is well taken. Do you know if anyone is providing information with customized subsets at present?

DR. COSTER: I do not. I think that's a better question directed to the database companies, but I would say that I think I don't get the sense that our pharmacies are today eager to necessarily customize the information to individual patients. I mean, that may be a feature that develops as the technology develops, but I don't get the sense that our pharmacies are like anxious to make this information customized for patients yet.

I mean, they want to provide them the best information they can, and there's a difference between customizing information and editing information out to fit onto. I think that's the concern. Are some software vendors or pharmacies, in fact, just editing out sections of information to fit within certain areas?

That's more of a concern to me right now than is customization. I think that will develop over time.

CHAIRMAN GROSS: Yes, Arthur Levine.

MR. LEVIN: One quick comment. I just want us to be cautious about how much we burden written patient information in terms of exquisite detail. I mean, I think at least for myself and other advocates we think of this as sort of the safety net issue, and not to look to this piece of written information to convey every single big of individualized patient information for every patient.

If you burden it with that, it will be impossible to produce this information.

CHAIRMAN GROSS: Well, I don't want to steal any more of Sharlea's thunder. So Sharlea Leatherwood of the National Community of Pharmacists Association, would you like to present and then we'll go on with the questions after?

MS. LEATHERWOOD: Thank you, Peter.

I hope there won't be a lot of duplicate information and I'll add to instead of duplicating what's been said.

Again, I'll just reiterate. I am a pharmacist, and I am a small business owner in Kansas City, Missouri, and I'm currently chairman of the NCPA Executive Committee.

The National Community Pharmacists Association is a 104 year old organization representing the proprietary and professional interest of independent pharmacies. There are more than 24,000 independent pharmacies in the United States, and they dispense nearly half of the nation's retail prescriptions.

NCPA would like to thank the Advisory Committee for the invitation to provide background and feedback regarding the evaluation of written information provided in community pharmacy. NCPA is pleased with the study's report that nearly 90 percent of patients are receiving patient information when they go to a community pharmacy nearly five years ahead of the benchmark established in 1996 at the Keystone conference.

However, we share the committee's eagerness to insure the quality of written information, and I am on the panel that reviewed the usefulness of the information collected in the survey. After evaluating dozens of patient leaflets, and though nearly all of them provided useful information to the patient, I did find variability in the topics that were covered and the depth in which they were covered.

In describing the pipeline of information flow, I'll try to begin by a description going upstream. With the filling of each prescription, a patient's drug monograph is generated, and as John mentioned, it is generally being generated as part of the label. So it all comes out of the same printer.

In my pharmacy the monograph is attached to the patient's bag after receiving verbal counseling from me or one of the pharmacists that work for me. My pharmacy and nearly all independent pharmacies receive patient information through their computer software vendor, as John has stated.

Nearly all independent pharmacies are computerized and they lease or purchase software support from one of numerous pharmacy dispensing system vendors in the marketplace.

My pharmacy receives updates about twice a month from the software vendor, and these updates are usually done after store hours since the updates are sometimes very time consuming.

The cost of these updates is added to the software support charge from the computer vendor. Changes to monographs or new drug monographs are added during these updates. The pharmacy does not have the ability to alter the patient monographs.

In fact, their agreement with the software vendor usually forbids the modification of the information.

The size of the patient leaflet may vary because the limited space in the pharmacy department limits the number of printers in the pharmacy. The same printer that is generating the two inch by two and three-quarter inch prescription labels may also print the computer monographs on the remainder of the page.

The kinds of printers used in pharmacies also vary widely. Some pharmacies use laser printers, while others may use dot matrix printers, and the type and availability of the printer and the dispensing software that is used, all influence the size of the patient monograph.

Continuing up the information pipeline, our understanding is that the majority of software vendors supporting independent pharmacy computer systems buy their information from First DataBank or Medi-span, and again, our understanding is that the computer vendors are also forbidden from changing any of the information they purchase from First DataBank, Medi-span or other suppliers.

We understand that First DataBank and Medi-span receive their information from primary sources, most prominently from the pharmaceutical manufacturer. Our understanding is that First DataBank and Medi-span take the information given to them from the manufacturer's professional package insert and incorporate it into the patient monograph information sold to computer vendors.

That is our understanding. However, representatives from this organization, as John stated, can better describe the flow of information into their companies and into the software vendors.

During the study period, it was my understanding there was only one provider of monograph information with no other major competitor. There was only one source of this information, and this lack of competition may have negatively impacted the quality of information delivered to the software vendors and then to the pharmacies.

In my pharmacy, we've been giving written information on all prescriptions since 1988. I use them as I counsel my patients about their therapy. We give the monograph to the patient while they are waiting for the prescription to be counted and labeled, and then I point out the various ways to avoid the possible problems and what to do about them if they occur.

My customers have always appreciated this. However, I have to say that the marketplace has driven my patients to other high volume settings, and the quality service that we provide has not necessarily been rewarded.

However, it's only through this verbal interchange that I detect possible probabilities of problems. Some patients do respond that their doctors told them everything, but as I continue to hit the highlights in the monograph, they realize that there's more that they really need to know.

Some physicians have been upset over the years about my interventions, but the benefits certainly outweigh the problems. Undoubtedly quality information is essential in providing care to patients, but I can't stress enough how the addition of oral information from a pharmacist makes the written information come to life for the patient.

In many cases, the written information will prompt the patient to ask me questions while I am counseling them. It's not uncommon for the patient to express relief that the side effect that he or she read about is rare or unlikely. I'm able to assure them and provide guidance on what to do should a side effect occur.

And I mention this just to reinforce that no matter how much effort is placed in trying to perfect the written information, it only augments the pharmacist's verbal information.

I also just wanted to comment since after looking at Dr. Svarstad's report, we did note that independent community pharmacies did not provide as much written information meeting the criteria. And at NCPA we are looking at that.

We believe that verbal communication and personal relationships have been the cornerstone of our business. So the verbal and personal relationship has been the real crux of our interaction with our patients.

And also the other factor is that the technology needs to be upgraded and continually upgraded in independent community pharmacies. Again, as John mentioned, I think that that awareness by the pharmacist of what criteria needs to be used to measure their monographs needs to be given to the pharmacist so they know what they're looking at.

And in fact, I recently just purchased a new computer, and since I was involved in the group that looked at the monographs, I knew what I wanted. So I asked all of the computer vendors to give me a copy of their monograph, and I analyzed it.

But I don't know any other pharmacist that would have been able to do that. So I think that's a real breakdown in what's going on here. The pharmacists need to know what criteria to use in evaluating the systems they have now and getting those upgraded or looking for new systems and making sure they meet the criteria.

Thank you for inviting me to share this information and the perspective from the independent community pharmacy, and I'd be happy to answer any questions.

CHAIRMAN GROSS: Michael, you had a question before? Dr. Cohen.

DR. COHEN: Not so much a question as a comment or actually a follow-up to Dr. Crawford before, but you mentioned it as well.

The information that the vendors supply comes from the package insert or from the official labeling of the product, and there often is quite a delay, and I guess I need to know is it the same database that provides the patient information, that provides information that we use in the pharmacies to detect drug interactions or duplications, et cetera.

Because I can tell you that we've received reports over the years where there's been such a delay that patients have actually been seriously injured or even killed with drug interactions that have been missed, when there's been a known problem that never reached it to the drug information stage.

And I'm thinking of cisapride, for example. It took over a year before we got the drug information into the computer system in a way that it was interactive.

So if that's the kind of delay that we would see with the patient information, you can see the problem there, and that's something that would have to be addressed as well.

CHAIRMAN GROSS: Any comment, John or Sharlea, about delays in getting new information?

MS. LEATHERWOOD: I think that we are going to have to ask those that are the players in that because on our end we really don't know.

We get clinical updates. I do in my pharmacy twice monthly, but I have to pay extra to get it twice monthly, and I'm not sure that all pharmacies do that. So that's one issue.

And the other issue is when do our vendors get it so that we can get it updated.

DR. COHEN: Yeah, if they're waiting only until it gets into the package insert, that could be a tremendous delay before FDA and the company agree to have a black box warning or whatever the situation is, long after reports have appeared in the literature.

CHAIRMAN GROSS: Yes, Arthur.

MR. LEVIN: Just one comment. I think in thinking about written prescription information we do have to think about it as stand alone because there's considerable evidence that oral counseling by pharmacists doesn't occur, and there's a GAO report and other literature that demonstrates unfortunately that over 90 is not being honored in practice.

So I think given that reality, one of the things that I'd like us to keep in mind is that we can't count on everyone, every pharmacy, every pharmacist living up to over 90 and their professional responsibilities to provide counseling to patients, and that many patients may leave the pharmacy without that and with only the written piece of paper.

So we have to make sure that that written piece of paper does what we want it to do, as if nothing else will be done. That's not to say there isn't tremendous value added to oral counseling both by the prescriber and by the dispenser, but we can't count on it.

CHAIRMAN GROSS: Stephanie.

DR. CRAWFORD: Both of the last speakers mentioned or acknowledged that probably most practicing pharmacists -- I'm not practicing, but including me -- were not as familiar with the Keystone criteria. I wanted to ask if NCPA, ACS, or other professional or trade associations representing pharmacists, pharmaceutical spectra are providing educational efforts to provide more information and education for the practitioners.

DR. COSTER: Well, I'll just say for NACDS, I mean, we continually remind our members to look for information that Keystone compliant and print the information that's given to them by the database companies, assuming that it is Keystone compliant.

I don't think we have reached down to the level of educating practicing pharmacists in chains about the Keystone Group or Keystone criteria. Maybe that's something that we need to revisit. You know, we'd be happy to look at ways to do that.

Maybe it's something the colleges ought to be doing as well, you know, in educating pharmacists coming out from school about, you know, what quality written information looks like.

But I mean, I can speak for NACDS and say we have over the past five years tried to consistently remind our membership that, you know, they should be producing information that is of the appropriate length, the appropriate content, the appropriate type size.

Sometimes it's hard to break through to them, and hopefully some of the pharmacists practicing would put some pressure upwards and say, you know, this information really isn't helpful to consumers.

MS. LEATHERWOOD: And I am kind of winging it a little bit here. We did do some articles, especially when the med. guide discussions were initially starting, and so we've had articles in our journal, and we have had at least one or two, I believe, sessions at our annual meeting.

So that information though is dependent then though on who attended the meeting and who read the article. The other source of information would be local and state pharmacy associations. You know, we need to involve them also in the information.

CHAIRMAN GROSS: Bill, any comments about educating pharmacy students about being Keystone compliant?

DR. CAMPBELL: We're not the problem. We do everything well.

(Laughter.)

DR. CAMPBELL: I think a point well taken. It certainly is a challenge. I think the modern pharmacy curriculum is very attentive to training, developing practitioners who are capable of providing effective counseling. I think it's a good question though whether we use this specific terminology and criteria developed by the Keystone Group that would allow the transference of that into practice.

I wanted to make an analogy, I guess. In today's world, if we would not have satisfied the term "Keystone" and substitute HIPAA, one does not talk to a vendor in this world without the vendor giving assurance that the product they're providing is HIPAA compliant or is intended to be HIPAA compliant or is moving toward that.

And yet it sounds as if the vendors in your world, John and Sharlea, are not giving you that assurance. Is that correct?

Sharlea, you've gone through the process of reviewing a number of different products. In any of that conversation, did anyone make the representation that Keystone compliance was part of the commitment they were making to you?

MS. LEATHERWOOD: No, they did not. In fact, I evaluated over two years about almost ten computer software systems at least, and not one of them made any comments about the quality of their monograph. I asked for it, and of course, I was working with the sales force, and they were interested to know that there should be some criteria, but they were not aware of it.

DR. COSTER: May I just follow up on that?

CHAIRMAN GROSS: Yes, go ahead, John.

DR. COSTER: My professional opinion is that I think it's -- and you know, the point of this is to not point fingers -- but I think it's somewhere in the middle something is happening, and it may be at the software level, the software vendor level where most of the focus should be because, frankly, I've worked independent, and I've worked in chain pharmacy. I don't think -- you know, without disparaging pharmacists, many pharmacists do the analysis that Sharlea did to determine the type of information she's being sold.

So you know, it might be useful to focus on how do we make sure that the software vendors are providing information to pharmacists that meet whatever quality criteria we agree are what's necessary for patients. You know, there's probably some editing going on by pharmacies. There may be information that's not totally compliant being produced by the database companies, but it seems to me like that should be where the focus should be, in trying to figure out what happens in the middle between the pharmacy and the database companies.

CHAIRMAN GROSS: Bill, I wonder if you could comment on what aspects of HIPAA you think compliance should be sought just for the audience.

DR. CAMPBELL: Would you clarify? What aspect of HIPAA?

CHAIRMAN GROSS: Yeah, as far as HIPAA is concerned, it's a rather broad statement. What you felt that HIPAA compliance -- you asked if their information or how they relate to patients was HIPAA compliant. What aspects were you referring to?

DR. CAMPBELL: Oh, I'm sorry. The point was just to make the comparison. If we're talking to a vendor today or we're looking at our health care system, the interface, the person at the interface that Dr. Coster was just describing, the person who's generating and managing that data is very sure to tell us that the HIPAA requirements with regard to security, with regard to confidentiality, with regard to the specific details of the Health Information Portability Act are being met in the product they're providing, it isn't that we would use those criteria.

The point I was trying to make is that the Keystone criteria exists in this other interface, and it would be very useful if those vendors would do the same sort of things with the Keystone criteria that are HIPAA vendors are doing with the HIPAA criteria. I wasn't suggesting we'd use the HIPAA criteria for this problem.

CHAIRMAN GROSS: Yes, sure. Okay. Thank you.

Yes, Arthur.

MR. LEVIN: I just want to take us back a little bit to Keystone because I think it's not so easy to certify Keystone compliant as we're making it seem. One of the -- while this was a consensus process, you'll note if you read the report there were some points of nonconsensus in the report that was submitted to the Secretary and actually some options asking the Secretary to choose between some competing options to deal with certain issues.

We certainly had a lot of discussion about some gold standard or seal of approval, Good Housekeeping, UL, whatever, but then the question was: who evaluates the material to award that seal of approval?

And that was one of the issues on the evaluation of the material going forward in which we had some serious inability to reach consensus and which in the report we made two competing suggestions.

Interesting enough, one of them, which came from the minority of members of the Keystone Group, principally consumer members, was to have an FDA advisory committee process that would evaluate not just the progress of meeting the goals for 2001 and 2006, but evaluate the quality and usefulness of the information independently.

And so in sort of a backwards way we've sort of gotten there, but I don't think we could talk about people representing themselves as Keystone compliant just by representing themselves that they're Keystone compliant. We have to have some independent, objective way of evaluating the quality of that information.

And maybe that's something this committee can get involved in, but it's easier said than done.

CHAIRMAN GROSS: Yes, Michael.

DR. COHEN: Just you mentioned before that there were some pharmacies, I guess, chain pharmacies, that were providing their own information, producing their own information. Can you give us an idea of how frequently that is done?

DR. COSTER: No, if that's what you understood, I don't know of any chain that's producing their own information. What they're doing instead of using a software vendor, they produce their own software and license the information directly from the database company rather than a pharmacy contracting like Sharlea's with a software vendor.

I'm unaware of any pharmacy, chain pharmacy, that's writing their own written information. I think everyone buys it either through a licensing agreement they have with a vendor or through a software company they might --

DR. COHEN: Well, some of the mail operations, for example, are very large organizations that might have their own drug information sections that do this. I'm not sure that, you know, that isn't done. I thought maybe you'd know something about that.

DR. COSTER: I can't speak on behalf of what the mail order companies or what any other setting, for that matter. Hospitals might be doing that in the out-patient setting. I just don't know. It's possible.

CHAIRMAN GROSS: Yes, John.

DR. SULLIVAN: It seems to me that we're highly dependent on, if you like, the translation for what comes from the highly regulated package insert to what the database companies produce, and obviously the information has to be sort of accurate, timely, and understandable, and it's unclear what the sort of quality control is on this.

So I think that's sort of one of the issues that you were alluding to where we have to sort of look at it and evaluate it as we go forward.

DR. COSTER: I guess, you know, unlike the mandatory -- there's a mandatory med. guide program which FDA has and then there's the voluntary program. I mean, the mandatory program was a part of the original rule and wasn't finalized back in '96, but was finalized later on, and then there's the voluntary program.

So, you know, our perspective from pharmacy at least is that this information is voluntary provided by pharmacies. We want to continue to work through the private sector to improve it because the concern is that ultimately there'll be FDA regulation of pharmacy practice, which is really a state board issue.

So I guess we're trying to work within a private sector plan to move forward and providing quality written information and trying to avert any type of government regulation of the practice of pharmacy or medicine for that matter. I don't know if you're going to hear from AMA later, but I remember back when we were doing the Keystone criteria. They had some of the same concerns about the potential prescriptiveness of written information or FDA regulation of written information.

So there's a way you can strike the balance without regulating the voluntary provided information. That's what we would prefer in terms of moving forward.

CHAIRMAN GROSS: Okay, Arthur.

MR. LEVIN: I certainly don't want to go back and relive this discussion that went on for four months at Keystone, and certainly preceded Keystone in terms of the discussion of mandatory versus voluntary.

I might note that the only folks who spoke up in favor of a med. guide program as proposed in August of '95 at the meeting held in February were patients and consumers, the people who I think have the most important reason to be interested in getting this information.

Every other stakeholder spoke against the '95 proposal.

I mean, again, not trying to rehash that discussion, but I would be curious to find out what problems the few mandates have caused. I mean, we've had a requirement that information be dispensed with estrogen, with hormonally based products for some time. I think UDs required a patient package insert for some time.

We've had drugs that require medication guides under the '98 statute. So, I mean, I'd be curious. Now that we have these sort of little test cases, what's the problem? Why is this something that pharmacies should be concerned about?

What is the kind of problem, you know, and how is it interfering in the practice of pharmacy? I don't get it.

CHAIRMAN GROSS: Sharlea?

MS. LEATHERWOOD: I think that any time that we get more mandates and, you know, one mandate after another, our practice becomes overlaid with sometimes difficult things to fulfill.

I always give information, but there have been cases in my professional practice where you had to be very careful about how you gave that information and helped the patient understand it. It took more time.

When you have a mandate, that means that you must give it, and I've even had physicians who have asked me not to give it in the same complete manner as others.

And I realize the patient has that right to know, and I feel that they do, and I've tried to help them understand that, but the mandate then would require that you give it no matter what their situation is, and that's one aspect.

Again, as John said, we are regulated by state boards of pharmacy, and we really do not need another overlay of another government body to license us. We are inspected regularly, and you know, the NABP and the boards of pharmacy -- certainly any issues that need to be addressed with the professional pharmacy, I think, should come through that channel and a way to improve what we do.

I know you said the oral information. The requirement in Missouri is that you ask the patient if they have any questions. So in the busy, high volume settings that we have today, the question is asked, and it may not even be asked in a way that would have the patient answer it.

So that being said, a mandate doesn't necessarily mean that it's going to get you where you want to go. It doesn't necessarily mean that the patient is going to get the best written information all the time.

It's there; it's available. It's required by law. It isn't done. So I think if we can get voluntary compliance and work through the current system rather than to create another overlay of regulation, that it will be a win-win for everyone.

CHAIRMAN GROSS: Okay. Thank you, Sharlea.

I think we'll move on now to Bonnie Svarstad. Dr. Svarstad will present a report of the evaluation of the written patient information, penetration, and usefulness, and with a Hollywood flair. This is Part I, like Men in Black, Part I.

(Laughter.)

DR. SVARSTAD: Okay. Thank you to the committee for inviting me to present the results of this report. And I thank every who took the interest and the time to attend today, and I hope that you will feel free to ask questions as the time permits during the question and answer period, and I hope this report is useful to you.

First, I'd like to acknowledge my colleague, Professor Jeanine Mount at the University of Wisconsin, Madison. She's at another meeting, so can't attend today. But she certainly was very helpful in getting this report to fruition, as was our whole research team.

I think it's important, first of all, to acknowledge that this study was a very collaborative study in the sense that it was done in cooperation with FDA and the staff, especially, I think, Dr. Ellen Tabak, who has been providing us with assistance and support from the very beginning.

This study was done under very tight time constraints, and sh was very helpful in making sure that we were able to do that.

I also thank the NABP, the National Association of Boards of Pharmacy, for providing their support, and certainly to the national expert panel who has played a very critical role, and I'm glad that Sharlea is here to answer any questions about what that was like and what the role that she played.

The national expert expert panel was made up of 16 individuals. These individuals, the majority of them were nominated by NABP. To get to this list of individuals, NAPB, as I understand it, invited seven pharmacy organizations to nominate individuals from their organization to serve on this panel.

We also made nominations based on our understanding of individuals around the country who were specialists in drug and health communication and pharmacotherapy. So it was a collaborative process, one in which in the end we had individuals who were then either experienced pharmacy practitioners and/or experts in pharmacotherapy and communications hoping to get a broad perspective.

We had faculty from nine different colleagues and universities, and I'd like to just list the panelists so that you can see who was involved. You may not know all of these individuals, but I think they've been very active nationally in these issues:

Mary Amato

Heidi Anderson-Harper

Bob Beardsley

Dr. Chester A. Bond

Marie Gardner

Betty Dong

Carole Kimberlin

Sharlea you've met.

Duane Kirking

Matt Osterhaus

Anthony Provenzano

Mary Pubentz

Betsy Sleath

Jenene Spencer

Judith Hanson

Gayle Dicter

And myself as chair.

We operated as a committee largely through modern technology. We mailed things to them. We used E-mail a lot, and they returned things to us via Fed.Ex., et cetera. So we tried to do this as efficiently as possible. And I thank them for all of their efforts.

Their role, I should have said, was diverse. First off, they looked over the criteria and commented on the criteria. They also commented on the expert evaluation forms and other features of the study methodology. But their main role, I think in the end was to actually evaluate the information sheets.

Now, I should say just a couple of things about past studies that have been done here. As several speakers have already noted, I think it's important to note that the distribution of patient information has increased dramatically. From 1982, one of the first studies, nationwide studies done by FDA, they found 16 percent of the patients reporting written information, some kind of written information, increasing to 74 percent in 1998.

In the interim study, that is, the study that was done in 1999, in eight studies we found that 87 percent of the patients or shoppers who went into the pharmacies were given some kind of information, but the quality was highly variable.

And I will talk about how we define some information. In this case, in this study, and as in the previous study, we included any written information beyond the individualized prescription label. So if it was one line, it was considered information. If it was two lines, it was considered information. If it was two pages, it was considered information.

So we will try to break that down for you as the study goes on, but it's important to see that when we say a certain percentage of individuals received information, it's meaning information of all kinds or of all lengths. Okay?

Now, how is this particular study different? One of the most important differences is that pharmacies were sampled from a national electronic list. In the previous study we only looked at eight volunteer states. So this makes it, as far as I know, the only nationwide study of drug information conducted in the world, and I'm familiar with studies that have been done in other countries.

Professional shoppers visited the pharmacies. In the prior study we had some inspectors and some temporary staff. These varied from one state to the other as to how much training and experience they had. In this particular case, professional shoppers were hired by the same professional research firm to visit all pharmacies. So I think the standardization was improved.

Both experts and consumers rated information. In the previous interim study only the experts. So I think we have a very important addition in this study by asking consumers to rate the information.

We also performed additional analyses.

Could someone bring me the water glass? John, thank you very much. I'm kind of fighting the aftermaths of bronchitis and sinus infection. So thank you.

Primary aims of this study are shown here. What percentage of the patients receive information? How do experts and consumers rate it? And how well does it meet the criteria?

These are the primary aims of the study. Recently, in the last few months, we received additional support from FDA to do some additional analysis as to how expert and consumer ratings compare and to identify some of the factors that influence the variability and information. Is it influenced by pharmacy type as well as by leaflet characteristics?

The objectives for today are really to review the study design and procedures and to get questions, to review the evaluation criteria, and to show you the forms, and to present results in two parts, leaflet distribution in the first part, in the second part factors that might influence the ratings.

Now, for the study design, I think as most of you know shoppers acting as patients presented four prescriptions at each pharmacy. Leaflets were mailed back to us at UW. We then mailed them out to the experts and to the facilitators for the consumer evaluation, and all of those came back to UW where we ultimately did the analysis.

Now, about the sampling of the pharmacies, you can hopefully read here the pharmacies that were excluded. We excluded some states. I wish they would have included them, Hawaii and Puerto Rico so that at least I could have gone to a warm place.

But we decided or the FDA decided to limit us to that in that respect. We did not include government settings and hospitals, clinics, long-term care, mail order, et cetera.

This left though a large number of pharmacies nationwide, 57-some thousand, and it's the 57-some thousand that these pharmacies were selected from.

Ultimately after discussing different procedures, we decided to do a simple random sample of 384 pharmacies, making it an excellent, I think, sample.

And in the end we had 65 percent of the pharmacies were chain or what I should refer to as multi-unit and 35 percent independent, and I think the previous speakers have noted that this is pretty representative.

We know, for example, at least these statistics vary from one year to the next. The statistics that I looked at, for example, in terms of the volume of prescriptions that are dispensed in different settings, I think the data that I saw were 66 percent of the prescriptions were dispensed in chain or multi-units. So I think we're pretty close, but of course, we do not have mail order.

Data ended up being collected in 44 different states. Now the observer protocol.

As I noted, shoppers were hired by a professional shopper firm. Seventy-two percent of the visits were by females, 66 percent by persons 45 and over, with a mean age of 50.

Now, the reason for that really is to make this realistic since these medications were for diabetes, heart disease, et cetera. You don't want someone 20 years old going in and presenting a prescription for nitroglycerine for obvious reasons. We wanted it to be as realistic as possible.

All shoppers had a standard scenario to make this uniform from one state and one pharmacy to the next. The standard scenario briefly involved these four new prescriptions: atenolol, glyburide, atorvastatin, nitroglycerin sublingual.

And we can talk about why those medications were selected during the period if you wish, but I think it's interesting that these prescriptions are among the top ranking or top most frequently dispensed prescriptions. Certainly the diagnosis of diabetes and hyperlipidemia are among the top ten reasons for patient visits.

So I think we have a scenario here that's not looking at the bizarre or not looking at the rare, either rarely prescribed drug or rare diagnosis. We're looking at common and frequently used medications.

The patient was encouraged and required to not ask questions or initiate talk. In other words, they should not be seeking information. They should let the process unfold. This, in fact, is probably quite realistic as most pharmacists would tell you. The patients do not generally seek information or ask many questions; fairly passive.

If asked though, the patient was prepared to respond with the standard scenario, and they were told they had a scenario, and that was given in the final report. I don't want to try to read it here, but if you have questions or interests in that, you can go to the final report and see exactly what the patient was asked to tell the pharmacist if they were asked, "Why are you being given this medication? How did the circumstance arise that you got these prescriptions?"

Basically they were to tell the pharmacist that they had recently, very recently, just been diagnosed as having diabetes and some heart disease, very vague. And they were to say they had never used the medication before, if asked.

So this would be typically then a patient who does generally, I think, need information, if you will. They've not used the medication before.

The shopper mailed the materials to the shopper firm. The firm removed identifiers so that the experts and the consumers would not have privileged information. All leaflets, brochures, and other materials then mailed to us, all items, as I said before, referred to as leaflets.

Now, let's talk about the evaluation forms. Each included the eight general criteria that Tom talked about before. To operationalize those criteria or to quantify them, there were 62 to 63 subcriteria for each form, and these were drug specific.

In other words, there were four forms, one for each drug. The eight criteria were from the 1996 action plan for useful information, and the subcriteria as much as possible were based on approved labeling unless the committee felt that there was some reason to deviate from that or to add something that they were aware of that was evident in the professional literature.

The forms were revised until all panelists approved. So, as Sharlea would tell you, this was an iterative process. Materials were sent out. Panelists commented. Comments were incorporated. They were sent out again until the 16 individuals were comfortable with the forms.

Now, these are the criteria, and I won't go through them again since Tom, I think, did that quite nicely, but if you have questions about really any of these particular criteria, I would be glad to comment later on their significance from a consumer's perspective.

I've been doing research on consumer understanding of their medication regimens and patient adherence, as well as patient's perceptions of their medication since 1968. That's a confession. Patient adherence and education are my areas of research interest, and so I've paid some attention to this issue, and I'd be glad to comment.

I'll just give a few comments here on a few of them. If you take the first one as an example, you might say, well, why does the patient need to k now both generic and brand names. That's a perfectly reasonable question.

Well, one reason is that people go to multiple physicians. They may get multiple brands or they may get a generic from one physician and a brand from another. They look at these two names. They don't look like they're the same, and they end up taking both products. Sometimes people are too compliant.

So I think it's quite important, for example, that the patients ultimately learn both the generic and the brand name and so when the Keystone criteria cam out with this, I thought that's a good idea.

And we could go through each of these. If you look at number three, specific directions about how to use, monitor and get the most benefit, there are empirical studies to show that patients who get more specific information about how to take the medication are, in fact, more adherent. So I think this is something that consumers really need and benefit from.

Five, six, seven, and eight are, I think, all pretty obvious. I probably should comment about eight. This has always been a challenge to try to define this, but I think we've done considerably better this time around than we were last time because in the 1999 study, many of the criteria were lumped together, collapsed in a way that was hard to separate them out.

The forms in the latest study though are broken down in a very explicit way, and we did that, I think, for a couple of reasons.

One is so that ultimately the pharmacies, the pharmacy organizations, the vendors and other interested parties could see specifically where leaflets have strengths and limitations.

And, secondly, they could see how it is that these general criteria from Keystone were operationalized. So, for example, instead of just saying it's easy to read, we need some specific points under that to say, well, what makes it easy to read.

Well, the literature on aging and the literature on education of adults would say, for example, that putting information in bullets is helpful to the consumer. Using a certain font size is helpful to the consumer. Putting space between the lines is helpful to the consumer. Using headings and separating those headings from the text is helpful to the consumer.

On in this particular evaluation and rating form, we separated out and put each of those points and then tried to measure it as objectively as we could.

I should relate one story so that it's not totally serious here, but I normally, when doing research like this, like to do what I'm asking people to do so that I see how it's going. So I actually conducted one of the consumer evaluation groups myself, and it ended up that the consumer group that was rating the sheets were pretty typical of folks that use nitroglycerine, glyburide, et cetera. In other words, there were quite a number of widowed women in their late 70s and 80s.

And at one point as we were getting started, the woman said, "Would you mind if I could go home for a minute?"

Why would you want to -- "excuse me?" I said.

"I need to go home, just next door, and get my magnifying glass."

When you see something like this, it kind of comes home to you about why it is that things like font size space, et cetera, are important. So I probably come to this experience from watching consumers.

But those are the eight criteria. Now, the scoring method. Each criteria rate by four to ten subcriteria. Each subcriteria rated as to whether there was full, partial, or no adherence, and we wanted to separate that out rather than lump it together as it was done the last time.

Computer calculated the percentage of all points obtained, and we were aiming to get a scale from zero to 100 percent so that ultimately you could have a standard scale, if you will, for comparative purposes.

Now, is this the point that I said I was going to show the form? Now, the committee has seen this in the report, but this is primarily for anyone that's not seen the report.

How is that in the back of the room and for the committee, I guess, primarily? Can you see that?

Okay. You can basically see that the criterion of -- the first of the six criteria are whether the information is specifically specific, comprehensive, et cetera, for the patient to be able to use the medication, and the first criterion is that the leaflet includes the drug names and indications for use.

In this case it's for a atorvastatin, and you see that under that, this is the general criterion, and here at the subcriteria. In this case there is six subcriteria, and over here there are two little boxes.

If it was fully met, both boxes would be checked. One would be checked if it's partial adherence; blank if noon of it. Okay. So you basically have the possibility of six times two, 12 points for this particular criterion.

You'd go to the second one. In this particular situation it's contraindications and what to do if applicable. Do not take this medication if you are, and they list the three subcriteria, again, two boxes. In this case, tell your provider or pharmacist if any of these exist, two boxes, and so on.

Specific directions about how to use, monitor, and get the most benefit. The third criterion, and the items are subcriteria under that.

In this case, there are nine subcriteria. Again, two boxes for each. So there would be 18 points for this particular section. For example it's important to take this medication regularly or to help you remember, take it at the same time each day. If you miss a dose, take it as soon as possible, and then be more explicit about that. Do not take two doses at the same time.

Splitting it out enhances the reliability, that is, the agreement from one person to the next. What we're trying to do is to get criteria that are explicit enough so that if Expert A picks it up, goes off to this room, Expert B picks it up and goes to this room, they'll come back with the same score. It has to be reliable.

Precautions, fourth criterion.

Fifth criterion, symptoms of serious or adverse reactions and what to do. Notice how these are split out by what you should be telling, that is, how serious, and splitting out the serious ones from the less serious ones, and those that the patients can tolerate. Merely report it if it doesn't go away or bothers you versus those that are serious.

So the criteria are kind of interesting in the sense that they not only spell out the side effects, but something about what you should be doing, an action implication.

Now, the scoring categories, as you might guess when you've got so many points, you want to try to figure out how to represent and report this to audiences like this and to committees like this. So what we decided to do was you can report from zero to 100 percent, which tells you what the mean score was or what the individual score was, but it's also helpful sometimes to report levels so that when you see graphics, you can see, well, what percent fell at this high level, moderate level, low level, or very low level.

So we've categorized the information by these scoring categories here. That's level of adherence to criteria. See, Level 5 is the optimal or the highest, I should say, 80 to 100 percent, and Level 1, here, is only zero to 19 percent.

In other words, if a leaflet met 80 percent of the points possible, it would be in the fifth level of adherence. If it met zero to 19 percent of the points, it would fall into Level 1.

So when I show the graphs now, we will be referring back to this Level 1 through 5.

Now, before we went anywhere, we tested in a very usual and standard way and whether, in fact, the experts were able to agree after rating things independently. So each expert was assigned to one of four drug groups based on their expertise or experience.

They then were given a subset of the leaflets and asked to rate them independently, in other words, without talking with anyone, which is easy. They were at different places.

We then analyzed the results. The first time we did it, we found some issues especially with Criterion 7, which is on scientific accuracy. They tended to rate them as low on that criterion when they didn't present much information.

So we tried to clarify, no, this is really accuracy regardless of length or accuracy regardless of amount or accuracy regardless of content.

So we discussed that and then redid it again, and there is much better agreement. So we cleared up any problems or disagreements or confusion about the criteria so that things were clear, and we made corrections to the form so it was clear.

At that point then, the final reliability test, we got excellent reliability statistically speaking, and we've used other methods here, and I won't get into the technical side of that except to say that we've got good agreement.

Now, how were these leaflets actually rated? The process, once we got to the reliable form, each expert stayed in their one of four groups. This enhances reliability. So you basically have one group of four individuals rating all of the nitroglycerine sheets; one group in the glyburide group rating all of the glyburide, et cetera.

What this means though ultimately is that there are four experts rating the leaflets for a single pharmacy. Okay? Now, each leaflet though is rated only by one expert, and not duplicated once we got reliability.

In the end, experts rated 1,367 pharmacy generated leaflets. That's a lot of paper.

They also rated 31 manufactured generated nitroglycerine leaflets, and I'm not going to give the results on those today. They're in the final report though if you have an interest, and we certainly can talk about that later.

Now, let's go to the consumer evaluation form, one form with 12 items. It had to be fairly straightforward. We didn't have a drug specific form. We wanted a generic form because consumers were not rating scientific accuracy. Instead they were rating issues that can be rated by consumers and of interest to consumers.

The items, we tried to take and build on the 1996 action plan and to make it consistent with that, and we also had a pilot study that was done in 1999, Krass, et al. Basically that study tried to validate the items, and I can go into that a little bit later, but it might sidetrack us a little bit here if I go into that too much.

The most important point, I think, is that these 12 items were rated using semantic differential scale. That means that they're given words, and they're asked to rate the leaflet on a score of one to five for each one of these items, and these items go from poor to good.

So if it's very poor, they'll rate it a one. If it's very good, they'll give it a five, and in between, two, three four.

So 12 times the number of items means that, again, you can standardize this into the percentage of scores possible. So we, again, can either report item by item, or we can report the total possible points from zero to 100 percent.

Level of adherence. We're going to try to use the same for the consumers, but I'm also going to report some item-by-item findings because that can be kind of interesting.

To repeat, at the very high levels of adherence it's a five. At the very low, it's a one, and I'll show you the form. You can kind of see how this form is laid out.

Below is a list of words describing the attached information sheet. For each item, circle one number that best describes how you would feel if you were taking this medication for the first time and received this information sheet from the pharmacy.

And I'm going to go through these items in the slide shortly, but you'll see that there are a number of specific items, and then we have some overall assessments at the end.

Overall, what is your opinion about this information sheet? Please circle one number that best describes how you would feel if you received the information sheet. Hard to read, easy to read; hard to understand, easy to understand; not useful, useful.

Now, we've done some factor analysis for those of you that are statistics mavins, but I'm not going to go into that in detail, except to say that what we were trying to do was to get some items that measure legibility. How easy is it to read? And comprehensibility, how easy is it to understand, which are two different constructs or two different concepts, if you will.

The three specific legibility items are with regard to print size, print quality, and spacing between the lines. There's one overall item that relates to readability: how hard/easy is it to read?

Those items actually correlate quite well. The six comprehensibility items are listed up here. In this case it relates to how well the material is organized, its length. It can be either too short or too long.

Whether it's clear; whether it's perceived to be as helpful in a global sense. Completeness, is it incomplete/complete? And how easy is it to find important information? Now, that is if a person starts taking the medication and they keep the leaflet, they might want to go back to that leaflet and find that information again on side effects, for example, or the consumer can interpret this as they wish, but that might be the thought behind it.

The three summary items, again, reading, understanding, and useful.

Now, it doesn't make sense for the consumer to agree with each other because we've got different ages, different levels of education, et cetera, but it is important that the same individual will give you the same answer three or four or five days later, and that is called test/retest reliability. That is agreement from one testing to another.

So we did that, and we did some work before, but the final test was to take nine consumers who weren't affiliated with the project, ask them to independently rate 18 leaflets at two different sessions, and we got good test/retest reliability. And that's the point at which we said, "Okay. We're ready to go."

The consumer rating process was that we identified consumer facilitators in different parts of the country, usually people from pharmacy colleges because they have graduate student slaves who can go out and do things for you.

We had people in 11 states. We asked them to identify consumers towards the older range because we did not want to -- we wanted this, again, to be as realistic as possible and to at least be somewhat representative of the people who might be using these medications.

They recruited ultimately 154 consumers. They did it at senior centers, clinics, work organizations. Sometimes they met in church basements, apartment buildings, et cetera, different locations.

The process occurred in this way. The facilitator, after identifying a potential rater, arranged to meet with these individuals, eight to 15 individuals per session, and the reason for doing that was so that the facilitator could hand out a packet to each consumer, and the consumer then would open the packet. They would get instructions about how to do this, and then they would be asked to rate the materials in that packet.

In other words, this was not a focus group session where people are discussing this with each other. They're trying to do this independently. Each rater independently rated about ten leaflets.

Rater characteristics. Mean age, 61 years; range, 20 to 89. So we had quite a bit of variability, and you can see that the raters are probably consistent with what we know about medication users. Sixty-eight percent female; that is a little bit predominant females. Eighty-nine percent white, which is not totally representative of the U.S. population, but it does have some race/ethnic diversity in it.

Seventy-seven percent used medication daily. We asked them whether or not they used medications and how many.

And then we had some educational diversity. Now, I think we can talk about representativeness of this group later on, but I think probably the most important thing to say here is that when you look at national health statistics, surveys that are done of prescription medication users we find obviously that the percentage of individuals who use medication increases dramatically with age.

And by the time you are getting to the older age levels -- let me see if I can find some statistics that I wrote in the margin here.

If you look at those individuals under 18, about 21 percent use medication. Of course, we did not include children. They had to be 18 years or older.

In the 18 to 64 year range, 39 percent of the individuals nationwide used one or more prescription medications, and when you look at 65 or older, fully 74 percent use one or more medications.

So I think what we're trying to do here is to have raters that reflect the medication users.

Okay. Let's move on to the results, Part 1. The percentage of observers given any written information you can see in this table here. That is, for atenolol, close to 90; glyburide, 89; atorvastatin, 89; nitroglycerine, 88. In other words, it was pretty much the same across medication type.

Now, let's look at the expert ratings. All criteria combined for 1,367 leaflets. Remember we're going to look at the five levels up here. This is the lowest level, zero to 19 percent adherence. This is the highest level, 80 to 100 percent adherence.

Down this side we have percent, percent of leaflets actually hitting that level, and then down here at the axis, here I have the different drugs so that you can see whether or not there are any differences by drug.

This is the first for atenolol. Now, let's kind of look at this for a moment. What you see is, first off, there are no Level 5s here. Zero percent of the leaflets met the highest level. Twenty percent of the leaflets met Level 4. Fifty-six percent of the leaflets met Level 3. Ten percent of the leaflets met Level 2, and three percent of the leaflets met Level 1, according to the expert ratings.

It's interesting that these patterns or the trends are pretty similar for the glyburide and atorvastatin. A little bit better ratings for nitroglycerine, but you note again that there were no leaflets as distributed anyway that met the highest level.

Now, what we did next was to look at the individual criterion or criteria, one through eight you remember. The highest ratings overall were for scientific accuracy, criterion number seven.

The next highest ratings or moderate ratings were for number one and number three, that is, the names and indication and durations.

Low ratings though were observed for numbers five and six, that is, the side effects information, what to do, and general information.

And the lowest ratings were for Criteria 2, 4 and 8, that is, the risk information, contraindications, precautions, and legibility comprehensibility, according to the expert.

Now, what i've done in these next few slides here is to simply show graphics for one of the four drugs. I'm picking atenolol because it's the first one that we looked at before. Otherwise I think we'd get snowed with detail here, and I do this so that you can kind of see what the distribution is like for each one of these criteria.

You see immediately that the Criterion 7, which is scientific accuracy unbiased and up to date, was met very well in terms of 95 percent of the leaflets meeting Level 5, and that's pretty much the same for all drugs.

The moderate ratings though for Criteria 1 and 3 are seen in this slide, and you see that 32 percent met Level 5 and 11 percent met Level 4 for name and indications. That is, you begin to see the variability now of information here.

On directions you also see variability, 19 percent at Level 5, 47 percent at Level 4. If you put that together, that's about 63 percent meeting four or five, if you want to look at it that way, but there were some that were quite low here, too.

Now, in the low category, this is Criteria 5 and 6. This is for Criterion 5, adverse drug reactions, and in what to do, only about 13 percent met that criterion at the Level 5, and about 14 percent met it at Level 4. And these were in the pretty low category here, about 48 percent of them getting in the Level 1 or 2 on adverse drug reactions.

And general information, remember general information can include a variety of things, including the publisher name and date and encouragement of the patient to ask questions.

Criterion 2 and 4, now, these are the lowest ratings. You see the variability again, but on contraindications five percent meet this level. Twenty-seven percent meet this level, and these, of course, are at the low level. Seven and 14 percent of the leaflets meet the criteria at the higher levels for precautions.

Now, the lowest ratings were in the area of legibility, comprehensibility, and you see that according to the experts and the subcriteria that they laid out none of the leaflets that were obtained met Level 5. Eighteen percent met Level 4.

Now, if you want specific data on which of these subcriteria were met or not met there, those are in the tables, five to eight, and we can talk about that later.

Now, let's look at the consumer. Consumer ratings, all items, that is, all 12 items, and we have here, again, our levels, and here we have percentage of leaflets meeting a given level, and let's start with atenolol, as we did with the professionals.

Now, what you see here is a little bit higher rating by the consumer because you do have them rating 24 percent of them meeting this level, but if you look carefully at the professional, you'll also see that they have more at the low level and fewer in the middle, which is kind of interesting.

I think that's because readability is quite an issue with the consumers, and four of the 12 items for the consumers related to legibility. I should say legibility to be clear here.

So they're a little bit more positive in the sense of here, but they're more negative down here. So they, like the experts, are rating the leaflets as variable in quality.

You pretty much see the same trends for the other three drugs. Twenty percent of the glyburide are given the highest level; 28 percent to atorvastatin are given the highest level; and 29 percent of the leaflets for nitroglycerine are at this level.

It's quite remarkable, and you should see similarity pretty much if the standard format is being used within a pharmacy and with these different drugs. So you see that consistency as you did for the professionals.

Now, let's look at item by item because I think there's a little bit of interesting findings there. We look at scores varied by item, and if you look at the item, the lowest scores were for print size, print quality, spacing, and overall readability. Moderate to high scores were for easy to understand and useful, and I guess one way of summarizing these data, which I'll show you in a moment, is that 36 percent of all leaflets -- and consumers rated nearly 1,300 leaflets -- but 36 percent of all leaflets were given lower ratings on readability, that is, a Level 1 or 2.

Remember that they rated each item one to five. So now we're looking at the item to avoid any confusion. We're saying what percentage of the leaflets were given a one through five on this summary item.

This one is ease of reading. It was one of the bottom items or one of the three overall items, and you see that they considered that 20 percent were at level 5, that 19 percent were at Level 1 or poor, and 17 percent were at Level 2. That is, that's where I got the 36 percent.

Thirty-six percent of the leaflets were rated a one or two, that is, at the poor level, when we look at readability. In the final report -- and I think that the report sent out to the committee -- those tables were quite misaligned. So what I would encourage you to do is to go to the Web version where there's a PDF version file that will show these tables in detail if you wish further detail than what's there.

Now, in terms of ease of understanding, you notice here that the percentage of leaflets, that there aren't as many leaflets rated poorly by the consumer. They saw ease of understanding as being a little bit better than readability. About 19 percent were in the poor level there.

Usefulness, 17 percent were either a one or two in terms of usefulness, 30 percent at the high Level 5, 32 percent at the four level. In other words, they were clearly making distinctions between or among these leaflets.

Now, let's just say a few words about expert versus consumer ratings before we stop the presentation of the first part of results.

The question arises, you know, as to how the consumers' rating compares with the expert rating. I think it's difficult to do this because in a sense, the expert is rating items or concepts that are different than the consumer.

For example, we didn't ask the consumer to evaluate scientific accuracy. Okay? We didn't ask them to evaluate that particular criterion. So it's kind of like comparing apples and oranges a little bit, not completely, but a little bit.

So I caution us to be careful when comparing expert and consumer evaluations.

It's also important because remember of the 12 items for the consumer, four of them, three specific items and one overall item, pertain to legibility, and eight items refer to comprehensibility.

And so in a sense you have a different somewhat weighting for the consumers, weighted on those things that I think the Keystone and others would agree is more important for the consumer to evaluate.

With that said, let's look at an overview of what we found, and more analysis still needs to be done on this because this is a fairly new analysis.

Overall we found low correlation between the expert and the consumer, but I think it's important for me though to say that they were significantly related.

In fact, the behavioral scientist would probably say that's pretty interesting that they're related at all, but the correlations for total scores, that is, the total percent, zero to 100 percent for the expert rating and the consumer ratings were related in the .25 level, which is pretty low, but it's still related.

So I conclude that the two evaluators are bringing different things here. They're bringing a different perspective to the rating process, both being important.

When we look at the experts' total rating of usefulness and the consumer's rating of usefulness, we do see a significant association, and I'll show you a slide on that in a moment, but when you look at expert rating of usefulness with the consumer readability, that's where you see no association, and that to me is pretty expected.

That is, why should you expect that scientific accuracy is related to readability? It's kind of as you would expect.

This table shows you in one case the issue of consumer rating of usefulness, which is down this side, with the mean overall expert rating.

Now, I need to explain this a little bit. This means we're talking about an item here. The consumer rated the leaflet on a one to five score from poor to good, and I've got the aggregate item for the expert over here, and what you see is that for those leaflets that were rated poor in terms of poor usefulness by the consumer, the expert rating was only 40 percent adherence. Okay?

And if you look at the fifth level, that is, the good level, according to the consumer, you see 55 percent as the mean expert rating. That is, s the consumers and the experts kind of agreed here in a linear way on the poor to good items, higher for the leaflets identified as not to at all useful. The experts also gave them the lowest rating. That's what I would conclude form this.

That's a long way of getting around to that, isn't it? Okay. Let me try to just summarize a few key points here, and then I will stop for this part of the results.

First, I think that we found that 89 percent of the consumers nationwide in this particular study were given some sort of information. That information can range from one line or two lines to a page and a half, and I'll talk about length when we get to the next part of the results.

Secondly, I think we saw that both the expert and the consumer ratings vary by the criterion. For the experts, they were most critical -- experts were most critical on the contraindications precautions and legibility comprehensibility, and for the consumers, I think they were most critical of readability.

I think those are the main findings and perhaps the most obvious thing here is that they still vary quite a bit from one to the other.

Now, on the next section of findings, we're going to talk about this variability, and we're going to say, well, now, why the variability. Could it be due to the consumer background characteristics, age, education? And could it be due to pharmacy characteristics? Do they vary from chain independents?

And third and most importantly, do they vary by the leaflet characteristic? And I'll focus in that set of results on length, font size, and vendor, and version of leaflet within that.

So I'll stop there. Thank you.

CHAIRMAN GROSS: Bonnie, thank you very much.

At this particular point we'll take a break, and we will reconvene in 15 minutes.

Thank you.

CHAIRMAN GROSS: If I could get everybody's attention, I think it's time to reconvene the meeting. So we're ready for Men in Black, Part II. I mean the report of evaluation of written patient information penetration, and usefulness, Part 2.

Dr. Svarstad will begin.

DR. SVARSTAD: Thank you, everyone, who donated Hall's lozenges, Life Savers. I hope my voice -- I think it will hold out, but thank you for putting up with the coughs.

Okay. The second set of results. Let's start with leaflet ratings by consumer characteristics, and first off, what we basically tried to do here was determine whether gender, age, education, race, ethnicity, current drug use was in any way associated with high or low ratings. So we did a variety of statistical tests, and the first, easy conclusion was that ratings, consumer ratings, that is, were unrelated to gender, age, education, and current drug use.

One of four ratings for one of four drugs was related to race not in a strong way, but I report it nevertheless. White rates gave somewhat higher ratings than non-white raters for nitroglycerine leaflets, but race was unrelated to other ratings for the other drugs.

And for those of you that do statistics every day, you know that when you do a large number of comparisons, you're expecting that at least occasionally you will get a significant. So this one finding may simply reflect that we're doing many tests here. So I would not attach a great deal of significance to it.

So I think we conclude at least given the population of raters, that we didn't see that consumer ratings were correlated significantly with their background demographic characteristics.

Now, let's look at pharmacy type, and here I have a graphic showing leaflet distribution and overall ratings by experts and consumers by pharmacy type. This again pertains to over 1,300 leaflets, and the pharmacies are simply categorized independent versus chain, and these terms are debated within research circles. So knowing that it's kind of hard to make these categorizations, and this perhaps is multi-unit organization, but in any case for the sake of simplification here, I've put them in two categories.

And what we see is that the percentage of patients, shoppers who are given any kind of information, you know, a very partial, small piece of information or a full leaflet, did vary significantly by pharmacy type with the percentage of shoppers receiving information was 79 percent in the independent or smaller pharmacies and 98 percent in the chains. And that's significant at the .001 level or greater.

When you look at the overall mean ratings of the experts that this rates now from zero to 100 percent, you see that the mean rating for the professional of leaflets obtained from independents was 43, and the mean for the chain was 55. Again, it was significant, a significant difference.

Interestingly enough, the consumers also rated them significantly differently in the same direction. The mean consumer rating was 49 for these pharmacies and 65 for these pharmacies.

So we do see differences by pharmacy type, and I think that I have some slides a little bit later to comment about that further, but I should say that I think that this is not as simple as it seems, but it would appear that pharmacy versus chain are either using different systems or they're implementing them in different ways.

But I have some samples. I brought quite a few samples for you today so that you can kind of see some of this.

Okay. Now, let's go to length. We categorized the leaflets by length, by those that were under 5.6 inches in length, this size, you know, half of a standard sheet of paper, or 5.6 to 11 inches, this size paper or somewhere in between or over 11 inches, which would be a second page.

We did not find any three-page leaflets, and I unfortunately did not bring a slide on the actual percentage, but I can get that to you by this afternoon because I brought my computer, and I'll dig out the data, but I think the distribution is more towards the low end, and it was pretty rare for us to see two-page leaflets.

So what we have done is categorize the results for what these short leaflets, these middle range leaflets, and the longer leaflets, and we have the mean expert rating for each leaflet length controlling for drug here.

Now, what this shows is that the mean expert rating for these very short pieces of information I should call them is 44. For the ones at this level it's 56 and 57 for those that are at this level.

So I'm sure if you did a statistical test to compare these two, you would not find it significantly different, but it is between here and here. Okay.

See pretty much the same trend for each of the drugs. Higher expert ratings for the ones that are at this length here and quite low ratings for the shorter ones, and I've brought examples of leaflets in this range, as well as in this range. So I'll show you those later.

Let's look at the consumer ratings. The same approach. We're asking what the mean overall rating for the consumer is on a scale of zero to 100 percent for leaflets that varied by length.

Lower ratings for those that are shorter, and a little bit higher or significantly higher for those in the 5.6 or half page to a page and a half, or less rather.

We see similar trends here, but they're probably more marked than you saw for the expert, that is, the mean rating for the consumer is 50 percent here and 70 percent up here. That is, there is some tendency to give leaflets that were over here a higher rating.

Is that associated with length? Not necessarily because it could very well be that a leaflet that is somewhat longer has a different format and different font, and I'll get into that a little later.

In other words, a leaflet that's a full page long as opposed to a half page long may use bullets, may use different spacing, different font, which the consumers are reacting to. So it's important not to be misled that consumers are necessarily asking for or expecting long leaflets. That's not what these findings show at all.

Now, let's look at the issue of font size. We had staff do this. After asking staff to measure font size, we got these notes back saying, "Isn't there somebody else that could do a better job of this?"

So we did have staff do this and trying to do it in a consistent and standard way. So the results here, we're categorizing leaflets at smaller than ten point or ten point and greater in font size.

And you see that, first off, for atenolol the smaller font size leaflets did get lower ratings. Now, remember that this is an aggregate rating. So if you look at readability or print size, you're going to see much stronger results, but we're looking at overall aggregate rating right here.

Pretty much the same trend for all four drugs. In other words, consumers are giving lower ratings to those shorter leaflets.

Now, let's look at the rating of readability. If you remember, at the end of the consumer forum, we asked them overall how easy or hard was it to read on a scale of one to five in this particular case.

Here you see the one to five with five being the best and one being the poorest. Let's look at atenolol. You see a pretty marked difference in readability by this somewhat crude classification, and the same trend for the other drugs.

That is, on leaflets with small font size measured by staff independently, consumer ratings tend to be or are significantly lower, meaning there is a link here between the objective measurements and the consumer rating on those rating forms. And these are all significant at the .001 level or better.

Now, the next question was: does it vary by vendor? And Sharlea Leatherwood noted before, and I think as John Coster noted before, there's been merging and much activity in who is actually providing the information. It wasn't our role, and we didn't have the resources to really investigate this except to say that the data are consistent with the idea that there now are very few vendors evident.

One of the difficulties that we had in studying this was that the pharmacy did not always include the publisher information and the publication date, making it difficult to identify where the information came from.

So that was evident in only probably 55 percent of the leaflets. So what we actually did was we did obtain copies from one vendor, the main vendor, of their leaflets, and we went back and reanalyzed the data so that we could try to determine what percentage of the leaflets actually used data from that vendor and get a better estimate perhaps of how this vendor frequency or distribution looks in the pharmacies that we visited.

So we've classified them here into three categories, and I'll mention a fourth category. The first, the vendor was -- now we're looking at atenolol leaflets because we couldn't go back to 1,300 of them, and so I picked the drug class and focused on that one.

Vendor could not be ascertained at all in 46 cases or 13.5 percent. The Vendor 1, could be identified by comparing what we saw on the sheet to information we got from Vendor 1. I say partial message because in that particular case the pharmacy was printing off either the patient counseling message or the warning label message.

The vendor has shorter messages that are available that I think are certainly not intended to be the full monograph. They're called patient counseling message or warning message. They're very short, and I'll show you some examples.

We found those being used in five percent of the cases or 17 cases. In the remaining cases are 81.5 percent of the cases, 277 atenolol leaflets. We found that Vendor 1 was the vendor, data vendor.

Now, that reflects this state of affairs in the year that we collected it.

Now, because we kind of anticipated that there would not be that much variability, and we did have some interest in what kinds of information are provided in hospitals and institutions, we added information from a second vendor to the ratings by experts and consumers, but we are not including those data in the main report because they're not from community pharmacies. It's what I would call comparison leaflets.

And we put institutional here because these leaflets are, as I understand it, primarily distributed in hospitals, out-patient pharmacies, out-patient situations or in-patient.

But in the tables then, you'll see Vendor 1, partial; Vendor 1, full; or Vendor 2 with this little asterisk, meaning that's a comparison leaflet, and I'll show you examples later.

Okay. Now, the first thing that we've found that's pretty obvious to the data vendors, I'm sure, and to everyone here perhaps, and that is that maybe it's not so obvious, but making that kind of split-out by Vendor 1, partial message; Vendor 1, full message, does account somewhat for these leaflets that are extremely short.

For the leaflets where we could not identify the vendor, 83 percent of them were under five inches. Think about that for a moment. For the leaflets where we could not identify the vendor, 83 percent were under 5.6 inches. They fit within this piece of paper.

The partial messages, overwhelmingly 94 percent were of the short variety. When you got to the full message, you had about 27 percent still being at this level of shortness, and I think that these -- and I'll give you an example here -- I think those are primarily the ones where you've got a full leaflet compressed into a half page using a font size that even I can't -- you know, that's hard to read, but I'll show you those examples. So you have several things happening here.

The Vendor 2 leaflet takes up a full page and about this much of the next page, and it compares with Vendor 1, full message leaflets that are slightly more than one page. Okay. So we do have variability on length.

Now, let's look at the variability by date of vendor in terms of ratings by experts and consumers. First off, we know that there is substantial variability by date of vendor with the highest being Vendor 1 and this comparison Vendor 2, and the lowest being those partial messages and unidentified vendors.

This is the vendor not identifiable. This is the Vendor 1 where it's partial. That is, the pharmacy has only printed out the patient counseling message, not the full monograph.

And this is Vendor 1 with a full monograph, and this is the comparison two, comparison leaflet called Vendor 2. I should have put motion on here so you could just see one drug at a time because this is kind of information overload here, but let's stick with our first one so that we don't get too overloaded.

These are the mean expert ratings for each vendor type. You see the mean rating for the experts was 32 for this unidentified vendors and 28 for these partial messages, and then it jumps up to 56 percent for the full leaflet from Vendor 1 as printed out by the pharmacy, I should always say, because the pharmacy can influence how it's printed out.

And then the comparison leaflet is this bar here. So it reached the 75 percent for that one.

Pretty much the same here, although it's not quite the same trend, which is why I give both drugs.

And the third one is pretty similar to this, and you see, again, that the full leaflets, whether they're from Vendor 1 or 2, are rated much more highly by experts than these unidentified leaflets or these partial leaflets.

So vendor certainly has more influence than consumer characteristics and even pharmacy type.

Now, let's look at by criterion. Here's Criterion 1, name or indication, and you see here that these short, unidentified vendor and short messages are weighted quite low, a mean of 32 and 28.

It jumps up to 56 and 75 for Vendor 1 and 2. You see a similar pattern here and this is contraindications. Now, this is why it's very important to look at vendor, because this is much different than if you lumped all of these together.

It suggests that if these pharmacies here that are using Vendor 1 system and have full access to the full monograph, the decision to use only partial is the one that's kind of influencing the rating there. Do you see that?

Now, when you get to directions, they're very low here and somewhat moderate. Precautions are very low here and neither one of them is meeting criteria on cautions as fully as the criteria would require.

When you look at adverse drug reactions and what to do, you see low ratings, again, for these shorter, partial messages, and you see somewhat better for the full leaflet, and you see very high rating for the Vendor 2.

General information, low here and kind of low-moderate for both of these.

Accuracy, it's interesting that accuracy is pretty high overall, which was reflected in the earlier findings, but it is kind of interesting. I don't recall the statistical findings to know whether these -- you know, why that dips down a little bit, but the overall message is that accuracy is not the issue there. It's completeness of the information, specificity of the information, and those other characteristics of the content.

Now, if you look, this is, I think, somewhat interesting. According to the experts and the expert criteria, the unidentified vendor, the vendor partial, and the vendor full, all had about the same ratings on legibility and comprehensibility. Vendor 2 had much higher ratings on legibility and comprehensibility and had a mean score of 83, and you'll see why this is when I show you the example.

So to conclude here, the ratings by vendor are, I think, quite interesting, and they show that there are significant differences between vendor and within vendor as to how it's implemented.

And finally, I think it suggests that it is possible to get much higher ratings if you look at these leaflets that are being distributed in the institution because they do, in fact, get higher ratings on these criteria, but not all criteria. So it's not as simple.

Now, let's look at the consumer. Consumers also rated the unidentified leaflets and the partial leaflets at lower levels, and they rated Vendor 1, full monograph, is higher than Vendor 1, partial message, and they gave higher ratings to the Vendor 2 on atenolol, glyburide, atorvastatin, and nitroglycerine.

So you see somewhat the same patterns across all four drugs with lower ratings being given for the first two, moderate to variable ratings for Vendor 1, full message, and higher ratings for the Vendor 2 comparison leaflets.

Now, this is by these three items that are at the end of the consumer form: easy to read, easy to understand, and useful, with one being poor and five being the best, and this is where I think the form kind of comes through as being pretty sensitive in the sense that you see that the short messages and the full message -- this here, they seem somewhat hard to read on these partial messages for some reason, but overall the unidentified Vendor 1 receive moderate scores on this readability issues.

And of course, that readability, the print size, print quality, and spacing. It's somewhat a function of the vendor because the vendor can influence that, but it's also influenced, as I aid before, by the pharmacy.

And much higher on easy to read. The 4.7, almost a five. Similar over here and similar over here.

So we conclude that both in terms of the overall aggregate rating, as well as individual ratings by item the consumers are giving lower ratings to the unidentified vendors and lower ratings to pharmacies that use only partial messages from Vendor 1.

Now, data vendor by pharmacy. You see something kind of interesting here. I just did this a few days ago, but you see that the use of vendor does vary by pharmacy type.

With Vendor 1, full leaflet being more likely to be used in chain pharmacies and somewhat lower in independent pharmacies, the unidentified vendors are more likely to occur in the independent pharmacies than they are in the chain pharmacies, as is this partial implementation of Vendor 1.

Now, let's take a look at the Vendor 1 leaflets to see if there are differences in versions and also to determine whether or not the pharmacy organization or the software company that does that information, whether there are changes to the database, either additions or deletions.

To do this we did a subanalysis. We analyzed full leaflets used by 16 pharmacy organizations. We actually analyzed 155 leaflets from those organizations.

It's just kind of interesting to see that five of the organizations used what we call version one. Five organizations used version two, and six organizations used version three.

Now, you may say, "Well, why are there these three versions?"

Well, remember, as previous speakers noted, there were several database vendors prior to this study and those during the period of study were identified by one data vendor.

So you've got several databases here that are being maintained by a single vendor, and I'm not the person to ask how that will be working or how that works or where it's going in the future. I suspect that we've got people in the audience and around the table that can comment on that better than I, except to say that there were, we found, three basically different versions, and with the help of vendor one who sent us prototypes.

We then compared the actual leaflet against the prototype, which would then tell us whether or not sections had been eliminated by the pharmacy organization or their software vendor. That was what we were trying to ask.

The results are as follows. The overall ratings do vary somewhat by leaflet version, just that basic question, as they're implemented in practice. No prototype fully met the criteria, and some organizations did, in fact, add or delete information from the prototype, but not to a great deal.

There are small sections that are omitted or small sections that are added, but we did not find wholesale editing, at least for the material from these 16 organizations.

Whether the unidentified vendors obtain information from Vendor 1 and make alterations, we cannot necessarily say because we couldn't identify what their sources was.

Now I've probably totally confused you, but I hope that's clear.

Expert ratings by leaflet version. So we have Version 1, Version 2, Version 3. One of these versions has five sections to it. Another has eight sections to it usually.

And without getting into it in great detail, if you look at the prototype, they do vary when you look at them a little bit. So what we're now looking at is, well, what do the expert ratings show for these different versions, all maintained by the same vendor?

What you see is kind of what I just said. When you look at name and indication for use, these two versions meet the criteria quite well because their mean rating was 83 and 85 percent of the points on this criteria. But this version, leaflet Version 3, does not.

On the other hand, when you get to contraindications, Versions 1 and 2 fall down. That's loose language. One and two have lower ratings. Version 3 has a relatively high rating.

There is not much difference when you look at directions or criterion 3. All versions are pretty close, although Version 2 and 3 are somewhat higher.

Precautions. There's not a great deal of variability here, but it is significant, but when you look at the figures you say, well, these two are a little bit higher than this one and this one specifically, but you don't see the kind of variability that you see here.

If we look at the side effects information, you see that leaflet one has a much lower rating, 36 percent of the criteria met versus 53 percent for Version 3. In other words, this leaflet does not meet criteria as well as this one does.

And you see quite a bit of difference here on general information. This version included much of the required information on this criteria, whereas these two versions did not.

You see that they all, again, are pretty good or very good -- excuse me -- on accuracy and legitimacy -- legibility and comprehensibility are pretty comparable here.

Okay. Now, in other words, leaflet version did tell you something, and you need to look at the individual criteria, and it's kind of interesting because what happens then is that since one version meets some criteria and not the other, they kind of come together with similar scores, but they don't have similar scores when you look at individual criteria.

Okay. Now we looked at additions and deletions from the prototypes. This is Version 1. This is Version 2. This is Version 3. I had a pharmacist graduate student go through and compare the leaflets for each of these organizations against the prototype and tell me exactly how they differed, and then I did the same only for a smaller number, but to verify this.

And what we found was kind of interesting. On Version 1 we found no editing of -- well, I should tell you what we did find. One out of five organizations using this version deleted the publisher and disclaimer information, and those were the only deletions that we found for that version.

And two gave out the full monograph, plus the label, which doesn't really have anything to do with additions/deletions. It just tells you how the pharmacy is implementing that version.

You assume then or you can conclude then that four out of the five organizations made no changes in the leaflet and that none of them changed the content of the section within it. That is, they didn't start tinkering with the side effects or the contraindications or whatever.

Version 2, one organization deleted the additional information section and added label. In other words, this was the only change in

Version 2.

For Version 3, this was a little bit more complex because I suppose that this particular version may or may not -- I think it potentially is more changeable because lines and sections are marked with text markers so that you could take sections out, but as several individuals have noted, licensing agreements are supposed to cover some of this, and that's not my area. All I can say is what happened from our sample.

Version 3, five organizations deleted the warning box, which I'm not sure if this is really required or whether it's an optional, but they deleted the atenolol warning box or warning section at the very beginning of the leaflet. That's not to say that they eliminated any warning about atenolol. It's just that whatever warning was there was somewhere else. It may or may not have been the full warning required, but it certainly wasn't highlighted for the patient by separating it out in some way even though the prototype did separate it out.

Version 3, five organizations also eliminated the overdose section completely, just eliminated. And four deleted the drug names and notes. Now, that might vary somewhat by drug, but five organizations added the disclaimer about this information doesn't include all uses, side effects, drug interactions, et cetera. So they added some information.

So what do you conclude from this? Largely or generally, there is not an editing within a section. In some cases for some versions there is a removal of sections.

Ratings of the distributed versus the prototype leaflets, the example of atenolol. We've got the distributed leaflets that we collected versus the prototype that we obtained from Vendor 1, and what you see generally is pretty close ratings here, except when you get to here and to some extent here. You can kind of see that.

On number two, contraindications. Number four is precautions. That's where the atenolol thing might come in so that this might reflect some variability between the distributed and the prototype because there may be some -- let's see now here. Just a minute. Let me look at this.

This one is pretty much the same, and we can't do statistical tests because there's only one prototype. Nothing makes sense.

Overall it's pretty close here that even though a few organizations eliminated a warning box, it reflects the fact that somewhere else in the document the warning was included. Okay? That's what I conclude form this, and that's just Version 1.

And actually Version 1 did not have a warning box. So I'm kind of talking out loud here.

Version 2, you see they are also quite similar, but there are some discrepancies. You see that, for example, the prototype is a little bit better here and here than the actual distributed. And if I went back and I compared which organizations I could pinpoint for you -- and we kind of did this, but I don't want to identify pharmacy organizations here. Nothing is served as I see it by that.

But what was interesting is that we could pretty much identify where the low ratings were likely to occur based on what we knew about changes in the prototype.

A little bit more here now on Version 2. Remember I said earlier that this is the version that has the text markers. So it may be easier for these corporations or software vendors to remove certain sections, and this is where there were more changes.

And you see more differences between the prototype and the actual. For example, here, here, here. Five is side effects, and this would be the outcome of a number of organizations eliminating the overdose section, for example.

So what do you conclude? There's more changing with some versions than others, and when the changes do occur, they do seem to reflect the ratings for certain criteria in predictable ways.

I think we've reached now the conclusions, and I'm going to give these conclusions and then I would appreciate switching over to the examples, and then that will be concluding this second set.

The conclusions then are that, first, highest ratings have been for scientific accuracy and being nonprofessional without a doubt. The lowest ratings are for information about contraindications and precautions.

Third, the lowest ratings are for leaflets that are extremely short, less than five pages long or have a font size that's extremely small.

The lowest ratings are also for leaflets from independent pharmacies and unidentified vendors.

Finally, there is no prototype that fully met all eight criteria, and under that, experts and consumers were both critical of legibility.

What is the conclusion here? It is that pharmacy organizations can influence the ratings by first selecting the vendor and the leaflet version from that vendor, however that plays itself out.

Secondly, they can influence it by modifying the leaflets themselves, at least those versions that are modifiable. Now, licensing agreements, I can't really speak to that.

The third bullet that I should have had here is that pharmacy organizations can influence legibility by influencing print size, print quality, and readability, the font size.

So I think that there are some areas that could be improved.

Now, let me now shift to the final step here, which is to show some examples and with each example, I will give you what the mean expert rating was for that sample.

Some of those have been distributed to the committee, I believe, and I am not going to go through all of those. I'm just going to go through a few of them. So would you help me?

Thank you.

Partly a function of the font. This is the case from Pharmacy 313, and that was all the information that the patient got. "Do not stop med. abruptly," and then it was repeated. "Do not stop med. abruptly."

This is not the auxiliary label on the bottle. This is the information that was on a piece of paper that the patient was able to take.

The expert rating on that was 16. Curiously enough, the consumer rating was also 16. So I think they agreed.

Here's another one that would be either a partial message or certainly an unidentified vendor. This is take with/after food or milk. Do not stop med. abruptly.

One wonders whether this is coming from the same vendor since the same -- or it could be coming from a pharmacy that's somehow -- well, I don't know. It's not identified.

Please remember some doctor offices require 24 hours' notice on refills.

The mean on that was 16.

(Pause in proceedings.)

DR. SVARSTAD: I should note that the patient names on here are fake names. They are fake names, and this was done for a number of reasons. And I hope that there aren't any physician names, but these are physician consultants that did this, and we've certainly tried to remove any other names. But I will try to pay some attention here to make sure.

Yeah, I just want you to know that the patient names are not real.

Okay. Thank you for reminding me.

I'm going to cover up even the fake names, if I can. Here is from Vendor 1, and I'm sorry you can't read the details of this, but you see that this is, I think, as I recall, this is from Version 1, and you see a number of sections there.

You don't see publisher, but you do see other information about how to refill, go to the Internet, et cetera.

The mean on that was a 51.

This is another example. The mean rating on this one was a 41. You see kind of a question-answer -- boy, I wish I could get that better. Why am I taking this drug? To treat heart and/or blood pressure problems. How should I take it? Are there any side effects? How do I store this? If I should miss a dose? What about generics?

The how should I take it: follow M.D. directions. Do not miss doses, and do not suddenly stop taking this without M.D. okay. Tell M.D. of other drugs you use/illnesses you have/allergies/if pregnant.

The slashes are a little hard to follow.

Are there any side effects? Very unlikely, but report cold hands/feet, swollen hands/feet, mental changes, bruising, bleeding, weakness, trouble breathing.

This is an example of a patient counseling message that was printed off of Vendor 1. It's exactly word to word from Vendor 1 rather than the full monograph from Vendor 1.

Follow directions, period. Do not stop without doctor approval. May cause drowsiness/dizziness. Drive with caution. Notify your doctor if you intend to become pregnant. Check with doctor before taking other medicine. Promptly report unusual symptoms, effects to doctor. Inform doctor/dentist prior to any surgery.

This received a 27, which would put it in Level 2.

I can't get this to work as well as I would like, but I wanted to show this one as an illustration of the font size. This is the leaflet, and I would estimate that the content is maybe three to four inches, and I don't have the data file with me, but the font size is extremely small.

So this would be an example and is an example of Vendor 1 material that's been compressed down to a very small font size, but it's colorful. But the content would receive about the same score as the other content would through that version, except on legibility.

Here is another one. You can see the difference on font, but the similarity in information. You see the familiar structure, common uses, how to use this medicine, cautions, possible side effects, before using this medicine, and overdose. This is Version 3.

Yes, that's Version 3, but you see in the caution section there, if you could read it, which you probably can't, it starts out by saying, "Do not stop taking this medicine without checking with your doctor."

That would be considered partially adherent because it talks about do not stop suddenly, but it doesn't talk about the potential need for gradual dose reduction and it does not take that material and put it up front at the top.

And actually this version from the vendor, it was up at the top, but that was removed.

Okay. You also see something characteristic about the information from this vendor or these pharmacies that I'll note here. Notice how the cautions -- that long paragraph. There are no bullets. The material kind of runs together.

Now, from a consumer perspective, that's hard to read, and even the experts, they would send me back this note, "I can't find the information." And I suppose they were trying to find it quickly, but sometimes they had to read through a leaflet two or three times to make sure they found the information or gave the leaflet a chance.

But I'll show you a little bit differently how that kind of information could be reorganized or at least presented in a way that meets the Keystone criteria because the Keystone criteria would say there should be bullets. The Keystone criteria would say there should be more spacing between the lines. The Keystone criteria would -- font size in this case is okay probably. I'm guessing.

Now, this is one where -- here's the black box warning up at the top, and that was in the original prototype from the vendor. This particular pharmacy organization kept it in. The others had taken it out.

And you also see that this one includes the overdose section, and a number of the other organizations had eliminated that. This was in the prototype.

This leaflet actually was one of the highest rated leaflets, but still only got a 61 percent probably because a little bit more legible, but it was, I think, Version 1. No, actually it's Version 3.

You see that while it doesn't have the warning box up there, it does have the overdose information. It does have additional information, and it includes the vendor publication date, et cetera. So if you wanted to trace it, you could.

Now, I've just got a couple more and then I'm going to finish. Here is one of the few leaflets that we saw that were two pages. I'm not sure. This document camera doesn't like it for some reason.

The point, I guess is that -- I can't seem to get it to work properly.

This is the first page, and the first page shows -- and this is from Vendor 1, common uses, how to use this medication, cautions, and then the second page shows possible side effects, the publisher or the vendor -- excuse me -- the vendor date, publication date, and the disclaimer. But it's still only rated a 57 probably because print quality was poor.

Now, I'm afraid that the bottom line here is that we did not see a lot of two-page leaflets. They're all rather short here, and if anything, the experts pointed out that there was for some organizations more information presented on the backs of the sheet, et cetera. And you won't be able to see this very well, but for this organization, you have the side -- it's a fold-out, and on one page you see your natural vitamin center, your thoughts please, quick tips to relieve small stresses. I use "stressed out." Some nutrition information and I think kind of interesting, health hotlines, and this is the drug information, prescription information. That one received a 45.

And here is an example and the final example. This is from the Vendor 2 comparison sheet, which we did not edit this or change it in any way. We just printed it off the Web at our institution.

And you see here this had a mean rating of 75 percent from the professionals, and the consumers gave it a mean of 97 percent.

Often in the open ended, the consumers would say this is great or this is the best, but I think that what they were probably responding to was the very different format here.

You see that there is quite a bit of white space. You see that headings are on separate lines as the Keystone criteria had recommended. You see that bullets are used to separate information as the Keystone criteria had suggested, and you see a font size that's consistent with what the Keystone criteria suggested.

So I think it's rather interesting that both the experts and the consumers rated this more highly. Now, a practical question is: could this information be reduced to one page? And you know, those questions, I think we have not tried to bring it down to one page, but I do think it would be possible. I put it up there as a comparison, not something that's actually being given.

Okay. So that ends my presentation of the second part of results, and I hope it has been clear. Thank you.

CHAIRMAN GROSS: Thank you very much, Bonnie. It's a fascinating study and a tremendous amount of useful information.

Are there any questions? Yes, Arthur.

MR. LEVIN: Yeah, I sort of have a problem in understanding where the eight criteria are derived from because one of them, which happens to be one that gets high marks when a lot of other things don't, is a criterion in the Keystone report, and the others are components of what is useful information, and they're sort of different. I mean, they're a little bit of apples and oranges.

And the reason I'm concerned is because the sort of high marks of scientifically accurate, nonbiased, non-promotional sort of may give people hope that we're actually making progress when I think the results of this study tell us that we're not making any progress and things are pretty dismal 34 years later.

And the reason I'm concerned is I don't know how something is scientifically accurate if it doesn't follow the definition in the Keystone report, which is information consistent with or derived from FDA approved labeling, and if it fails to meet some of these component requirements, it's not following the label.

In other words, if you leave out a contraindication that's in the label, then how is this scientifically accurate?

So I have a problem, a disconnect between the raters giving, you know, an average 90 percent compliance ratings to that particular criteria, and then low marks to its constituent parts. I don't know how you get from that low mark of constituent parts to a 90 percent.

And to me it's very important to sort of tease this out because if we didn't have that 90 percent, I think we'd all say this is just totally dismal, and the 90 percent sort of says, "Well, there's some progress. And I don't think it's real. I think it's illusory, and I think it comes about by confusing what was called a criteria in the Keystone report, making that one of eight criteria when the other seven are sort of components, as described in that report.

CHAIRMAN GROSS: Bonnie, do you want to comment?

DR. SVARSTAD: My reading of the Keystone report, but I wasn't on the Keystone Committee, but my reading of the Keystone report was that the Keystone -- that as a committee you were trying to identify the criteria that would be included in useful, and that scientific accuracy was one component of useful. And that's why they're separated out as they were.

We all had somewhat difficulty interpreting the Keystone criteria on accuracy, nonpromotional, et cetera, and I think that what the panel was trying to do here was to separate the concepts or the constructs of completeness or specificity or legibility and accuracy.

You can be accurate in what you say, but incomplete. But if you define useful as accurate and accurate by some other criteria, then you would, of course, get confused. But I think I don't have my copy of the Keystone Committee report here, but it did, it seem to me, separate out these different criteria, and that's what the panel was trying to get at.

I think when you look, for example, at these "do not stop medication abruptly," you know, that's an extreme. Now, is that accurate? Well, that statement is accurate, but is it complete? No. Is it specific? No. Is it legible? You'd have to look at the thing to see if it's legible. Is it comprehensible? Probably.

You can have something short and incomplete, but still be quite readable and still quite understandable and, according to the panel, still accurate for that statement. That's, I think, how the panel proceeded.

CHAIRMAN GROSS: Okay. Ruth.

DR. DAY: I'd like to thank Dr. Svarstad and all of her collaborators on this project. It is most useful and very thorough, and I'd like to just note there have been a tremendous number of changes since the interim study.

The inclusion of vendor analyses and the consumer panel and changes in a lot of the methodology has really been terrific.

There's one part that still bothers me. It's not easy to solve, but I would like clarification about how some of the data were then collected. Sometimes there are multiple idea units in a given line item, which is a subcriterion. For example, for glyburide on Criterion 5.5, which is adverse events, it says allergic reaction.

Under the allergic reaction it says fevers, chills, rash, and trouble breathing. So if I were one of your expert raters on the panel and only one of those was present, I suppose I'd give it a partial. If two were present, I'd give it a partial, and so on and so forth.

So every time that a given criterion only gets one point as opposed to two, there could be different reasons for that. One out of two is missing or three out of four is missing and so on.

And furthermore, there might be different criteria that the raters use to decide on partial credit. So could you tell us a little bit about what the instructions were to the raters? Because that gets to the guts of what the data are that you get to begin with.

DR. SVARSTAD: Right. Certainly, the side effects section or Criterion 5 is the hardest one with regard to that. Why is that hard? You mentioned a number of side effects. Well, the other problem is that there are many different ways to word that,a nd if we think we're trying to arrive only at one wording, we'd never reach full adherence.

So I directed the panel to help in clarifying this, and in most cases -- and I'm not sure whether you have got the version that the panelists actually used, but it spelled out that you have to list two of these four to be considered fully adherent. You have to list one of these three to be considered fully adherent.

So this was spelled out quite carefully, yeah.

DR. DAY: All right, and just one other question. You didn't get a chance today to tell about the readability analyses that were done by objective methods using the Gunning Fog Index.

DR. SVARSTAD: Yeah.

DR. DAY: And it's one of many. And I did note in the full report that you said that you did that analysis on the section we started out about how to use or take the medication.

DR. SVARSTAD: Right.

DR. DAY: And you had to choose something. Why did you choose that, in particular?

DR. SVARSTAD: We chose that one -- I think that's a very good question, Ruth -- because where -- and the experts will tell you this, too -- where you start and should you sample and so forth. We started there because generally that was the first section that the consumer was confronted with when reading these that had full sentences and that really would be considered helpful or useful to the use of the medication.

Now, the logic would go a little bit like this. If it starts by being unreadable or overly complex or overly long words, long sentences, et cetera, that you lose the consumer there because most people start at the beginning. They don't start at the end.

Now, with that said, if you went to the side effects section and you started doing a readability assessment, you may find a different result, and I think further analysis certainly would be possible.

I think that as you know, Ruth, there's a lot of difference of opinion among experts about using any of these readability scales for medical material because, you know, you in a sense have difficulty translating certain side effects into common language without losing the information. So that's why we started there.

DR. DAY: Well, I think that's good rationale, and it's a good first start at all of this. We've been doing readability analyses on TV ads and Internet and pharmacy leaflets and the PI, the approved labeling, and we get systematic differences in readability as a function of the content areas.

DR. SVARSTAD: Yes.

DR. DAY: It's kind of interesting.

CHAIRMAN GROSS: Yes, Jackie.

DR. GARDNER: Bonnie, since our function is risk management and communicating risk, I'm interested in what the consumers had input into in your study, and it feels as if we consistently get poor results in the areas of high concern to us, which would be precautions, contraindications, adverse effects, and so on.

And yet it isn't clear to me that the consumers were asked specifically about how important they thought this was or how well -- readability? Maybe it was in there. I don't know.

And my question, I guess, related to that is: did you have information about the consumers, about whether they were taking any of these drugs they were evaluating, and could there be a sub-analysis according to whether usefulness was different between people who had some experience with the drug and knew what they thought was important versus people who were just trying to read a document to evaluate it?

DR. SVARSTAD: That's a good question, Jackie. We went around and around about how to handle the folks that you would anticipate had used the medication.

The facilitators were requested to go through and ask anyone to identify -- on the background information sheet they had to list the drugs that they had used before. They were actually asked whether they had used any of the four drugs and if so, which ones.

If they had used it, they were not generally asked to evaluate it. Why? Because these are new users. We were trying to generalize to new users, not former users or current users, et cetera.

It's still an interesting question, and I'm sure we probably, if we looked very carefully, we probably have some that slipped in there, you know, that have already used it, but it an interesting other kind of study that one could easily do with the leaflets.

DR. GARDNER: Then I guess my question would be in a more global --

DR. SVARSTAD: But these were quite experienced consumers because 77 percent of them were taking one or more med. on a chronic basis.

DR. GARDNER: Yeah. My bigger question then would be in your opinion, knowing what you know then about your study, do you feel that consumers have had any input into the results related to issues of safety and risk.

DR. SVARSTAD: Yes. We did ask them about completeness, and you might say, well, we didn't go in and ask them why do you think it's incomplete, but we did this primarily based on a pilot study where we did go in and ask them, you know, please rate the amount of information on each of these topics, and we listed out the topics.

We did that in the 1999 pilot study. My impression was that this global assessment of completeness, helpfulness, usefulness was tapping into the issue of whether there's enough information.

Now, can we say precisely which aspects of communication are missing? No. We were kind of, I think, based with the question of how you do that with a large sample like this.

But I do not want people to take away from this that consumers were only concerned about readability because the data actually show that they were quite critical of these other components. I just haven't presented them here today.

CHAIRMAN GROSS: Dr. Brian Strom.

DR. STROM: Bonnie, this is very impressive and a very important body of work. One of the things that's clear is the dramatic disconnect between the proportion of patients who get material and the quality of the material they're getting, and yet your Vendor 2 data indicate it is possible to do it right.

Can you explore with us again where the Vendor 2 data came from? Who was generating those?

DR. SVARSTAD: It came from one of the -- well, without identifying the vendor -- do you want me to identify the vendor? Are you asking me to do that?

I'm not sure. This is a well known vendor.

DR. STROM: So it is one of the commercial vendors?

DR. SVARSTAD: Yes, it's a very well known vendor, and it was mentioned by a previous speaker.

(Laughter.)

DR. GARDNER: But predominantly it came from an institution. Didn't you say it came from an institutional --

DR. SVARSTAD: Well, this vendor -- we did not see any sheets from this vendor in the community pharmacies. You know, an offhanded comment on my part is that my impression is that Vendor 1 has the predominant provider of community pharmacies and that Vendor 2 may be focused largely on institutional.

And it's not exact. When I say it's possible, I think it's possible to adhere to the Keystone criteria in an efficient way. I don't think it's simple to simply say, "Oh, well, go to Vendor 2 and buy their database," because we're still trying to integrate databases here.

I think what's happened in the U.S. is very interesting, and that is that you've got information now being distributed out there. It's computerized, and having it linked to the dispensing system is critical to its implementation.

If you go to Australia, it's not integrated. There are separate databases, and adherence is very low. Distribution rate is very low.

CHAIRMAN GROSS: A general question for my information. Has anyone asked Vendor 1 and 2 if they're aware of the Keystone criteria and if they are, why they chose -- why the ones who didn't use it chose not to?

Is that known?

DR. SVARSTAD: I would not want to speak for Vendor 1 or Vendor 2. I haven't really asked them.

I'm sure that there are -- well, I think that if you can see these different versions here, the complexity of this particular study was that several vendors were -- material from several vendors were maintained by a single vendor this time around, and that's part of it.

How Vendor 1 feels about different criteria and what would be needed to implement all of the criteria, I think, is something that you'd have to ask them about, but you know, there are a lot of products. So I think that if you were to, for example, implement the criteria about legibility, it would mean reformatting the information so that it's not all lumped together, et cetera.

CHAIRMAN GROSS: I guess my concern is if this is a voluntary system and we're relying on goodwill, it would be interesting to find out what the attitudes of the data vendors are about the Keystone criteria.

Arthur?

MR. LEVIN: Just a point of information. Several data vendors were part of the Keystone process, and certainly one of them that has been mentioned by other speakers today as a major player was at the table.

So for them not to be aware of, you know, what that process was and what the conclusions and summary, I have to believe that they know exactly what the criteria are.

CHAIRMAN GROSS: Okay. Stephanie.

DR. CRAWFORD: Thank you.

Bonnie, I wish to echo the compliments expressed to you and your collaborators previously with respect to the insightfulness and comprehension of this report or comprehensiveness.

I especially applaud the efforts to include the consumer ratings, but as my students know, some always is going to fall on my "but."

With the consumer panel being 89 percent white and 54 percent education behind high school, I did question the representativeness of it with respect to medication users in consideration of the very high prevalence of heart disease, high blood pressure, high cholesterol, diabetes among African American and Latino populations.

You did try to address it in looking at and said that the race of consumers was largely unrelated to their ratings, at least when they were dichotomized. That's why there's non-white, but as we know, there are problems with the sample size and lumping all of the groups together.

So for this report I only ask that that be kept in mind when considering and interpreting the results, and certainly if there is future research to continue these good efforts you've started, we should try to get a much more diverse panel, perhaps even some more qualitative data analysis as well.

CHAIRMAN GROSS: John.

DR. COSTER: I just want to go back to the comment before Stephanie's, and again, this is an issue that I think you should address to the database companies that speak later, but there was, in fact, mergers and acquisitions going on in the marketplace. I don't know to what extent the leaflets that were collected reflected all of that happening in 2001.

There used to be a short form, as I said before which was discontinued in April of 2000. Whether the systems were still using the short form is another issue.

I didn't know there were three versions of this particular vendor's information, but I think that that is an issue that is worth exploring, whether or not those things happening in the market, in fact, affected the information that was collected.

DR. SVARSTAD: I think, if I may --

CHAIRMAN GROSS: Please.

DR. SVARSTAD: -- I think that the abbreviated monograph, we did not see that. However, we did see, as I noted, about 17 cases where they were just printing out the patient counseling message and rather than the full monograph, just to answer your question.

DR. COSTER: I think though, and this is something that you should address to the database companies, there may have been a patient counseling message. There may have been a short form, and there may have been a long form.

And I don't know if the database companies produce the counseling messages as well or if they are produced by, you know, other entities.

DR. SVARSTAD: Vendor 1 and the versions under Vendor 1 include both the patient counseling message, which is short, and the full monograph. They're called somewhat different things, as I understand it.

CHAIRMAN GROSS: Okay. Thank you all very much.

It has been an excellent session this morning. We will now adjourn for lunch and reconvene shortly after one.

(Whereupon, at 12:15 p.m., the meeting was recessed for lunch, to reconvene at 1:00 p.m., the same day.)

(1:11 p.m.)

CHAIRMAN GROSS: Good afternoon, everybody. Thank you for all coming back.

We're going to proceed now with the open public hearing, and I'd like to ask Dr. Ratto, Dr. McEvoy, Donna Storey, Thomas Menighan, Ray Bullman, and Dr. Sasich to please come up to the front.

I guess most of you are already here, and Tish Pahl.

Now, with respect to all of the other participants, we ask in the interest of fairness that the people how are about to speak address any current or previous financial involvement with any firm whose product they may wish to comment on.

The first speaker is Dr. Nicholas Ratto, manager of Consumer Drug Information Group with First DataBank, the knowledge inside in San Bruno, California, and he has up to seven minutes.

DR. RATTO: Thank you.

I wanted to give a couple of quick highlights on my background. It's very similar to the clinical pharmacists at First DataBank.

Earlier in my career I practiced in a number of health care settings, including acute and ambulatory care. My responsibilities included direct patient care in pharmacist operated triage, diabetes and anti-coagulation clinics during my 11 years in the VA system, as well as direct participation on medical and infectious disease teams.

Consequently I've personally counseled many hundreds of patients, as have my colleagues.

The written patient education survey that Bonnie reviewed earlier utilizes a scoring document which we consider to be valid, though we do take issue with a few of the criteria on each individual drug surveyed.

We also suggest that in future surveys selected authoritative, secondary references, such as the HSF drug information reference source, be utilized in conjunction with the professional labeling.

As an example, we discovered a labeling reference to, quote, unquote, reaction to allergy shots for atenolol that did not have any literature information backing it up, as per a Medline search.

The conclusion regarding the survey is that we -- and that includes all that are involved in written patient education, including the meg. guide system through FDA, have work to do regarding overall quality improvement. I think that's clear.

First DataBank has, in fact, developed a clinically well substantiated and field tested, thorough editorial policy and procedure for patient education. We will compare that to the scoring guidelines that came through the recent survey as well for any additional updates.

We are in the process of reviewing the 2000 monographs for full compliance with this particular policy that we have in place at this time, given that that policy has evolved over time as requirements for patient education have evolved over the last ten to 12 years, and also the number of monographs involved.

There are those inside and outside of FDA that would tout the FDA approved med. guides as the best solution to this quality issue that we face. However, I do want to point out that even the med. guides are not fully action plan compliant.

For example, I performed a cursory review of Ziagen, which is abacavir, med. guide, and found that while it contained a considerable amount of useful risk information, it did lack any advice related to to other medications being taken, and did not give advice regarding suspected overdoses or storage information, along with a couple other areas, and only partially met criteria for missed dose advice, as well as information about keeping it away from children, et cetera.

Now, my point here, please do not misunderstand. My point here is not to criticize FDA or deflect the discussion away from First DataBank or any other provider, but merely to demonstrate that as was stated earlier, no written document is idea at this time.

Those that tout the FDA approved med. guides and the routine distribution of the professional FDA approved labeling to patients -- and I emphasize the word "routine distribution" -- are highly skewed towards the risks of drug therapy.

Again, don't misunderstand me. Provision of risk information is entirely appropriate and necessary.

Distribution of the professional labeling to selected patients at the discretion of the pharmacist or physician is appropriate, however, not at the expense of quality of life and benefit information.

And I'm not really speaking about the benefit noted in the survey criterion which deals with maximizing drug effectiveness. I'm basically discussing quality of life.

A majority of patients in my experience and informally corroborated by conversations with colleagues that may include David Blair, who is a past NCPIE Communicator of the Year honoree; most patients do not have either the formal education or the medical knowledge to put risk information into proper perspective without direct assistance from a health care professional.

For example, these patients, upon reading of the risk of death due to rhabdomyalysis (phonetic) from the cholesterol lowering statin drugs may frequently refuse to take the medication. This could result, of course, in a negative impact on quality of life. The patient, for example, may suffer a premature or preventable major cardiovascular event, such as a myocardial infarction.

This insidious problem of noncompliance is frequently not adequately addressed given the difficulty of characterizing or tracking it. Studies already show that medication compliance rates are in the 50 percent range, which is an unacceptably low number in our opinion.

The risk information does need to be communicated. There's no question about that, but along with benefit information. For example, in our monographs, we explicitly state that statins help prevent heart attacks and strokes. When the indication is made of a possible fatal outcome for a drug, we note the incidence of that potential fatality by saying that it's either rare or infrequent, depending on what the literature supports.

This gives the patient a more balanced picture of risk and benefit. Non-clinicians or ex clinicians may tend to lose sight of these critical issues in the zeal to fully inform a patient.

First DataBank's clinical pharmacist staff is solely interested in assisting health care customers in improving patient care. Furthermore, we believe that no written document can ever fully substitute for a personal interaction with a professional. Every patient is unique, and each has their own knowledge base, misconceptions, biases or barriers to communication.

The health care professional lends crucial perspective and individualized advice to the patient which cannot be capsulized in any leaflet. The written patient education material is an essential component of this process, but inherently never can stand alone if your goal is a fully educated patient.

Efforts must be made to utilize the proven methods of freeing up pharmacists' time to counsel patients, such as automation aids and use of certified pharmacy technicians.

In conclusion, I reiterate our proposal to FDA for ongoing periodic dialogue and feedback related to our written patient education information. The purpose would be to address quality issues, and I suggest this would best be accomplished in cooperation with some of the clinician members of Dr. Svarstad's group whereby constructive interchange would occur regarding content and format of monographs.

Perhaps as appropriate, the action plan or scoring guideline sheet criteria may be revisited in the future, which was actually mentioned earlier as well by a previous speaker.

Other drug information providers and various stakeholders would be welcome in the discussion as well.

CHAIRMAN GROSS: Thank you very much.

The next speaker is Dr. Gerald McEvoy, Assistant Vice President for Drug Information of the American Society of Health System Pharmacists.

DR. McEVOY: Good afternoon. The American Society of Health System Pharmacists appreciates the opportunity to provide comments to this committee.

My presentation has not been paid for by any organization or pharmaceutical company. ASHP does receive monies from external organizations through their purchase of advertising in our journal, leasing of exhibit space at our annual conventions, and through corporate sponsorship, which is wholly disclosed to participants of selected continuing education related publications.

ASHP is a 30,000 member national pharmacy association that represents pharmacists who practice in hospitals, health maintenance organizations, long-term care facilities, home care, and other components of health care systems.

ASHP has a long history of medication error prevention efforts, and we believe that the mission of pharmacist is to help people make the best use of their medicines. Assisting pharmacists in fulfilling this mission is ASHP's primary objective.

The society has extensive publishing and educational programs designed to help members improve their professional practice, and it is the national accrediting organization for pharmacy residency and pharmacy technician training programs.

ASHP believes that private sector publishers, including professional associations like us, must play an important role in the creation and dissemination of useful medication information. ASHP has long been an advocate of the role of pharmacists in providing useful written and oral counseling to patient about their medications, and we have a 25-year history of publishing medication information intended to educating patients about their drug therapy.

ASHP was a member of the Keystone Group, and was one of the first private sector publishers to incorporate the guidelines of their 1996 action plan for criteria, goals, layout, and language on useful prescription information in its patient resources.

I might mention that that effort took us about two years to complete. We began it in 1997 and completed it in 1998.

ASHP applauds the progress made by community pharmacies in voluntarily providing written information on prescription drugs. The results of the study clearly indicate that gains have been made in that regard in terms of the numbers of patients who are receiving such written information. Almost 90 percent of them in this study did, and that compares with figures of around 55 to 64 percent in surveys that were conducted in the mid-1990s.

While this certainly is a laudable achievement, we also recognize that continued attention to improving the usefulness of this information remains important, as reflected in widely variable scoring of the information quality, particularly regarding the risks of treatment.

However, as acknowledged in the 1996 action plan, it is expected that as the plan is implemented, additional information will be gained regarding what constitutes useful, and that any associated guidelines should be subject to periodic review, evaluation and refinement.

Therefore, ASHP believes that the current study should be viewed principally as a further refinement of the definition of useful rather than as an indictment of the current voluntary efforts. In fact, careful inspection of the criteria used in the current report indicates that usefulness was defined in many cases by criteria that were not specifically enumerated in the 1996 action plan.

For example, the plan does not specify the inclusion of pharmacologic therapeutic class information as a component of what is considered sufficiently specific and comprehensive. Yet this weighs heavily in the current report findings where three out of eight subcriteria used to measure this component in the glyburide information are about the provision of pharmacologic therapeutic information.

Another example is the specific inclusion of a statement that atorvastatin is an HMG-CoA reductase inhibitor, a very cumbersome class designation that probably has very little meaning to patients relative to the more commonly used term, "statins."

The source and rationale for some criteria also are unclear. for example, the origins of a precaution about kidney disease and atorvastatin; the eight-hour missed dose window specified for atenolol, atorvastatin and glyburide.

The reason that I bring up these examples is that we as publishers need to be part of the process. We need to understand the basis of these statements because they are going to be applied as yardsticks for our information.

In the spirit of the action plan regarding the evolving nature of the definition of usefulness, what seems most important is that criteria that will be used in judging the usefulness of written consumer information should be widely agreed upon and circulated to both public and private publishers so that they will be fully aware of the yardsticks against which their information will be measured.

In doing so, however, it is important that FDA also not lose sight of the goal of the action plan that some flexibility in content be allowed.

Missing from the current report are recommendations on how to further improve the usefulness of this information. Therefore, ASHP recommends that FDA solicit advice in the form of an advisory panel of experts and public and private sector stakeholders regarding further refinement of the definition of usefulness and the associated specific criteria that will be used in evaluating adherence to this definition.

The panel also should recommend mechanisms for insuring that publishers and providers of consumer medication information are fully advised about such ongoing developments to that appropriate changes can be implemented in their data.

Likewise attention should be given to possible implementation of other recommendations included in the action plan. As part of this strategy, the advisory panel should be charged with identifying priority areas and interventions for improving the usefulness of this information and should provide advice on possible interventions in the development and distribution of the information.

ASHP strongly believes that the proper course for FDA is to defer regulatory action at this time while pharmacy organizations and private sector medication information publishers and providers maintain their commitment to improve the usefulness of information that is provided to 95 percent of patients by 2006.

As part of ASHP's commitment to the mission of pharmacists for helping patients make the best use of their medications, the society will continue to follow the findings of and make recommendations to FDA and other groups, as well as make appropriate enhancements to its patient medication information aimed at improving usefulness.

In addition, ASHP remains ready to assist the FDA in further implementing the recommendations of the 1996 action plan both as a professional pharmacy association and publisher, and in serving any formal advisory capacity the agency pursues in this regard.

Thank you.

CHAIRMAN GROSS: Thank you very much.

Donna Storey is next, and she has a personal story to relate to us.

DR. STOREY: Thank you for the opportunity to speak here today.

My mother, Monica George, died of Rezulin induced liver failure in September 1998. She is one of the 66 Rezulin fatalities officially acknowledged by the FDA.

I understand that this committee was created, in part, as a result of an FDA report on lessons learned from the handling of the Rezulin fiasco. However, I was very concerned to discover that one member of this committee recently appeared as an expert witness for Warner-Lambert in a Rezulin trial here in Rockville involving my mother's case.

In his testimony, he described Rezulin as a success story and a model case. He also stated that from the public health point of view, there was no reason to recommend monitoring liver functions the first year the drug was on the market because that could lead to warning fatigue.

Rezulin may, indeed, be the model for how things do work, but should this be your model for future drugs?

The Rezulin story begins with the very troubling circumstances under which the drug was approved. For further information, this has been well documented by David Willman in his series of articles on Rezulin in the L.A. Times.

However, in keeping with today's topic, I'll focus on what happened after the drug was on the market.

As reports of serious liver events began to come in only months after approval, the FDA and the drug's maker, Warner-Lambert, responded by sending "Dear Doctor" letters calling for increased liver monitoring. It took almost two years for a black box warning to reach the PDR.

The question is: how much of this information reached the patients already taking Rezulin?

I believe that the answer is very little, indeed. In fact, it reached few doctors. Some of the country's most prominent hepatologists who treated my mother were woefully ignorant of the mounting evidence of Rezulin's toxicity to the liver.

Most troubling was the FDA's reaction to the death of Audrey Jones and Rosa Delia Valenzuela, two patients involved in clinical trials of Rezulin. Both women suffered liver failure in spite of strict monitoring, their liver enzymes rising precipitously only weeks after normal results.

Although this was a clear indication that liver monitoring was not effective, the FDA never made any public comment on these cases.

My mother began taking Rezulin in November 1997 based on information her doctor received months before from a company salesman. The doctor stated under oath that he did not read "Dear Doctor" letters.

Would my mother, a registered nurse, have stopped taking Rezulin if she had known of the growing number of reported liver problems?

Although I'm confident the answer is yes, the real point today is that she was never given the choice. The current system penalizes patients who begin a new drug early on, in essence putting them in the position of unwitting participants in a poorly controlled clinical trial.

As a consumer, I have a few suggestions for improving this situation. When the safety profile of any drug changes, this information should immediately be made available in plain language a part of the patient information leaflet we're talking about here when the prescription is refilled.

These changes should be highlighted prominently, in red, for example, at the top of the page and dated. And I'd also recommend a consultation with the pharmacist should be required.

I also suggest that a newly approved drug, especially one approved on the fast track, be identified as such on the label, including a caution that the complete safety profile is not yet known.

And it's also vital to make the reporting of adverse events not voluntary, but really mandatory for health care professionals so that we can build an accurate safety profile in the first place.

I know that some argue this kind of disclosure would only frighten patients, but we really should consider who is being protected when this information is withheld.

Doctors are spared phone calls from worried patients, but any physician or pharmacist who truly values patient welfare should at least be willing to answer a few questions about medication and reevaluate the risk-benefit tradeoff for an individual patient.

Drug companies have also fiercely resisted changes of this sort.

I'd like to return to the Rezulin example for a moment. Three weeks before my mother died in indescribable agony Warner-Lambert held a party. This is the flyer for it. "Celebrate Rezulin at the Billion Dollar Bash. It's Become a Blockbuster Drug."

This demonstrates the enormous benefits to drug companies if concerns about warning fatigue override concerns about safety. Rezulin would never have earned a total of $2.1 billion if it had only been prescribed to the relatively small population of insulin dependent Type II diabetics who did not respond well to other therapies. For these patients, the benefit was clearly worth the risk.

It was never worth the risk for a mild diabetic like my mother, who was in good health and had a hemoglobin A1c of seven before she began taking this so-called miracle drug.

Yes, all drugs have risks, but unfortunately, in the current environment where efficacy is misleadingly determined by surrogate endpoints, adverse side effects are consistently downplayed and profit is valued over human life to the point that some drug companies offer to indemnify doctors who are sued for prescribing their drug, as Warner-Lambert did with Rezulin.

All of the risk falls on the patient, all the more so if we are denied access to crucial information.

As I've done more research about drug safety in the aftermath of my mother's death, I've been horrified to learn that the Rezulin model has, in fact, been repeated over and over again in the past ten years. No one seems to be learning anything.

As members of the Drug Safety and Risk Management Advisory Committee, you are in a unique position of power. You can keep using Rezulin as a model of how things should be done. You can keep information from patients and provide political cover for FDA missteps.

You can use your appointment to this committee to make extra income serving as an expert witness for pharmaceutical companies or you can see Rezulin as a cautionary tale. You can advise the FDA to enact changes that will inform and thereby protect consumers.

I urge you to use your influence to address the serious systemic problems with the safety of prescription drugs so that American consumers who take an FDA approved drug need no longer wonder if they take their lives in their hands.

I would also like to submit for the record the transcript of Dr. Brian Strom's testimony from January 28th, 2002 in the case Andrea Shaw, et al, v. Warner-Lambert, Parke Davis.

Thank you.

CHAIRMAN GROSS: Thank you, Dr. Storey.

Next is Thomas Menighan, immediate past President of the American Pharmaceutical Association.

MR. MENIGHAN: Good afternoon. Thank you for the opportunity to present the views of the nation's pharmacists.

I'm Tom Menighan, a long time community pharmacist and home infusion practitioner. For the last two years, I've been involved in the provision of health information and communication capabilities to consumers and pharmacists via the Internet.

I am immediate past President of APhA and today am appearing on behalf of more than 50,000 practicing pharmacists, pharmaceutical scientists, student pharmacists, and pharmacy technicians.

We frequently partner with groups to develop educational materials for pharmacists and consumers. However, we did not receive any funding today to participate,a nd I am representing solely our members and our association.

We applaud the FDA for stimulating and our pharmacist members for providing written materials to consumer. Yet as evidenced in the evaluation of written information provided, the December 2001 report, many challenges remain.

The biggest challenge, however, is not in making written information useful. Rather, it's getting written information actually used by consumers.

For those of you who sat in the restaurant next door at lunchtime, an alarm went off. I looked around the room, and I noticed nobody responded. Nobody got up. Nobody changed what they were doing. They went about their business.

I tell you; I submit to you alarms go off all day long every day in our lives, and we've learned to ignore them. There's to much noise out there. Absent someone saying directly to you as an individual, "This is important. Pay attention," most people won't.

To insure the safe and effective use, pharmacists help patients manage their medications with oral consultation, written information, and increasingly other services. Written CMI, the subject of today's meeting, is one method to provide patients with information on proper use.

We support the provision of better information, including written CMI, about drug therapy. Our profession has made great strides in this area, as suggested by other speakers and recent reports.

However, the results of the study also show that the quality of information distributed varies and did not meet the criteria for usefulness 100 percent of the time. While we agree that CMI can be improved, determination of specific inefficiencies and the outcomes of change will require continued research.

One very concrete way of gaining improvements would be, as suggested previously, to more broadly publish the criteria used in the study and then to challenge vendors and publishers to meet or exceed the criteria.

Yet no matter how well patient information is written, it's useless unless patients use it. Written information is an adjunct for communicating to patients. The primary mechanism continues to be one-on-one encounters between health care professionals and patients so that new information can be factored together with their routine.

This is especially true for older patients with multiple chronic medicines and confusing therapeutic regimens. Written information can support and enhance medication therapy management services, but written information alone without accompanying oral consultation is insufficient to meet the needs of consumers and will do little to improve patient comprehension and compliance.

Without the pharmacist emphasizing the importance of written information to individuals, we risk patients throwing it away just like junk mail.

Customization, not standardization, is part of the answer. It's important to note that CMI developers should be encouraged to improve the quality of patient information, and that criteria for evaluation should be publicized. APHA will not support government regulations that would specify the content, precise language or the specific design of CMI.

Patient information must be tailored to each patient and used to supplement information provided by the pharmacist and other health professionals. Attempts to standardize the content would reduce our ability to provide information specific to the particular drug and the particular patient.

We should, instead, foster innovation that takes full advantage of technology, pharmacists' knowledge of their patients to create better educational experiences for consumers.

Regulation may unintentionally hamper our ability to provide customized information to individual patients. If encouraged, consumers will ask questions that bring the written information into their consciousness and lead to improved care. The ability to customize that information is key.

Vendors who have written information should be encouraged to keep that information contemporary. Information and relative weights of various components that should be communicated to patients will vary for each product.

For example, proper storage instructions are more important for products subject to degradation, such as antibiotics that are reconstituted at the pharmacy.

For other products, such as solids, storage conditions may be less important. A patient with asthma on multiple drugs will be more interested in information on interactions and dosage adjustments to maintain proper care.

We understand the agency recognizes progress made in distributing patient information and is not moving to regulate CMI at this time. We strongly support the FDA's efforts to improve appropriate use of medications through patient education activities, and we are committed to providing and improving educational efforts of pharmacists with their patients.

In summary, the nation's pharmacists urge FDA to, one, continue promoting the voluntary distribution of written CMI as an adjunct to oral counseling.

Two, publish criteria to help vendors shape CMI for pharmacy management systems while allowing for innovation to customize and meet individual patient needs.

Three, encourage increased use as well as the usefulness of written information through support of medication therapy management services.

Thank you for your consideration of our views.

CHAIRMAN GROSS: Thank you, Mr. Menighan.

Next is Ray Bullman, Executive Vice President of the National Council on Patient Information and Education.

MR. BULLMAN: Thank you.

My name is Ray Bullman. I'm the Executive Vice President of the National Council on Patient Information and Education, a nonprofit coalition of 135 organizations whose mission is to stimulate and improve communication of information on appropriate use of medicines.

As such, NCPIE served on the Keystone Committee in 1996 to develop the action plan for the provision of useful prescription medicine information.

My presentation today is not supported by any external organization or pharmaceutical company. NCPIE does accept unrestricted educational grant support from pharmaceutical manufacturers and foundations.

Also, please note that the following comments do not necessarily represent the opinion of all members of our coalition.

A review of initiatives to improve consumer medicine information is important to appreciate the historical perspective in which the advisory committee will make its recommendations. Many of these were mentioned by Tom McGinnis this morning. I would like to add to his comprehensive presentation the following.

One, the Omnibus Budget Reconciliation Act of 1990, or OBRA '90, which mandated that pharmacists extend an offer to counsel Medicaid recipients about their prescription medicine, subsequent to implementation of this federal provision in 1992, nearly all states amended their Pharmacy Practice Acts to extend the offer to counsel to non-Medicaid customers as well.

Two, Health People 2000 and Healthy People 2010, both address prescriber and pharmacist counseling, communication about medicine's appropriate use and potential risks, and quality of written medicine information.

Three, "To Err Is Human," released by the Institute of Medicine November 1999, which focused national attention on the magnitude and impact of medication errors, especially in hospitals. The report has stimulated an unprecedented level of programming, collaboration, and research to understand and eliminate avoidable medication errors.

Additionally, ongoing national outreach campaigns, such as FDA's own Take Time to Care initiative, and NCPIE's talk about Prescriptions Month, National Brown Bag Medicine Review Program, and most recently Be MedWise, launched in January of this year continue to stimulate and reinforce the need for quality medicine communication between consumers and health care providers.

A key element of each of these campaigns is that CMI is most effective when it features high quality oral counseling with supplemental written information that is mediated by the health care provider.

It is only with the full commitment of all health care professionals to actually talk with patients about prescriptions in a meaningful way that patients will understand the possible risks and realize their medicine's full benefits through enhanced CMI.

I recommend a CMI research agenda that includes the following issues:

Number one, how much information is too much? For those prescription medicines that require medication guides, do we know their effect on patient understanding of possible risks?

Do we know the extent to which the medication guides contribute to appropriate use?

Do we know how medication guides have affected patient adherence and health outcomes?

Number two, what effect does a simplified format for CMI have on safe medication practices? For example, what post marketing research is being done or considered on the new drug facts label now required on most nonprescription medicines?

Number three, focusing, for example, on the five or six prescription medicines most commonly prescribed and used by persons age 65 and older and considering different formats for and quantity of information conveyed on pharmacy generated leaflets; different types of follow-up contact from physicians, pharmacists, nurse prescribers, and physician assistants with various time frames of starting a new prescription.

Number four, advice to use one pharmacy for all your medicines and complete the patient profile form are common suggestions to promote safe medicine use. What percent of patients age 65 and older have such forms on file at their local pharmacy? Are these patients asked each time they come in for a new prescription to fill out and/or update their form? Are patients routinely asked about OTCs and dietary supplements they may be using so that this information can be added to the profile?

Number five, much attention has been focused on adoption of computerized physician order entry systems primarily in hospitals as a way to reduce medication errors. While the advent of PDA technology has made this an option for ambulatory care settings as well, implementation to date is extremely limited.

In the year 2000, Dr. Susanna Bedell cited discrepancies of up to 75 percent in reported versus recorded medications. Dr. Bedell's research was conducted in physicians' offices.

What if community pharmacies sent a copy of high risk patient's profile forms to each of the prescribing physicians? To what degree could such technologically enhanced pharmacy prescriber communication improve CMI overall?

Finally, the research findings reported by Dr. Svarstad today serve as an important baseline from which subsequent improvements in CMI can be measured. I suggest that FDA reevaluate CMI in conjunction with the mid-course review of Health People 2010.

I would also suggest that further assessments include CMI offered via the Internet. There are far more drug information purveyors offering CMI on line directly to consumers than there are those that provide CMI databases to retail community based pharmacy.

Such a schedule would place the reevaluation in 2005 to then be repeated at the end of the decade. This is a logical approach and time frame to support FDA's role as the lead federal agency for monitoring progress to meet the Health People 2010 drug safety objectives, two of which are to increase the proportion of patients receiving information that meets guidelines for usefulness when their prescriptions are dispensed, and secondly, to increase the proportion of patients who receive verbal counseling from prescribers and pharmacists on appropriate use and potential risks of medications.

NCPIE remains committed to working to insure that consumers receive useful information about their prescription medicines.

Thank you very much.

CHAIRMAN GROSS: Thank you, Mr. Bullman.

Next is Dr. Larry Sasich, who represents the Public Citizens Health Research Group.

DR. SASICH: Thank you very much for this opportunity.

My name is Larry Sasich. I'm with Public Citizens Health Research Group in Washington, D.C., and neither the organization nor myself have any conflicts of interest that would bear on today's meeting.

The Food and Drug Administration's characterization of the results presented here today in the 2001 evaluation as showing a private sector making progress and meeting the goals of providing the public with useful written prescription information is disgraceful.

Likewise, the finding that the overwhelming majority of pharmacy generated leaflets adhered fully to the criteria of being scientifically accurate is appalling and is apparently a failure of the studies' authors in the FDA to understand the definition of scientifically accurate, as defined in the 1996 action plan.

The action plan is the basis for the evaluation of the quality of written information being distributed to consumers by pharmacists and was agreed to by commercial information vendors, trade lobbies representing pharmacy and medicine and consumer groups. There was nothing unknown to the people who are now producing unregulated commercial information vendors. They were all at the table. They knew what the rules were years ago.

If the Food and Drug Administration and the study's authors had adhered to the action plan, their conclusion would have been simple. No prescription drug consumer that gets one of these patient information leaflets is receiving written drug information that meets minimum acceptable quality standards of the action plan.

The action plan criteria are minimum. They're a floor.

Public Citizen was a member of the Steering Committee that negotiated the action plan in December of 1996, and the plan is very clear as to what constitutes acceptable information that will count towards the quantitative goal of 75 percent of consumers receiving useful drug information.

Page 16 of the action plan states only written information that is useful will count towards the quantitative goals of the plan, and to go back a little bit, Public Law 104-180 was enacted in 1995 and led to the action plan. This law required the action plan to achieve goals consistent with the goals of the FDA's 1995 proposed medication guide rule.

The agency's stated standard for the termination of information usefulness was each sample of patient information leaflet will be scored on each criterion using acceptable and not acceptable cutoff points. FDA believes that for a particular information sheet to be judged as acceptable overall, it must receive an acceptable rating on each of the individual components.

During the highly contentious debate that resulted in the action plan, partial credit was not envisioned, discussed or agreed to by the Steering Committee for patient information leaflets distributed by pharmacists. It is impossible to comprehend any usefulness for patient safety information that on average contains only 50 percent of the minimum required information as documented in the FDA's 2001 evaluation.

In fact, safety information that is incomplete is misleading and potentially danger and some information is not better than none at all. Please read the short vignette at the beginning of our written comments about seven year old Cory Christian (phonetic) and what happens when parents rely on information that is incomplete handed to them by a health care provider.

Since the FDA's resurrection of the 1995 medication guide rule of the 1979 proposed rule to require patient package inserts, or PPIs, based primarily on a drug's approved product labeling, this has been a theme that goes back to 1979. Consumers and the agency have been looking for the information that's contained in professional product labeling.

There have been at least five surveys or systematic examinations of the quality of patient information leaflets distributed by pharmacists. In 1995, the agency examined the adequacy of written drug information produced by eight commercial information vendors.

For example, none of the vendors mentioned the contraindication for the use of enalapril when allergic reactions or angioedema occurred during previous treatment with similar drugs. This is potentially life saving information for patients.

A study published in April 1996 assessed whether 50 Washington, D.C. area pharmacies would simultaneously dispense prescriptions for the potentially life threatening combination of urethramycin (phonetic) and the antihistamine terfenadine, which has since come off the market.

In May 1993, patient labeling was added to terfenadine's professional product labeling. This information specifically warned in upper case, bold letters not to use terfenadine with urethramycin. Patients were also warned that this interaction could cause death.

The FDA's and the manufacturer's expectations were that this information would be provided to patients by pharmacists. Some commercial information vendors voluntarily chose not to include this information in their leaflets, and pharmacists voluntarily chose to dispense unregulated patient information leaflets that omitted life saving information rather than distributing FDA approved patient labeling for terfenadine that warned of the urethramycin drug interaction.

Public Citizen obtained patient information leaflets for 15 different nonsteroidal anti-inflammatory drugs in April 1997 distributed by community pharmacists. A total of 59 leaflets produced by four commercial information vendors were evaluated using four criteria based on the 1995 proposed medication guide rule. None of the private sector leaflets met the criteria.

In a study conducted by Private Citizen conducted in April 1998, 15 licensed pharmacists evaluated the PILs for five fluoroquinolone antibiotics produced by four unregulated commercial information vendors according to the scientific accuracy criteria of the action plan. The information content of these patient information leaflets was not satisfactory to meet the scientific accuracy criteria of the action plan.

Public Citizen commented on the methodologic inadequacy of the FDA's 2000 survey. Despite the shortcomings of this FDA funded survey, only 12.5 percent of pills distributed with the drug ibuprofen informed consumers of the drug's contraindications and only 5.3 percent included the specific precautions, their significance and how consumers could avoid harm.

Rather than demonstrating progress, as the FDA seems to believe, the private sector has shown a consistent inability over the years to produce useful drug information according to agreed upon guidelines.

The authors of the 2001 evaluation, as they did in their 2000 survey, failed to comprehend the action plan's simple definition of scientifically accurate: information consistent with or derived from FDA approved labeling.

The private sector leaflet for nitroglycerine is one example of a lack of accuracy found in these leaflets. There are others that are in our written testimony.

The professional product labeling for nitroglycerine clearly indicates the use of this drug with sildenafil together as contraindicated. These leaflets were evaluated for containing the subcriteria about the use of nitroglycerine in combination with sildenafil.

Only 32.7 percent of these leaflets were fully compliant. Unbelievably, 99.1 percent of the leaflets were found to be scientifically accurate.

The private sector leaflets omitted the majority of important safety information for consumers that is available in these drugs' professional product labeling. The FDA and the authors of the 2001 evaluation are negligent in portraying to the public that the majority of these leaflets are scientifically accurate.

We are now 22 years past the private sector's promise to develop a variety of systems that would meet the goals of the FDA's 1979 proposed rule that have required patient package inserts, or PPIs, for ten classes of prescription drugs.

Spearheaded by trade groups representing pharmacy in medicine, a lobbying effort was undertaken that caused the PPI regulation to be amongst the most controversial issued in the last months of the Carter administration. Needless to say, consumers favor the proposed PPI program.

The day after President Reagan's inauguration in 1998, the White House called the FDA to make it clear that the PPI regulation was not to be enforced. This would not be the last time that an elected representative of the people would attempt to prevent the public access to high quality written drug information.

On two occasions in the recent past, Michael Crapo of Idaho penned legislative language to prohibit the FDA form implementing the medication guide rule.

In 1982, the FDA officially rescinded the regulation in favor of a voluntary plan. Private sector initiative commenced with the formation of the National Council on Patient Information and Education and the consistent failure of the private sector to deliver what was promised, culminating in the 2001 evaluation.

The failure of the private sector to meet the quality goals established in the action plan and thus, the failure to achieve the distribution goal of 75 percent of patients getting scientifically accurate information leaves only one option under Public Law 104-180, and I quote. "The Secretary," meaning the Secretary of Health and Human Services, "shall seek public comment on other initiatives that may be carried out to meet such goals."

We urge the Drug Safety and Risk Management Advisory Committee make a single recommendation to the FDA. The FDA should follow the process as defined in Public Law 104-180 and go forward as rapidly as possible with implementing the action plan by regulation. Giving the private sector a free ride until 2006 to meet the goals of the action plan would be irresponsible.

Thank you very much.

CHAIRMAN GROSS: Thank you, Dr. Sasich.

The last speaker for the public hearing segment is Tish Pahl of Health Resources Publishing Company.

MS. PAHL: Good afternoon. My name is Tish Pahl of the law firm of Olsson, Frank and Weeda here in Washington, D.C.

I'm speaking today on behalf of Health Resource Publishing Company of St. Louis, Missouri.

It is likely that leaflets Health Resource publishes in retail pharmacies were reviewed in Dr. Svarstad's study.

Health Resource thanks the committee for the opportunity to present its views. Health Resource has already submitted its written comments to the committee. Today we wish to elaborate briefly upon that written comment.

Health Resource commends Drs. Svarstad and her colleagues for the enormous effort evident in the 2001 evaluation. Measuring something as nebulous and subjective as usefulness is a daunting task.

Health Resource provides prescription drug information to consumers as the retail pharmacy level. Health Resource publishes customized educational newsletters at the pharmacy that are given to the customer with his or her prescription. One section of the newsletter provides prescription drug information that is intended to satisfy the useful information standards of Public Law 104-180 and the Keystone criteria set out in the action plan for the provision of useful prescription medicine information.

The Health Resource consumer medication information, or CMI, strives to be scientifically accurate, neutral, useful, and to be presented in a format that is easily understandable to consumers.

Qualified experts prepare the CMI. It is derived from authoritative references, such as FDA approved labeling. It is reviewed for completeness, accuracy, consumer comprehension, and is updated regularly.

Health Resource tries to get CMIs to a sixth grade reading level.

I will now turn to our brief substantive comments on the 2001 evaluation. First, the 2001 evaluation measures the usefulness of CMIs collected according to over 60 separate subcriteria for each drug. Many may not have anticipated that a CMI would be expected to contain this much information at this level of detail.

Health Resource repeats the call made earlier for greater, more open public discussion of the standards for setting the subcriteria that will measure usefulness.

Second, in Health Resource's view, more information in a CMI must be balanced against the need for that information to be legible and comprehensible to consumers. Health Resource believes it would have been very difficult for a CMI to include all of the information that was expected in the evaluation on a single sheet of paper without also compromising comprehension and legibility.

The information is so extensive, it would have had to have spilled onto additional pages in order to be readable. Health Resource's experience is very consistent with that observed in the study. Pharmacies have been very resistant to expanding a CMI beyond a single page.

We believe there are several reasons for this resistance. The single biggest concern is work load and work flow. An additional page multiplied by hundreds or thousands of prescriptions is an enormous increase in cost and work for a typically short staffed pharmacy. With more pages floating around a busy pharmacy, errors may also be more likely.

Health Resource understands that pharmacies are already under pressure from vendors to increase the amount of information in a CMI. Even before the 2001 evaluation, Health Resource has seen CMIs in as small as five point type as pharmacies struggle to include the information, but still keep a CMI to a single page.

CMIs must include the level of detail expected in the 2001 evaluation. The issue of limited space and legibility within that space must also be addressed.

Finally, Health Resource is concerned that consumers will not read detailed risk information. Consumer fatigue with long winded risk information is evident in the consumer reaction to the brief summary requirement that must accompany most prescription drug promotion.

According to FDA's recently released data, 70 percent of consumer survey respondents read little or none of the brief summary. Fewer people are reading the brief summary now than they did three years ago. In Health Resource's view, written information no matter how useful is not going to be a substitute for the advice of a consumer's health care professional.

To this end, Health Resource believes that a CMI should concisely focus upon those side effects, warnings, contraindications and precautions that are the most common and the most serious. The CMI should plainly state that it is not complete and that a consumer can obtain more information from his or her health care professional.

Thank you.

CHAIRMAN GROSS: Okay. Thank you very much.

That's the end of the public comment. The next speaker is Dr. Ruth Day from Duke University, who will give us a framework. The title of her talk is "Consumer Comprehension of Educational Material, Key Cognitive Principles."

DR. DAY: So the question is: how do consumers comprehend educational materials?

In order to answer this question, we need to consider a variety of key cognitive principles. Underlying those principles is the idea of cognitive accessibility.

Cognitive accessibility is the ease with which people, both consumers and professionals, can find, understand, remember, and use drug information and, of course, do so in a safe, effective, and efficient way.

So what are some of these cognitive principles? Well, there are too many to talk about today. I'm only going to focus on a few, but I would like to mention that they have been studied in carefully controlled laboratory studies for many years, all of them at least a decade and some of them as long as 50 years. So there's considerable empirical support for these principles.

Information load. Obviously too much is not good. How much is too much? We'll come back to that in just a moment.

We can manage information load better by using other cognitive principles, such as chunking. Chunking involves putting together what goes together and separating it out from surrounding information.

We can further enhance people's ability to understand a chunk by helping out with coding, how they're going to code that information into their minds. An obvious way is to put a title or a subtitle in front of it. That enables people to then understand the information better and also remember it better later.

Representation deals with different types of formats that can be used for chunks of information. Some formats help and some hinder comprehension, and we need to pay attention to that.

Location is important as well. If, for example, we have a long list of items within a chunk, such as side effects, it's very well documented that people will do better in processing the information at the beginning and the ends of the list, and they're going to miss the information in the middle.

So what can we do to enhance their processing of information throughout a list and other aspects of the leaflet?

Much has been said today about readability. There are objective measures, formulas for readability. There are many of them. However, they only do two things. They look at the length of sentences and they look at the familiarity of the works that are used. That's all they do, and that's where those measures of sixth grade level, eighth grade level, and so on come from.

There are many more things involved than comprehensibility. We need to take into account syntactic and semantic factors. For example, for syntactic, how complex is the grammar? So I can make up a sentence which is relatively short and it will do well in readability measures, but it could be so complex that it's hard to understand the information it contains.

Another measure in comprehensibility has to do with the number of idea units that are present. These are what are called propositions. So how many propositions are there in some information and how densely is it packed?

Obviously attention is a very important principle, and there are many different types of attentional processes. How do we get people's attention? How do we get them to be able to direct it to some information when they need that and separate it out from other information, and so forth?

We want people to do a variety of cognitive tasks with these leaflets, not just read it over when they get them, but to do a variety of other tasks which I'll talk more about in a moment.

And metacognition is another concept that I will come back to.

So load. How much is too much? This is on a lot of people's minds. Typically when we think about load, we look at information load. So how many pages, how many words, how many inches, and so on?

But it isn't so much information load that's important as cognitive load. How much mental work has to be done in order to understand the information?

So we can look at the number of mental steps, their complexity, and so on, and in some cases we can even find that something that's a little bit longer is easier to understand than something that's a little bit shorter, or vice versa.

So here's an example. This is an excerpt from a pharmacy leaflet. The source is at the bottom there, and it starts out, "Tell your doctor, nurse, and pharmacists if you," and then there's a whole bunch of contraindications, and so on.

So in the laboratory what we do is we show this type of information to people, and then we ask them questions about it either with the leaflet present or with it absent in order to test straight comprehension and memory.

The simplest question that you can ask as in any comprehension test, but a really simple warm-up question is: how many different things do you need to tell your health care provider before you use this information?

So you just saw that last display. How many different things were there, approximately?

Eight. Thank you, Tom.

Most people say seven or eight because of the bullet structure. Bullets are good, but a bullet is not a bullet is not a bullet. They can be used well or poorly.

This display shows that these bullets are not being used very well, and if I add this red line here, you can see there's a tendency to chunk all of the text together in a box, and those little bullet dots are floating off to the side.

There are other ways to use bullets. Let's take this same example and show it in a revised format. Even if you can't see the details here, you can see very quickly that there has been chunking, put together what goes together; separate it from other things around it; give it titles; give it some coding.

And when you look at this, and bullets have been used in a different way as well, but when you look at this, you can see there's far more than the seven or eight bullets that there appear to be to begin with. As a matter of fact, there are 18.

So people can better process the information in some formats than in others. So let's talk now about cognitive tasks.

What do people do with these leaflets? What can we test in the lab? And what do we want them to do and do they do out in the real world?

First of all, do they read it? So in the lab we can find out with different kinds of leaflets do they read it; how much time they spend. Do they read the whole thing? What do they skip, and so forth?

It's helpful to test memory also because people don't always have their leaflets handy, but what we get is a function of how we test for it. If we use a free recall procedure where we say, for example, what are the possible side effects that can occur with this medication, they don't do very well. That does not necessarily mean they don't know anything about it because if we switched and used a recognition experiment where we give, say, one side effect at a time and say could this side effect occur with this medicine; how about this one; what about this one, and so forth, then their performance goes up.

So what we get in terms of memory and then you'll see in a moment comprehension and everything else, it depends on how we test for it.

In terms of comprehension, there are a variety of paradigms we've used to test comprehension of text, pictograms, and so forth.

Problem solving tasks are essential because they go beyond the specific information given. So we can have various types of scenarios. What would you do if you were on this medication if such-and-such happened?

Search and find tasks are important. We've talked about that before.

Decision making is interesting. The decision maker is really the health care provider, to write a prescription for a given drug for a given patient.

However, when the patient gets this leaflet, can he or she look over all the contraindications and other information and say, "Yes, this seems appropriate for me, " or, "Ut-oh, forgot to tell my doctor that I have asthma," or diabetes or something of the sort.

So selection and de-selection in partnership with health care provider can be facilitated by these leaflets.

And finally, metacognition. In metacognition tasks, we can ask people how easy or hard was it to understand this information. How well do you think you understood it? Do you like it, an dos on and so forth?

But there's a problem here. Cognition is the process of knowing, while metacognition is the process of knowing how we know, being able to reflect on our own mental processes.

And what we find is there is often a very big gap between metacognition and cognition. People tend to overestimate how well they understand information, and I think that's a part of the results that we heard this morning with consumers rating comprehensibility type measures, a high in some cases. I doubt whether they would really do well on a true comprehension test.

So we can take a look at all of these cognitive tasks. We can see how well people do in the actual cognition tests in the laboratory and also in actions out in the real world.

We can look at accuracy, and we can also look at errors. So when they get something wrong, what kinds of errors do they make, and are those errors likely to have serious health consequences?

So now that we know format is so important, how do we go about selecting appropriate formats? Well, obviously we start with the usual content, the indications, contraindications, warnings, dosage, side effects, et cetera.

But that's only part of what we need to do. There are other dimensions involved, and at least two others are worth talking about today.

Format. We have options for each chunk of information. Shall we present it in text? That's the left to right cycling of words across the page, or a list in outline. Other types of representations we've developed in my lab, fans and trees, and so on.

The point being is that you can take the same information and when you present it in one format people might not do well with it. Okay? Why is that? Is it hard? Sure, but it might be the wrong format.

We have then switched formats and gotten dramatic improvement in people's ability to understand the information, and it sometimes is dramatic as an 80 percent improvement.

Finally, we have to make sure that we're serving all of the various types of cognitive tasks that people are going to do with these materials, such as being able to find and understand, remember and use the information, and it can happen that you have a certain combination of content by format, and that looks great, and you do a comprehension test in some way, and it looks like they understand and you feel good about it.

However, later on people might not be able to define the information, remember it or use it in an accurate way in everyday life. So how do we select a given format for a given chunk of information?

There's a tendency to start with a format and stick the content into it. That's one strategy. Another strategy is to start with a content, look at it carefully. Is it descriptive? Does it have a list imbedded within it? Is the list long or short, et cetera?

Then go try a format. Does it fit into Format 1? If not, try another. Does it fit well in Format 2? Not so good. And continue until there is a good fit, and so cognitive accessibility is maximized.

So, in conclusion, there's a lot that we can do to insure that specific information is present in these leaflets, in the approved professional labeling on the Internet, on TV everywhere. We can make sure that it's physically present.

However, if people cannot find, understand, remember and use this information, then it is functionally absent. So in the year 2000, we get Dr. Svarstad to redo the study and so on. What would it mean if we used the current research methods which get us to a certain point, and we came up with 100 percent adherence on all the criteria, even if everyone put in and modified those criteria to everyone's satisfaction?

The information could still be functionally absent. We must have materials designed based on cognitive principles and submitted to full comprehension testing.

Thank you very much.

CHAIRMAN GROSS: Ruth, thank you very much.

I think at this particular point questions can be asked of all of the presenters during the public hearing, as well as Dr. Day.

I'll start off by asking Dr. Ratto of First DataBank.

As an example of one of the DataBank vendors, how do you view the Keystone criteria? Are they used on a regular basis? Any comments on them?

DR. RATTO: Yes, the Keystone --

CHAIRMAN GROSS: Why don't you go up to the microphone if you don't mind.

DR. RATTO: When the Keystone guidelines ere established, we had incorporated information related to these guidelines. For example, we completed an enhancement in approximately April 2001 where we added the explicit warning section, whereby in previous editions of monographs we would have precautions and drug interactions, et cetera, imbedded in those various sections. We created an explicit section that essentially consumerized any boxed warning information in the labeling.

We also created an overdose section. I'm speaking now to the original First DataBank product, which is what I'm here to address,a nd when we added that overdose section, we also added an other uses section, and we're in the process of segregating out off label uses from label uses.

So we had taken that into account. We have, in fact, since then also created an XML version, which stands for extensible mark-up language. We have that version which includes bullet points and some of the readability issues that were addressed. It includes the extensive customizability for our customers. They can basically choose from a number of different formats to display these monographs.

Unfortunately at least with the technology that I understand it at this point, the primary use for that would be as a Web, either Internet or intranet type environment, but there is no intrinsic reason that I know of that they couldn't be printed, other than the fact that they are obviously going to be longer in length just based on the fact that a number of the sections, most explicitly the side effects precautions and drug interactions, have bullet point list items within them at this time.

And that's a product that we just released several months ago that has not yet to my knowledge had any major user involved with it at this point, but we have been touting that.

CHAIRMAN GROSS: So your latest product is compliant with all Keystone criteria?

DR. RATTO: What I'm saying is that the latest product incorporates all of the formatting issues -- a number of the formatting issues that are in the Keystone guidelines.

What I'm saying is that we have our editorial policy structured such that we have incorporated to our view the Keystone guideline criteria, and what we need to do is we're going to take a look at the scoring guidelines that just came out with Bonnie's report and incorporate any additional information that needs to be added to those.

And what we need to do and what we're in the process of doing is going back to our monographs and populating that information through all of the monographs.

CHAIRMAN GROSS: So at the time Bonnie's study was done, however, maybe the information sheets, the CMIs, were not totally compliant?

DR. RATTO: Correct.

CHAIRMAN GROSS: Dr. Cohen?

DR. COHEN: Well, that's more or less what I was going to ask.

Currently, can you say that all of your materials -- you're probably the leading provider of this drug information to pharmacy computer systems. Can you say that all of your material would contain all of the black box warnings that exist, all of the important side effects, all of the prominent contraindications for drugs?

DR. RATTO: What I need to state first off is that we are -- when this study was done, there was a -- the information within the company was, as Bonnie had said, segregated into separate versions that depended on the Medi-Span products as well as the original First DataBank product, and there is currently a divestiture process going on within the company related to the Medi-Span product, and by FTC regulation I really can't comment on the Medi-Span portion of the database.

I am here to comment on the original First DataBank portion of the database, and I can tell you as we just alluded to that we did go through -- there was no way to get a comprehensive list from any one source of all the individual products that have boxed warnings in the labeling. However, we made a good faith effort by going through the PDR, the GenRX source, and going through also at the time one other secondary reference source that had a number of the boxed warnings listed and tried to encapsulate every one of them that we could find.

So I'm confident that we have in the high 90 percent range, if we don't have 100 percent of them, already summarized, and we have -- what we are working on, as I said, is reviewing monographs that were created in the past.

CHAIRMAN GROSS: Well, perfection is tough, but pursuing it is certainly worthwhile. What would you propose as one of the major data vendors that all of the data vendors do to try to be as close to compliant with all of the eight categories and it subcategories to maintain this as an effective voluntary program so that there isn't pressure to remove the voluntary status?

DR. RATTO: Something that we have informally discussed with FDA and are pursuing now is dialogue feedback with the agency, and hopefully that will take the form of discussions with Dr. Svarstad and some of her groups. Certainly the entire group would be a difficult procedure.

But we have some discussions that are set up for tomorrow. I'll be visiting FDA offices, and we want to -- from our perspective, we want to get this off the ground in terms of having constructive dialogue interchange, and we obviously are working towards implementing our current editorial policy through our entire database from our perspective. We'd like to bring other stakeholders to this sort of dialogue and point-counterpoint quality improvement efforts with the agency.

Obviously that's going to be up to whichever providers are out there besides ourselves, but we certainly would welcome the participation of everyone involved for the overall improvement of the quality of the monographs.

DR. SULLIVAN: I was just wondering whether you have what you currently have in place, what sort of quality assurance or quality control. For example, do you ever go back to the sponsor companies to check with them whether they think that's appropriate or do you just hand it out for peer review or do you have internal people that audit what you're producing?

DR. RATTO: Well, first of all, we've had at least ten to 12 years of field testing, if you will, from the standpoint that all of our monographs, whether they are looked at in a physician office by a physician or a pharmacist or a patient. We have gotten feedback from those end users, if you will, and also our software vendors. Information is passed on through them, questioning either the information or perhaps the inclusion of additional information.

Basically any questions that are raised out in the field, whether they are validated or not, we will review them and take them under advisement. We respond back with information giving the reasons for the inclusion of that information or stating that we will consider that particular information.

With any off label uses, for example, we'll substantiate that with literature information usually through perhaps a secondary reference source, and I had earlier mentioned the AHFS drug information because they do authoritative literature searches for off-label information, but we are focusing on labeled uses in the uses section of the monograph.

We, along with that, we do monitor MedWatch. We do have information from manufacturers. We don't have a pipeline with every single manufacturer, but we do encourage their drug information pharmacist to send us information as early as possible if there is labeling issues that they have in terms of, you know, upcoming changes to their labeling, that sort of thing.

CHAIRMAN GROSS: Jackie.

MR. LEVIN: Just a point of information. It is my understanding that it is not -- that the law does not permit the inclusion of off label use in medication guides or in whatever we want to call these things; that to be scientifically accurate, they have to represent the information in the product label. The product label does not include off label use by law.

DR. RATTO: Actually the way the action plan criteria are set up, to my understanding it is that you may customize a monograph with off label uses, which is why we have an other uses section, which is where we're in the process of segregating out our off label uses. So that that part of the action plan criteria will be met because, you know, we have essentially reviewed all of the criteria that were set up within the action plan, and again, we want to look at the scoring guidelines as well and make sure we've incorporated all of that information into our policies.

CHAIRMAN GROSS: Jackie.

DR. GARDNER: Can you enlighten us with the logistics of the process by which even a perfect monograph with all of the criteria gets to the consumer from you?

We heard today about the issue of the vendors being -- I beg your pardon -- the software companies being a black box that things go into. Can you tell us something about licensing from the standpoint focusing on the logistics?

If it leave you, it goes through the license. How frequently are they updated with the software vendors? What kinds of options do they have to make changes, things like that? So that we get some idea of what the process is and the time frame?

DR. RATTO: Well, the first thing that I would have to say is that I don't have all of the details on that, but I'll give you what I know at least as a skeleton, and we are working with Dr. Svarstad and the FDA in terms of trying to elucidate that information as much as possible because, frankly, we don't have all of the information as to what our software middle men, if you will, are doing with the data.

What we do have at this point is a contractual statement that says that they will not alter that monograph in terms of deleting information. That's happened with all new contracts and contracts up for renewal. And that's our attempt, and I think it's, you know, basically an effective attempt to make sure that the information is being given to the consumer in the proper format and with the proper content.

Now, admittedly, that doesn't control the font size, for example, and that should be a concern for, you know, everyone in general, and that's something that is another story. I'm surprised to hear that most monographs came out of one page because when they're in the ten point font, which is what we send out, they certainly don't hit one page. So that is an issue.

What I would say is that as was mentioned earlier by Dr. Svarstad or actually I think it was John Coster who said that the monographs are delivered to either software vendor or directly to large chain pharmacies, depending on if they have their own processing system.

From that point, basically, you know, to be frank our control is not there in the sense that the control we have, if you will, is in the contracting, and beyond that we don't dictate. We do dictated in the contract that they do not delete information, but we don't sort of, if you will, squire the monograph all the way down to the patient level. So --

DR. GARDNER: But you said delivered. How often are they updated if there are updates?

DR. RATTO: Oh, okay.

DR. GARDNER: What's the process?

DR. RATTO: Okay. Sorry. That's another point you made.

Our process is set up such that we have the capability of updating people on a weekly basis. What I mean by that is that is available to all of our customers, a weekly update for clinical data.

There are customers that receive that information monthly, and frankly, we don't know how often. I imagine there are a few cases. I think that was brought up again earlier where customers may not update them I think it's very infrequent, but I imagine there's still a few people out there that are only updating quarterly.

That's something, again, that's out of our directly control, but we certainly offer weekly updates and encourage people to go with weekly or at most monthly updates.

CHAIRMAN GROSS: Bill.

DR. CAMPBELL: That was the question.

We heard some comments about variability of the information product that will get to the consumer as in updated information weekly, biweekly, monthly, and so forth, based upon price to the pharmacy, the community pharmacy.

And we also saw evidence that the same vendors were providing leaflets that might be less than 5.6 inches, greater than 5.6, and so on and so forth.

So the question: is there a disincentive at the pharmacy level for providing full information to consumers based upon the price of the leaflets?

DR. RATTO: Let's see now. My feeling would be no just based on the fact that we -- I'm here discussing essentially the original First DataBank product. There's only one of them at this point.

We discontinued the short monograph product. So that particular product is available in its entirety basically at one rate, and to my knowledge, I don't know the -- I shouldn't say that I know. I don't know whether there's a difference in weekly versus monthly updates. There probably is in terms of pricing, but I don't know that for a fact.

Certainly the new XML format that I mentioned, that's available at no extra charge. So I guess the bottom line is that the monograph that I'm referring to is basically a one price monograph, but again, I don't know what the pricing structure is for monthly versus weekly updates, if there's a difference there or not. That's in the sales and marketing area.

CHAIRMAN GROSS: Okay. Michael.

DR. COHEN: Yeah, I want to go back to something I asked about earlier or talked about earlier, and that is how rapidly you are capable of updating your own information system. Subsequently it would be made available to the others as we've just been discussing, but we've had a number of reports over the last few years of inordinate amounts of time to get important information into the system.

I mentioned cisapride. There have been other issues as well. When something is published in the literature, when there's evidence that there's a serious problem, it sometimes takes quite a bit of time to go through the process at FDA to get it in the actual labeling.

Are you able to respond to published articles where you believe that there is a serious problem or do you have to wait for a change in the labeling?

I've been told that that's the case. I've also been told from people in the database vendor companies that a report of a death, even though it's tied to a specific drug side effect or a drug adverse reaction is not enough to trigger an alert; that there would have to be multiple reports before something like that could actually appear in a drug information system.

So I'd like to get that cleared up because I think that's an important issue.

DR. RATTO: Well, for our system specifically, we do rely on FDA MedWatch and on labeling for updates primarily.

Now, if something comes up in the literature, one thing that we did want to explore with the agency is if we notice in the course of just reviewing, doing continuing education, whatever, looking at a journal article, we notice something that has not yet hit labeling, whether it's a warning on a particular adverse effect or a precaution or whatever, if there's a contact person, we can, you know, deal with at the FDA that can tell us whether this is under active review, make sure that it's already been put into the system for perhaps an accelerated look and deal with it that way, that's what we would like to do.

But we do not have -- with the volume of literature that's out there, we do not have someone that is -- we do not have a policy of reviewing all of the medical literature, primary literature. You know, that's a situation where, yeah, it might be ideal if you had --

DR. COHEN: Well, actually some of these have been situations where I know that practitioners have contacted the company to request that this be added.

DR. RATTO: Okay. Well, clearly I'll say this. If someone has contacted us with specific information at First DataBank and said, "Such-and-such drug interaction appears to be an issue. We want it to be investigated," we will do that. We will put that through our process promptly, and we will coordinate with the drug interaction people because there is a specific group that handles drug interactions that's separate from patient education per se, but we'll, you know, communicate with them, and that information is processed when there is a specific inquiry such as that.

CHAIRMAN GROSS: I have a question out of my ignorance. Is there a person at the FDA that First DataBank and other vendors could relate to when there is a change in licensing based on some complications so that they would have the information? Is there some kind of a communication that could be set up?

DR. TRONTELL: I think, as Nick Ratto has just described, there is the MedWatch program, which certainly publicizes and announces those formal actions that the agency has taken in terms of relabeling or "Dear Health Care Practitioner" letters that are sent out.

It's more complex in the area where there may be still yet some ongoing assessment of a particular safety signal. We can certainly do our best to establish such lines of communication, but when the agency hasn't yet completed its assessment, we may be in a difficult position to comment.

DR. RATTO: What I was specifically referring to is if there is some statement. For example, there was a consensus statement by the cardiology society, American Cardiology Society, recently related to doxazosin and hypertension use and having problems with patients developing congestive heart failure and other cardiovascular issues.

And we were in a bit of a quandary as to whether to include that per se just based on the statement that was made, which was, you know, strong caution should be advised when using it for hypertension.

If we had someone at the agency that we could, you know, just basically contact to ascertain whether that particular statement or any other similar to that, or perhaps a literature inquiry from one of our customers is on the radar screen essentially, that is something we were planning to discuss.

CHAIRMAN GROSS: Okay. Stephanie.

DR. CRAWFORD: What consideration, if any, has your company given to making some arrangement through which this information could be put on the Internet, directly accessible by the ultimate patient consumer?

DR. RATTO: Our company, in particular, is represented on the Internet by, I believe, Medscape, and there may be other relatively smaller users as well, but that one in particular comes to mind.

CHAIRMAN GROSS: Okay. If there are no other questions, we'll let you sit down. You're obviously in a very critical position in our discussions.

Does anyone have any questions of any of the other speakers? Yes, Michael.

DR. COHEN: For Dr. Day, considering all of the information that we're trying to jam pack into these patient information leaflets, do you see a role for icons of some sort? Is there a way to use icons to benefit information provision or communication?

DR. DAY: The answer is yes and the answer is no. It depends on how you use them. Pictograms can help, and there is a library of USP pictograms that have been tested in various ways. Usually they've just been tested, given to people and say, "What does this mean?"

I don't think that's adequate. There are a bunch of other things that need to be done. So a variety of tasks, and some of the testing that we've done, we've looked at what happens when you look at the pictogram versus a pictogram in the context of the leaflet, and when the leaflet does or does not have text, that is the meaning of the pictograms, and it's nearby.

So if you fulfill all of those things, if you have a pictogram and the text nearby, that's the maximum situation or the best situation.

And pictograms are potentially helpful for people who don't read well or perhaps have not very good English and so on and so forth, but then there are cross-cultural differences in the interpretation of pictograms.

So there is research going on elsewhere in the world where you take the USP pictogram library, and then you vary it as a function of the way your local icons would have them. For example, the way we package milk, if you're supposed to take or not take something with milk, our milk cartons look different from the way they do elsewhere in the world. So people wouldn't recognize them, and so on.

So, yes, there's a role to play, but every time you make a suggestion to add something like a pictogram, have you lost something else? So what didn't you put a pictogram on?

And so if you're not supposed to take it if you're pregnant and you use that with a really understandable pictogram, then does that mean you're going to decrease knowing something else?

So the answer is, yes, if it's done well and tested carefully.

DR. GARDNER: Ruth's comment reminded me that I wanted to ask Nick and the other vendors if any of this material is available in other languages at all.

Does anyone subscribe to the Spanish language version of your service?

DR. RATTO: We have our product available in Spanish, and we estimate within the next four months we'll have a French version. We're in the later stages of negotiating for a full translation with that, and we've hired a translator.

So that's primarily now for our Canadian customers, but it could be for any other French speaking.

CHAIRMAN GROSS: Okay, and believe it or not, we're done a little bit early. So what we will do at this point is take a break for 15 minutes, and we will reconvene and the committee will consider the three questions that are attached to your agenda.

So 15 minutes, and we'll reconvene, and that will be at 3:00 p.m. we'll reconvene.

Thank you.

CHAIRMAN GROSS: Okay. I think we're going to get started. We were a little bit ahead of schedule. We don't want to fritter that time away. So if everyone could take their seats, we will get started.

The main purpose of the remaining session is to consider the three questions that are attached to the agenda. The first question is: what additional analyses of the FDA, NABP, Svarstad study do committee members suggest should be done to answer any remaining issues about the adequacy of patient information?

So I'll entertain any comments from the members at the table. Sharlea.

MS. LEATHERWOOD: I just might ask. There were several comments, and I certainly wasn't aware of this, but there were comments that there were certain criteria that maybe were not appropriate. There was no basis for putting those particular subcriteria in the evaluation form.

And so I wondered if we should make sure that all of those criteria were based on something, and if not, then drop criteria and reevaluate the data.

CHAIRMAN GROSS: Well, there are a lot of criteria there, but, Bonnie, do you want to take a stab at anyone's?

DR. SVARSTAD: I think it's certainly possible. I mean, there are always judgment calls on this in the sense that the expert panel was working with the Keystone criteria, on the one side, and the approved labeling, on the other side, and how to interpret those is a judgment call.

And we certainly have the ability to drop items and reanalyze without certain items. So if the committee wanted that, and I think I, in fact, offered to do that to the FDA staff. That's one issue.

The second issue is that there may be items that are high or low priority, and it's possible to reanalyze the scores, eliminating low priority items.

But our mandate was to try to interpret at least the action plan as well as we could.

CHAIRMAN GROSS: Yeah. I mean, there are two purposes here. One is this was a research study that soon will be published, but the other issue is what action flows from these results.

Brian.

DR. STROM: Yeah, just in follow-up, a number of the public speaker were making comments about some specifics about the study. I would urge you not to bother to go back and readdress those specifics. I think, as you said, there are always judgment calls. There are always gray areas. One could argue about one point one way or the other.

None of that is going to change the substance of what the finding is or the findings were, which I think is very important, and I think what we need to worry about is the substance.

This was a study. This wasn't regulation. This wasn't saying you have to have that particular statement or you have failed regulation. Part of the problem with regulation is it ends up being too rigid.

And so I wouldn't want our focus -- I wouldn't want to generate a lot of undeserved work for Bonnie, and I wouldn't want our focus to be distracted from the larger findings of the study by worrying about what amounts to small technicalities that, if changed one way or the other, wouldn't change the bottom line answers.

CHAIRMAN GROSS: I suspect general reviewers will take care of a lot of that.

Yes, Ruth.

DR. DAY: Given the analyses that have been presented, there's quite a bit here. There's a lot more that could be done, and that was my understanding of what this question is about. What additional analysis --

CHAIRMAN GROSS: Correct.

DR. DAY: -- of the current data set?

And I made a list of a whole bunch of them, and then in talking with Bonnie I found, oh, she's already doing those, and so on.

One, in particular, I think the factor analysis of all those different criteria would be very interesting so that we can see what of all the various subcriteria cluster together and whether they do fit, and what underlying factors emerge, and if those are the same ones that are intended by the categories of the criteria, and that would do a little more to tell us about validity.

I think you've shown us a lot about reliability of the instrument, and we need to look a little bit more about validity in the sense of is it measuring what we say it's measuring. So that would be one thing.

Another thing, in the consumer data in the briefing book there was a partitioning of the first set of items and the last set of items, which made some sense. I would like to invite the researchers to reconsider that and repartition them in additional ways.

For example, putting together all of the ones that are about metacognition. So the remembering part was up in one category that you looked at and the others down somewhere else. So just relook and see if a different partitioning of those might be useful.

And then finally, just to mention one more, you mentioned that you didn't do the inter-rater reliabilities for the consumers because they varied so much, whereas you did that for the experts. I agree they vary a lot, but I think we need to document that.

Because any time there's information out, a given consumer says, "I don't like it that way. I want it this way," and somebody else says, "I don't like it that way. We need to do it that way>"

So what is that spread of reliabilities in the consumers versus the experts? And then if you could do some reliability within categories of consumers that are important, such as by age or by gender or by whatever seems useful.

I think there was a little bit more data mining there that we could get that would be useful.

CHAIRMAN GROSS: Okay. Arthur.

MR. LEVIN: I just had a cookie. I should be able to turn the light on.

I want to go back sort of a little more to the foundation question, which is the adequacy of patient information and what answer we have to that question.

I would say that the answer we have is it's not, and I would argue that the information we have from this study if we look at Public Law 104-180 would actually trigger the last part of that, Part E, which says not later than January 1, 2001 -- we're a little behind times -- "the Secretary of the Department of Health and Human Services shall review the status of private sector initiatives designed to achieve the goals of the plan described in Subsection A. If such goals are not achieved, the limitation in Subsection D shall not apply, and the Secretary shall seek public comment on other initiatives that may be carried out to meet such goals."

Now, I think there are lots of other initiatives that can be carried out to meet those goals, but I think we need to have a proactive process because we've been going now for more than two decades, and for those of us who have been on this issue for all of that time and maybe more, we're always coming to the same place, which is we have these huge gaps of time that go by.

And then when we go back and take a look, we find that the private sector initiative has not done the job.

I mean, I think it is really appalling if they can't get font size right. That's not rocket science, when everything, the med. guide, proposed reg. of '95, the Keystone plan, talks about, you know, how to make things readable in terms of appearance and somehow that doesn't translate to action in 2002. I think that's appalling, and it's a real failure on the part of this private sector effort.

In page 20 of the Keystone report, useful is described in the following way. "Prescription medicine information shall be useful to consumers."

"Useful" is defined as enabling the patient to use the medicine properly and appropriately, receive the maximum benefit and avoid harm. And I think what we've seen from this study is that we fail -- I mean, the effort has failed to meet the goal as set of 75 percent useful information by 2001.

By this definition of useful, by the results of the study, we're not there. It seems to me the law is clear and calls on the Secretary to take certain actions, and I think that's what our conclusions should be.

CHAIRMAN GROSS: Well, following along with Arthur's comment, I guess I would ask the committee to consider the question: should we ask the data vendors to present a joint proposal as to how they're going to comply with the Keystone criteria and then monitor that in a year or two to see if that's happening?

A number of people have brought up the issue that the system hasn't worked quite as well as it should. Should we, rather than sort of a helter-skelter approach, should we ask for a joint proposal from the people responsible for providing this information?

Bill.

DR. CAMPBELL: Let me go back to responding to the additional mining of data and also respond to that question.

Three and not momentous items, but I think the issue of experts rating readability and consumers rating readability is still a little unsettled, I guess, in my mind because the experts are, in fact, the consumers when you come to this point. I would rather know what the consumers' rating of readability is and call that the expert than the expert's professional reading of what consumer readability is.

I just think that ought to be revisited with a little different take on it, I think.

I didn't see a slide or table, I didn't think, that showed the distribution of leaflets by size. I saw them by size by product, by size by vendor, and so forth, but globally. Maybe that was there and I didn't see it, but that was some information I'd like to see.

And lastly, I just continue to be troubled by a bit of these structural issues that impede the movement of optimum maximum information to the pharmacy level, such as a vendor updating daily or weekly, but it not getting to the pharmacy except quarterly, and issues of that sort.

And as Sharlea mentioned, cost differentials. I would like to know if there are differentials; if there's a relationship of any kind based upon the rapidity, accessibility, and frequency of updating and that sort of thing with the other measures, global measures, of compliance.

And to your question, Peter, my suggestion is we need a Keystone II. I think we really need to convene a Keystone II, not just the vendors, but it's clear to me there is a difference of opinion in many circles on the interpretation of the original Keystone. So I think we need to really revisit that report and clarify and interpret what was intended.

And then I think that group should be charged to release a Keystone II report that would take these criteria and subcriteria and validate that they are, in fact, the appropriate criteria for use in measuring.

And those criteria can then be turned over to the group you suggested, the vendors, who will then have a template for implementation, and there will not be this sort of arguing and disagreement that, well, Keystone said this, but they didn't mean it, or they didn't mean it and they said it, and so forth.

A very important issue has to do with the labeling. Is the labeling the gold standard from which it should not depart or is it intended that the Keystone include it off label in other sources of information?

So I do believe Keystone II is appropriate.

CHAIRMAN GROSS: A question about Keystone II. If we do Keystone II, you mentioned the word "validate" the criteria, and validate could take several years. Could we have some discussion on that? Is that something that you want to do or can we take the Svarstad study and update Keystone and go from there and then have that checked and validated later on?

I mean time is a question that has to be considered.

DR. CAMPBELL: I didn't mean validate with outcomes data.

CHAIRMAN GROSS: Okay.

DR. CAMPBELL: I don't think we have time to do that. I think we have to face validate from the consensus, and then I think part of the Keystone Group recommendations, I would hope, would be a 2004 evaluation that would precede 2006.

CHAIRMAN GROSS: True.

DR. CAMPBELL: So we would have a fast turnaround to see where we are with that.

CHAIRMAN GROSS: Jackie, do you want to comment on that?

DR. GARDNER: Like a broken record, I guess. I'm back to the issue of our charge is related to risk management and safety, and in particular, I've been interested both from the background materials that we were given today about what was the resistance to the first or to the patient package inserts from the professionals, and then as each group came through today we heard about how you really can't bombard consumers with too much safety information because, you know, they just can't absorb it or they don't want to and so on.

I would like to have if we're going to do a Keystone II or something in the interval to have a good deal more consumer input into this question. We clearly from Bonnie's study -- even with what we do have, are not meeting those criteria related to safety communication, risk communication in these inserts, I mean, these leaflets.

And the question is: how are we going to do that? How are we going to meet that need?

And I think we have to find out from consumers, not from professionals and vendors, how much is too much and how do they want to see it? What way is an acceptable way to learn about these risks?

And I'd like to see more work done on that with the people who have to bear the brunt.

CHAIRMAN GROSS: Bonnie, do you have any sense of how much is too much from what you've done?

DR. SVARSTAD: Well, I think all of this is very complex, you know. Those of us that have been in patient information, Ruth and others that have been studying this, I think it's very hard to take an issue with as many complexities and end up saying because we can't agree, let's make the consumers make that decision on how much is too much.

That's not to say that we couldn't learn a lot by studies of information overload, but when I look at the bulk of these leaflets, I don't think there's a problem with information overload, quite frankly.

And I'm sorry, Bill, that I didn't have the data on the distribution of leaflet length. I have that. I just don't -- it's up in the room, you know. But the bulk of these are less than one page, certainly less than one page. And I think I did give some statistics that many of them were less than five inches.

I guess I'm also a little unsure about what a Keystone II would accomplish unless you really, really focus in on establishing priorities of the criteria, and I would agree that there might be criteria in here or subcriteria that you could in a consensus building with professionals and consumers, just as you did the first time around, saying, "Okay. Now we've gone through this," and we could give these items or these criteria or these subitems more priority than others in the interest of still staying within a reasonable length for consumer information.

But these are very difficult issues, and I think it's really hard to put it back in the consumer's lap because I think they will end up saying, "Do we want to know drug names? Yes," or if they said no, what would we then say?

I'd say you should know them because the studies show that you should know them, and you'll make fewer errors.

Contraindications, do they want to know about that? Well, I think we heard one consumer talk about that her mother would have benefitted, and I think we can all think of people who would benefit.

So even though there might be a few consumers who say, "I am scared by this information," the other consumers will say, "Well, we want to know it."

There may not be a consensus. We act as if there's a consensus among consumers. There isn't. They're like professionals. They have difference of opinions. They have different perceived needs.

So this is a very complex thing to come to. Specific directions? We know that specific directions reduces errors. Would I want you to go back and backtrack and revisit that issue if a consumer said, "No, I don't want specific directions"?

I'd say, "Oh, gee, that's taking about ten steps backwards."

Side effects? I mean, every consumer survey you read out there by sociologist, health service researcher, psychologist will say consumers want to know about side effects. We don't need anymore studies to know that.

So, you know, I'm not sure what you would get by additional surveys unless you were to really talk about things like formatting. I think Ruth's point here about formatting is a good one.

CHAIRMAN GROSS: Ruth, any comments on what we've been talking about?

DR. DAY: Does amount of information matter? It depends on how you show it, and so asking people do you want more or less of that, until you show it to them in different ways, I don't know what the answers mean.

CHAIRMAN GROSS: Okay. Jackie.

DR. GARDNER: I guess thanks for clarifying what I was trying to say, both of you, which is I don't know. I wasn't even suggesting surveys. I mean it's clear from your data, Bonnie that whatever it is, we're not doing it right if it's amount, if it's format, if it's whatever it is. I think that's the area that we need to focus on because those are the areas that are important for us in managing risk.

So I don't know that we need more surveys then. If it sounded like that, I hadn't thought it through, but I do think that that's the area where we need more information, however we get it, and we're failing to do that.

CHAIRMAN GROSS: Yes, Ruth.

DR. DAY: Just to follow up to what Jackie is saying, I agree we need more information from consumers, but I think it's about true comprehension and, you know, problem solving and then some perhaps actual use studies. I guess we'll get into that when we talk about recommendations, but there were DOA (phonetic) to do some post market surveillance and so on with this format versus that form, et cetera.

So yes, consumers; no, not more surveys.

CHAIRMAN GROSS: Okay. Arthur.

MR. LEVIN: I would certainly argue against a Keystone II. First of all, it took a statute, an act of Congress to get Keystone I, and I don't think we're going to get Congress to stipulate a Keystone II, and there's a whole history there which I won't bore you with, why we had a Keystone in the first place and where the statute comes from.

It just seems to me that it really depends on how you view the importance of written information for consumers. I think there are those of us who see this as the ultimate safety net; that for lots of reasons unfortunately well documented in the literature, the amount of counseling by prescribers is minimal. The amount of counseling by dispensers is minimal, and so what we're left with is a written piece of paper maybe. The only thing standing between the patient and harm, the patient optimizing the benefit of the drug or whatever.

So I mean, to continue to have this argument is just beyond me. I don't understand it. No one is suggesting, as people kept suggesting we were suggesting, that the written information is supposed to supplant physician or prescriber counseling or dispenser counseling. I think if prescribers and dispensers were doing the right thing, we might not need a written piece of paper.

But unfortunately, we know for a variety of reasons it just doesn't happen or it doesn't happen with enough certainty and frequency and adequacy to protect patients.

So I look at that piece of paper saying if you were being given a drug, what is it that you would like to know if you knew nothing else. What are the few, one, two, three bits of information that would be most important to you as a patient?

And to me answering that question says to me those are how to prevent harm and how to use the drug to optimize benefit. And if I came away with nothing else, that's what I'd want to know.

Why is this so complicated? I mean, we know how to do this, and I think the problem is we haven't had the will to make industry do what we want them to do, and every time we get close, we have opposition that pushes it further back and we're told, "Leave it to the private sector.

I think after 20-some odd years we have to say, "Why do we want to leave it to the private sector anymore? They haven't gotten it right. We have to change the way we do things."

So to me, the only responsible action in terms of the study, the survey, and the public health law and the public law is to make that Section E come to life, and that is to say the Secretary -- the January 1, 2001, which is now 2002, survey shows that we have not met the goal.

And if we haven't met the goal, the Secretary has to begin to take initiatives to meet that goal.

CHAIRMAN GROSS: Okay. I'm going to take the Chair's prerogative and ask us to move on to Question No. 2. Much of our discussion is going to be an overlap of a number of these areas, and I'd like to go through each of the questions, hear discussion, and then let's come up with some recommendations at the end of that.

I think that will be easier than dissecting it out artificially.

Okay. Question No. 2 says: what additional research does the committee recommend to document the areas and means for improving written patient medication information handed out by pharmacists?

The committee may wish to consider the following: the action plan or Keystone criteria and its subcriteria of usefulness and ability to assure maximum impact on appropriate patient use of prescription drugs.

For example, can individual criteria be analyzed to assess their impact on patient knowledge or behavior?

Methods to determine if Keystone criteria and subcriteria should be prioritized or others added or deleted.

And finally, the influence of overall length of written materials on consumer reading and comprehension of materials.

Some of this has been discussed already. Would anyone like to comment on that question? Yes.

DR. CRAWFORD: Not the subquestions. I would like to make a comment on the overall Question 2 about additional research to consider just so we don't miss some of the things we've said before.

I have questions about what are the barriers that are preventing some of the independent retail pharmacies from being at the same level and giving out some information at least as to change.

It's been highlighted quite a bit here earlier this morning that we're missing a big part of the information. Depending upon what source you look at, it's estimated that mail order pharmacies, although they're small in number, they are filling about 13 to 15 percent of the out-patient prescriptions. Id o think we need information from that segment both on the distribution and usefulness of the information they provide, as well as what the patient consumers feel about the information they receive, its readability, et cetera.

And also so that we don't lose the point, from the consumer perspectives which are critical to make sure we get a wide spectrum of consumers, patient consumers, in that process.

CHAIRMAN GROSS: Okay, Stephanie. Thank you.

Any other comments? Questions? Yes, Ruth and then Michael.

DR. DAY: In terms of improving the pharmacy leaflets, what content areas do we need to look at? I would say risks. All of the different risks really need improvement

There are a lot of things that were found wanting. They're so easy to fix, for example, making sure that the date off the leaflet is on there. I mean procedurally out in the real world, it gets a little difficult, but that's an easy thing to know how to address.

But what's more difficult is what are the most effective ways to communicate the different types of risk in written format, and it may be that that's just more difficult information, maybe not, but by exploring different formats for doing that, I think we'll make leaps and strides.

CHAIRMAN GROSS: Michael.

DR. COHEN: Yeah. I guess throughout the consumer movement we've heard comments about consumers being in a position to prevent a lot of the adverse drug reactions, and that's what this whole thing is about obviously, adverse drug events, but one area which is dear to me is medication errors.

And I think, you know, from our experience with the error reporting program, also with FDA's MedWatch program we have a lot of information in the database about recurring serious medication errors that I think we could communicate information to patients about and put them in a better position to prevent some of these.

A good example would be we've had a serious problem with giving drugs that are intended to be given weekly on a daily basis. Methotrexate is one. There have been several fatalities as a result of that.

To me any prescription for methotrexate for immunomodulation should be accompanied by information that would warn patients, you know, that this is to be given weekly and not on a daily basis. So that is just one example.

There are certainly others, and I'd love to see something built into your research to test. We have the information. It's just a matter of putting it together, and in fact, we've already been talking with the Office of Drug Safety about a project where we would actually go back into the database and try to pull out the most important medication errors that patients should know about or physicians should know about, pharmacists, et cetera, and develop that into a database that could be used.

So I'd like to see that.

CHAIRMAN GROSS: Yes, Bill.

DR. CAMPBELL: A lot of people have provided comments that I think were extremely valuable, and my problem is I've agreed with all of them, and you can either look at this situation as approaching 90 percent in terms of quantity and the ability from some of Dr. Svarstad's data providing a very doable leaflet that will achieve 100 percent in terms of qualitative measures.

So you can read that as saying we're, if not there, a step away or you can read it that it is highly -- it is very little increase in terms of quantitative and no increase in qualitative.

And I think the lynch (phonetic) here is what's the meaning of the criteria and subcriteria. That's really what we're missing. That's the way you close that confusion.

And it seems to me we have to -- I apologize for the term "Keystone II." I didn't mean it literally.

We have to go back, look at those criteria, and determine what they mean and validate them in terms of communicating them to others.

I used the HIPAA example earlier. No vendor would come in today without saying, "We are HIPAA compliant in what we're providing you."

And yet we have people saying that the vendors are not even using the term "Keystone" and clearly don't recognize it. So our problem are these criteria and subcriteria that have to be revisited and either revised or accepted.

CHAIRMAN GROSS: Jackie.

DR. GARDNER: I'd like to echo and be more specific about one part of Stephanie's suggestion that in looking at the barriers with having community pharmacy access and provision of adequate material, specifically the software vendors, what work can we do there?

It doesn't matter if First DataBank creates the perfect documentation, if it gets somehow diluted out before it gets to the pharmacy level.

CHAIRMAN GROSS: Yes, Sharlea.

MS. LEATHERWOOD: I might try to respond to that a little bit. That's where I think we have a big problem because I, again, focused on the 89 percent who actually had changed their behavior from 55 percent giving it out in I believe it was 1996 to 89 percent giving something out in 2001. So there was a behavior change, and that is such a difficult thing to achieve.

So that has been accomplished. I believe that the information that I've been giving out at my pharmacy was the correct information. It wasn't, but I believed that it was.

So I think pharmacists are in a very difficult position because we're given this monograph from our software vendor, and we give it to our consumers. We counsel them -- I do anyway, Arthur -- and we trust that that is the information we should be given.

So just to answer your question, I'm trying to think of ways to get the software vendors involved in this so that they can then carry it on to the pharmacists and, therefore, to the consumers.

One thing would be to have a discussion with them that we were going to perhaps develop a list of which pharmacy vendors have the appropriate monographs available and which do not. I think that just even the discussion of trying to put that together would incentivize them to all pretty much be compliant because it's a very competitive industry.

So, I mean, that's just one idea, but I do think we have to work with them, those of us in organization work. We've got to work with ASAP. That is their organization, and they have to help us get there also. So somehow we have to work with them.

CHAIRMAN GROSS: Ruth.

DR. DAY: In terms of additional research needs, I strongly recommend that we be more adventuresome about considering formats for pharmacy leaflets, even one-page leaflets.

Do consider the overall look, and also whether we can use different formats for different chunks of information.

And when you do that, you then increase the visual distinctiveness of each chunk of information, which will get more attention being paid to the different parts. "Oh, what's this? Oh, what's that? Oh, that looks interesting," and so forth, and hopefully effect comprehension and behavior.

So I think a strategy for doing this is to look at the leaflets we have now and look at those content areas, generate a variety of alternative formats. Test them in the laboratory, and test them for a variety of cognitive tasks for overall ability to find and use, attention paid, amount of reading and studying, but also memory, comprehension, problem solving, and decision making.

And based on those laboratory studies, then go out to some kind of actual use or field test, perhaps a collaboration of various stakeholders, putting some, you know, Alternative 1 versus Alternative 2 out there in the real world, given that they've met certain regulatory, legal, et cetera, criteria and try and see what happens after we get some fine tuning from the laboratory studies and we now have one or more different options for a variety of information. Test it in the real world.

And there can be all kinds of testing, whether it's follow-up surveys or even surveillance data, looking in one market versus another market, where there's a leaflet of Format Type A versus Type B, and so forth.

So I think there's a lot that we can do, but we need to be adventuresome in thinking about this issue.

CHAIRMAN GROSS: Brian.

DR. STROM: I like Ruth's suggestions a lot. I think it's very important we not see the research as an excuse to delay action, and we'll come to the Question 3 as yet, but I think it's also important that we recognize that none of this is set in stone; that there's a lot of information to be gathered; that this should evolve and improve as time goes on; that we should operationalize, do a better job of operationalizing now whatever we know now, but should continue to learn information, and the kind of things that Ruth is suggesting would help that.

I would argue, for example, that I think for different drugs, different things probably should be included in the label, but you risk including too much and diluting out the real message if you have a very precise list that has to be the same for every drug. Depending on the risks from a different drug, you might have different information provided.

But that needs to be tested. I mean, I think we have a lot of unknown, untested information, and I think there's lots of opportunity here for experimentation, both in a more controlled setting and in a real world setting.

You know, if you take an example like a warfarin-Bactrim interaction, for example, where there's no question it's well recognized, the drug is out there. You know it should be used, or the cisapride example that Michael was talking about before where you know the interactions. You know it's being used contrary to interactions. It's very well documented it was being used contrary to interactions.

Try different labels in different areas and use monitoring programs to look to see if people are using the drug despite that as experiments in order to evaluate it, again, not though to stop action now, but rather so that whatever is implemented now becomes a reasonable next step, and that things can continue to improve after it.

CHAIRMAN GROSS: Michael.

DR. COHEN: Yeah, someone should point out that not all of the information comes from these drug information vendors for pharmacies. A significant amount of information gets to patients through emergency rooms where they use different drug information vendors, entirely different, and I've seen some of this information, and it's not all that it should be, believe me.

Also, we have other specialty areas, like oncology, that frequently use the manufacturer provided patient information, as well as their own patient information, and there are other areas as well.

I'm not sure how to capture these, but I think that is something that we need to take into account because there are a significant number of patients that will receive that information as an alternative.

CHAIRMAN GROSS: Jackie.

DR. GARDNER: And to get it in this section of the meeting where it was mentioned before, we need to do a good deal more work with different racial, cultural, language understanding and processing than has been done to date as well.

CHAIRMAN GROSS: Okay. There is something in the performance improvement world called PDCA cycles: plan, do, check, act, and the cycle gets repeated.

And as has already been mentioned, this will be an evolving process. Because we're not going to come to the ultimate solution is no reason not to try to seek an intermediate solution and then improve on that.

So I think we're done with Questions 1 and 2. Question 3: suggested actions to achieve the 2006 goals. This does not mean that we wait until 2006 to do it, but we work on it now.

What actions do committee members suggest to improve consumer medication information to meet the 2006 goal of 95 percent of new prescriptions dispensed being accompanied by useful written information?

Please provide opinion on relative importance, low, medium or high, and time frame for implementation, immediate, near term, or long term. Sample topics can include legibility and comprehensibility of interventions, a means to insure that technical content on warnings, precautions and adverse events are complete; means to insure that data distributors understand what is Keystone compliant; processes for implementing improvements, such as workshops or FDA guidances; and who are the critical stakeholders.

So I think this is the crux of what our day has been devoted to. We need to come up with some recommendations and consider these issues.

Brian?

DR. STROM: I'd like to propose an accreditation process. I hate to use Joint Commission as a model because I'm not crazy about the way Joint Commission works, but in this situation it may be a model that works better.

From what I heard today, my sense is it's clearly not working, and there has to be a concrete change, and whether or not it's time to go fully to regulation or not, the question is: is there any other thing short of regulation that might lead to a concrete change?

I didn't hear any from any of the testimony we heard today, specific proposals that convinced me it would change. And so let me make a specific proposal. What if there were an independent body? Nonprofit independent is fine. That's why I used the Joint Commission model, and that in order to be accredited as a vendor to provide this kind of information, get the UL seal of approval, so to speak, you need a transparency in the process of how the labels are created.

You need a clear quality assurance process where there's a clear, ongoing, peer review, feedback, feedback to manufacturers as was suggested, so that there's an ongoing reactive process in order to continually improve it.

And then you have an expert committee, like the Joint Commission site visit, which spot checks in a random sample basis for any of the vendors the types of CMIs that are being handed out and rates them on a value rating, perhaps using something similar to what Bonnie described as the rating.

I don't think it could be done uniformly. What she did was an enormous amount of work and took obviously a huge amount of work just simply for drugs and even just creating the criteria, but in a sense what I'm saying is let's use her work not only for the information it provides, but let's learn from the process she created and try to institutionalize that process.

And then there would be a numerical rating basically. I'm glad to hear there are at least two vendors in the market, which means they can compete, and they can compete based on their rating, and that rating information would be public and would be made available to pharmacies to be able to use, in turn, in competition that we are using a firm that has the best rating possible in terms of patient information available.

And whether that accrediting body is created by some existing external organization, the FDA or -- except then it would be regulatory -- or the CERTs or some other organization or a Keystone group of organizations or a primary pharmacy organization, whatever the group, it should be an aggregate of private organizations with major consumer input included as part of it as well.

But it would create an accrediting body to basically say this is or is not a viable, credible set of information for patients, and in a sense, it's one more chance between now and 2006, though I wouldn't wait until 2006 to evaluate it, to say in a way that isn't quite as extreme as regulation, but is much more coercive than just leave it up to the market to do what it wants, that they will evaluate things concretely.

CHAIRMAN GROSS: Bill, did you have a comment?

DR. CAMPBELL: Well, just to weigh in and support that. Brian has provided much more articulate commentary on what I was thinking about earlier.

USP at one time made a similar sort of proposal as a non-federal and nonprofit organization that set standards. Rather than terming it accreditation via standard setting organization, you used the term UL, and that has been an idea that has floated around at various levels.

At one time National Association of Boards of Pharmacy and some Boards of Pharmacy were looking at the possibility of part of the regulation of the practice of pharmacy in the state. This was a critical issue of the information that was coming to the pharmacies and protecting the public health to set some standards on that.

So I think it's imminently doable. I think it is logically defensible, and it has the advantage of being something that could be turned around in a short period of time, and there are organizations nationally and statewide that are very interested in moving in and doing it.

So I applaud and support the proposal.

CHAIRMAN GROSS: I agree also with Brian. I'd like to add some comments.

I think before any whatever the accrediting body, whether it's that or whether the FDA issues a guidance and then monitors whether the guidance is being followed, I think before that, we would need to have a workshop of the data vendors, the software vendors, the pharmacists wherever they may be in the community, the hospital, the emergency room, with chains, VA, wherever they may be; get together with our group, with experts on formatting and other important areas to assure effectiveness of the information.

And we can talk about other stakeholders so that it can be said that everybody who needs to know about the Keystone criteria know about it, were there, and then the FDA could issue a guidance, could set up an accrediting body however it is to make sure that these are followed.

And then if they are not followed, then some action could be taken against that particular vendor or whoever the stakeholder is that isn't compliant.

Yes, Arthur.

MR. LEVIN: Let me talk first to your comments and then to Brian's.

I don't know how many ways I can say this, but believe me, everybody who needed to know what the Keystone process was about was at the table. It was a very inclusive process which, frankly, for some of us made it an extremely difficult and painful process, but everybody was there, and if they weren't actually at the table, they were at every meeting in the chairs around the table.

That produced a consensus document that we, I think, conclude didn't do it. So I'm not sure what continually bringing people together is going to accomplish in getting the task done unless we do have some way to make it count if you do what you're supposed to do and to make it count if you don't do what you're supposed to do.

And that's what the Keystone process sort of lacked, except it did set up two judgment days, 2001 and 2006, and I think, you know, we're letting 2001 judgment day go by without making a judgment, and I think that's unfortunate.

So we have to figure out where the authority comes from to make a judgment any earlier than 2006 because that's what the statute sets up.

With regard to accreditation, you know, I have the same opinion of JCAHO as you do, and I don't think accreditation works. I don't know that the FDA has deeming authority anyway. I don't know whether I want to create the precedent for the FDA to behave like CMS and deem things all over the place, and when everybody goes around and follows up, whether it's the IG or CMS' own process, follows up the accreditors. They find lots of problems with the accreditation process.

So I understand the intent, but I'm not really comfortable with the notion that accreditation is the way to go.

We talked about a Good Housekeeping seal of approval in the Keystone process, and the vendor said no. And it certainly wasn't going to be USP because they were a vendor, and the other vendors weren't going to say that's the seal of approval we wanted.

There were lots of the stuff that you people are talking about that we talked about and were voted down on time and time and time again in the Keystone process. We talked about a sort of interactive, you know, real time evaluation process by an FDA-like advisory committee. That was one of the options we presented.

We were voted down on that by all of the pharmacy groups, all of the vendors, and all of the manufacturers at the table.

So it's painful to me to hear that these are the solutions. We talked about this years ago, and these were not acceptable avenues to go down for any of the folks who are complaining about any possible move to regulation.

So it's unfortunate that you were all not at that table because we really went through a lot of these things that we're talking about today, and these things did not get anywhere because of the opposition of information purveyors, professional associations, pharmacy associations across the board to all of these suggestions.

CHAIRMAN GROSS: Remember in the spirit of democracy we're going to have to go around the table to the advisory committee members and get your individual opinions as to what you want to do.

Ruth.

DR. DAY: Here's something you didn't hear at the Keystone way back when. That is we need comprehension testing. I've already said that today, but part of this question in our actions to meet the 2006 goals is for us to say what should we do in the immediate term, the near term and the long term, and here's what I would propose.

We need immediately looking at alternative formats for the overall leaflets and subparts within it, cognitive testing, modification, and a reiterative cycle there. That can be done very quickly.

Then the near term is to try a pilot study of actually having these formats that work out in the real world and have patients have them, and we can do follow-up testing with those patients, whether it's some kind of phone survey or actual comprehension testing.

And then the long term is to start watching changes in the surveillance data before and after such things are put into place.

So that is something that is, I think, a new suggestion relative to what went before and is now parceled out in terms of the time frame.

CHAIRMAN GROSS: Yeah, Ruth, I think what you're talking about is perfecting the form, but I think even before we get to that point we need to have all of the information from Keystone put on the forms by all of the vendors, and then we can perfect that.

I don't know that we should be doing both simultaneously.

DR. DAY: Peter, I understand what you're saying, and I accept that perspective, on the one hand.

On the other hand, this testing that I am suggesting doesn't have to have all of the real information that's going to be on all individual drugs and so on. I'm talking about formats for general types of information, like side effects.

So no matter what the drug is or how many more side effects we're going to have to have or not, and so on and so forth, what is an effective means to get people to look at it, understand it, remember it, and use it? And that can be done on a limited basis with each type of information and so on and see: do we get improvement from 40 percent comprehension to 80 percent, 90 percent? And then we can say this is a better format.

And meanwhile the other people are figuring out what the criteria are and let's massage this a little bit, and so on. But these generic forms of representation once the data are in should stand, and so I think that they're not one and then the other, but could be parallel efforts going on at the same time and then come together.

CHAIRMAN GROSS: Sure.

Yes, Brian.

DR. STROM: Let me respond to Arthur's comments in a few ways. The straw we basically have heard is 1996 there was a process, a lot of suggestions and a lot of the suggestions were voted down as you're indicating.

We're now looking at the 2001 data and 2002, and it didn't work. We are making a judgment. It didn't work. And so what we're suggesting is let's go to some of those things that were voted down and saying it's now time to do it.

That's the response to that. I think I agree with you that I'm not crazy about the Joint Commission working, as I mentioned before, but I think there's a very key difference in what I'm suggesting versus the Joint Commission. The Joint Commission has a basically dichotomous decision rule. Either you're accredited or not, and not being accredited is so drastic that they almost never use it.

And so it still changes hospital behavior a lot, but it doesn't have the ability to drive incremental change as much as you would want.

I think the rating system that I was describing, assuming there's at least two people in the market, is very key to driving and motivating that.

The last comment is you talked about reluctance to have FDA's convening authority. I think that makes sense. That is, I think your reluctance makes sense.

I think the answer if people buy my suggestion is either if private industry is saying we still want to do it, either they volunteer now to organize such a convening authority and accrediting organization in a way that FDA and this committee feels is comfortable and has teeth and is real, or it's time for FDA to regulate.

CHAIRMAN GROSS: Alternatively, there are ways that FDA in between regulation and no regulation at all; there are things that FDA can do in between that.

DR. STROM: Either way it's compelling as opposed to leaving it up to industry. I think the point now is industry hasn't succeeded for 20 years or hasn't succeeded at this point. It has to volunteer to take the next more coercive step that it was reluctant to take in 1996 that was voted down or else it has to be forced to

CHAIRMAN GROSS: Yes, Paul.

DR. SELIGMAN: Arthur, you started the discussion by correctly pointing out that the law does call for the Secretary to act, and I'd be interested in your thoughts as to what those actions should be based on your experience.

MR. LEVIN: I mean, I sort of favor a mandate, but aside from that, and I don't mean to -- you know, I'll come back to it.

I certainly think that the suggestion of some of us in the Steering Committee of the Keystone process -- it's in the report, by the way -- that there be a sort of locus of responsibility within FDA and an advisory committee or an advisory committee-like process because this advisory committee did not exist in 1996.

It's just a recent creation -- to be sort of in charge of sort of this sort of real time evaluation of what's going on out there and sort of fully engaged with all of the participants in the process, to sort of, you know, move the process along in the right direction on a day-to-day basis, if you will, rather than these big glumps of time where there's sort of like, "Okay. Do this and then we'll wait until five years and then we'll evaluate it and tell you whether it's working or not," which has led us down this path of 20 some odd years of delay.

So I certainly think that there is this coincidence that here we were in Keystone suggesting something and maybe now this committee is the creature to sort of deliver on that promise.

I think if the Secretary and the Acting Commissioner recognized the failure to meet the goal and then proactively said, "This is what's going to happen. This committee is going to -- you know, an FDA advisory committee is now going to have responsibility for continual evaluation and movement of the plan forward. We're not going to wait until 2006, but this is going to be an ongoing activity, and that committee has the responsibility and the authority to bring together all of the players and to sort of figure out what a reasonable schedule of compliance will be and what the penalties will be for noncompliance along the road."

I mean, you know, as I said at the beginning, I'd like to see a mandate, but a mandate doesn't always make things happen, and I think there has to be other processes involved.

And I think having an FDA responsibility for evaluation and forward movement on the plan would be an important step.

When I said I didn't want FDA to get in the deeming business, that was my concern. I think the responsibility belongs with FDA, and I think the responsibility -- and I still agree with what we were suggesting back in '96 -- that an advisory committee or advisory committee-like process should be responsible for evaluation and moving the process forward towards the desired goal.

And I would agree with Ruth that you can do a lot of things simultaneously. I mean, I think the first job, as I say, things were missing. Get them in there. And while we're doing that, we're going to figure out how to do things better, but we've got to get the threshold; we've got to get the floor.

And that doesn't mean the floor works perfectly. It may even work very imperfectly, but it's what was required by statute. It is what was required by the action plan.

There's also flexibility here. This is a process piece, and that process can go forward, it can change, and we can learn and do things better. Nobody is arguing with that.

But that would be my take on it.

CHAIRMAN GROSS: For the benefit of the committee, can the FDA tell the committee what are the options that the FDA would have to deal with this? Exactly what is regulation and what is nonregulation and what are those options?

DR. TRONTELL: I'm going to try to answer that question because I think when we start talking about regulation, we have to look at, you know, where FDA, in fact, has authority to regulate a particular sector of the United States.

And in a sense, our regulatory authority is largely confined to dealing with drug manufacturers through our ability to regulate their products and to approve them and various materials associated with the approval of those.

So I think that where you get into what might be from a regulatory standpoint something potentially problematic, I think we have the power of persuasion certainly with the potential force of regulation behind us to try and exhort individuals to work cooperatively together, a guidance document without the back-up of a regulation, which would invoke the full possibilities of the public law, is something, you know, we would have to think.

Our hope is to have from the committee some suggestions as to process to pull this together.

I might take the liberty now, having tried to answer this question, to throw another question back to the committee because I see some ambiguity described here in how the criteria were interpreted in the strict subcriteria that have been described.

But I also see several principal players here, and this starts to get at the issue of regulation. Who might begin to address this? We've talked about the data vendors. We've talked about the software providers and the intermediaries and also the pharmacists who may operationally, if they have one printer that prints the label and a piece of paper, have to get something that fits into a ten inch by eight inch format and still do the job of what we're trying to accomplish.

And I'd appreciate hearing back from the committee any suggestions about how we can work with this array of players, with the moral force if not the regulatory force that the agency has to improve this information.

CHAIRMAN GROSS: Well, I guess the idea that we're stuck to one printing format in this day and age sounds inconceivable, but that's a separate issue.

Anybody else want to comment?

DR. STROM: Just to comment that in terms of how to work with all of the various players, my suggestion about an accrediting body or however you word it would be a way of having all the players involved in naming that.

And obviously FDA would have to play a major role in that process.

CHAIRMAN GROSS: So it sounds as though -- tell me if I'm wrong -- but it sounds as though there's a consensus that the stake holders need to meet; that whether you want to call it a workshop, a conference, information so that we're reinforming everybody about the Keystone criteria, that's really only part of it, and we need to hear any problems they may have so that everybody is on the same page so that we can move forward from there.

And so all of the players, all of the stakeholders need to be involved, and they need to be defined.

So starting from that point of view, is that -- do people agree that we need to get a group together?

Steph.

DR. CRAWFORD: Yes, i strongly agree with that. That was one of the suggestions I was going to make with the action plan, but a little different from how it was done before, from what we've heard today and what we've been reading, I have questions as to how well the information, the very important information from the Keystone criteria has been filtered down from the critical stakeholders, from the professions, the vendors, consumers, the agency, other users.

So a part of this workshop consensus conference if it were to come about, I think the critical stakeholders should also come with suggested or action plans of how they would sensitize, update the issues, the problems, the challenges to the practitioners and other players because I'm not sure if it went down from the high organized levels of the pharmacy, of the vendors, to the independent pharmacies, the community pharmacies, the mail service, and other institutionalized base out-patient pharmacies.

So I'm concerned it's being considered at the top without getting input or information to the people throughout at the lower levels organizationally.

DR. CRAWFORD: I'd like to just comment that while I would favor getting all of the stakeholders together, I'm not sure I would call it a consensus conference. I would think of it more as a state of the art in science or lack thereof conference, and I would invite, not just have it as a public offering; but I would specifically invite the various stakeholders to come, and there would be presentations, say, of Dr. Svarstad's study and where we are and the history from Tom McGinnis, whatever, something like a little mini what we did today.

And then an a priori set of problems. Here are the problems. How do they happen? How do we solve them? And what are suggestions?

And then get input in all of that. That might be very useful.

CHAIRMAN GROSS: Okay. Yes, John.

DR. SULLIVAN: I would certainly endorse your suggestions, Peter. We certainly have to do better, and getting all of the stakeholders together would be a start, whether it's a workshop or whatever format you would prefer. I think that's clear that we have to do that.

And then you can either take the carrot or the stick approach and you can move from there.

There clearly has been progress, but it's in no way optimal. I guess I would also like to just comment because I didn't get a chance to jump in before.

We can give an absolutely perfect -- something to the patients in written form that is absolutely perfect, but then we have no idea whether it just goes in the trash like 90 percent of the rest of the mail that we get every day. Certainly people that are intellectually curious will have already checked it out on the Internet, which lots of people do these days. There are multiple methods of getting information.

So we clearly have to do more research, but I think your suggestion, Peter, as a first step of getting the stakeholders together, and then if they can't come up with something to regulate themselves, then we can recommend to the agency that other steps be taken.

CHAIRMAN GROSS: Just to elaborate, the purpose of getting together would be to get everybody on the same page, and then that's why I wasn't clear exactly what form it would take as far as the FDA was concerned, was to have some type of oversight group, whether it's called a guidance or whether it's called a Joint Commission type accrediting agency. I'm not sure what that form should be. I'm not sure that we can solve that today.

And then whatever that group is, there would have to be some sense of what happens if one of the stakeholders doesn't comply. I mean, they're a carrot and stick. There has to be some sense that there would be some penalties if a stakeholder didn't comply. Otherwise we're going to be right back where we are.

Brian.

DR. STROM: I think you addressed a lot of my concern. I want to be clear that I certainly agree with the idea of having a meeting of all the stakeholders. I don't think it should be informational. I think the information is out there. I don't think that's the issue.

I think there needs to be a meeting of the stakeholders to decide what is the new structure that will be put in place that will have carrot and stick both as part of it, short of FDA having to impose something.

So it would be an action meeting. It would not be an informational meeting.

DR. DAY: But perhaps I was too gentle in the way I said that. It would start with educational. Here are the problems. How do we solve them? And, by the way, here's some options and, you know, some pretty strong ones.

And then get the objections up front before it's mandated or, you know, put out there. Get feedback and then go forward with something at the end of the meeting.

CHAIRMAN GROSS: In fairness to the stakeholders, we do need to hear if they have any particular problems with what we're talking about as far as putting it into action. You know, we need to make a decision on that.

Bill and then Arthur.

DR. CAMPBELL: Thank you.

Well, I do believe we have come a long way in five years, and I believe the world is really substantially different in terms of recognizing drug risk in this country. This committee is one example.

The FDA organization, the funding and concentration on post marketing use of drugs and so forth, and so I'm, frankly, much more optimistic, I suppose, than Arthur on this particular topic.

One of the major things that has happened has been the formation and coming together of the practitioner organizations, AMA, APhA, SHP, and so forth, around the white paper on the professional's role in developing effective risk management in drugs. I think FDA was a party to that paper as well.

So it seems to me we have already formed the nucleus of the constituency group, the stakeholder group that needs to come through that, and pardon the plug, but I also think the formation of the Centers for Education, Research, and Therapeutics, which is funded jointly under FDA and ARC to assure safe and effective use of drugs and to partner with public and private organizations to do that, and has a history now in developing workshops on drug safety, drug risk, drug communications and so forth, makes it a very natural next step.

CHAIRMAN GROSS: Arthur.

MR. LEVIN: Just, you know, Bill, I'm a big fan of CERT.

Here's my problem. I don't think people understand that what 104-180 did is tie the stick up. The stick that the Secretary had was tied by this piece of legislation.

Why? Because there was a proposed rule to mandate what was then called medication guides. Folks didn't want that, and so they got an act in Congress that for all practical purposes tied the hands of the Secretary, prohibiting the Secretary from enacting a mandate.

That's what this statute says, except that there would be two occasions on which a judgment could be made as to whether to untie the Secretary's hands or not. The one is the overdue judgment we're now letting pass, I think, from what I'm hearing in many ways, and the stick is still tied, my friends, until 2006 by statute.

That's what you have to understand. Where is the stick going to come from? Now, it may be a Good Housekeeping seal of approval and a competitive business with two vendors, and we'll probably end up with one vendor if that industry goes like every other, you know. There goes competition.

There is no stick if this opportunity goes by and we have to wait until 2006. The stick is not around until 2006. That's what this statute did.

Now, in my mind, when I responded to Paul -- and the FDA can correct me if I'm wrong on my understanding of what the statute does -- is that we have to give the stick back. That doesn't mean the Secretary has to use it. It simply unties the Secretary's hands as the Secretary's hands were untied before the enactment of this law.

This law was very specific in heading off the medical guide proposed rule of '95. This is nothing that goes back to 1938, to 1962 in the history of FDA regulation. This is a very specifically enacted law by those folks who did not want a mandate for medication.

Get rid of it, I say. Untie the Secretary's hands. That doesn't mean the Secretary has to do anything that he doesn't want to do or she doesn't want to do, but it begins the process of saying: hands are untied, folks. You haven't done it yet. We're going to work with you to get it done, but there's no longer this prohibition.

Otherwise we have to wait until 2006 to put any teeth behind this.

CHAIRMAN GROSS: Arthur, I'm not sure the committee is saying anything different from what you're saying. It's probably semantics, but what we seem to be saying is that there will be a stakeholders meeting. Call it whatever we're going to call that meeting where everybody reviews what was presented here, what's known, what has to be done. That's number one.

Number two, an oversight group gets set up. The form of that I don't think we can commit ourselves to today, although we'll see what everyone else thinks.

And then that oversight group has to have whether you call it a stick, enforcement measures or some action that they're going to take. In order for the oversight group to be effective, there needs to be -- it needs to be understood that there's some action they can take if compliance isn't achieved.

So those three areas, I think, address what we've been hearing today.

DR. STROM: Peter, can I formally move that we take a specific vote, whatever the wording specifically is, but that will untie the stick?

The point is based on the data we heard today, it is not yet successful in the way we want it to be. I think the next step from a process point of view is exactly what you describe, but I think it is important that it be done in the context of the stick being available, both because it will make that next step more effective and it will allow for a step to follow if the next step isn't effective.

DR. DAY: Could I please hear some words on what --

CHAIRMAN GROSS: Wait. We have a motion. We have a motion on the floor. Is there a second?

PARTICIPANT: Second.

DR. DAY: I want clarification of the motion, please. Would you put it in -- instead of a stick removal, okay?

(Laughter.)

DR. STROM: I agree with you. My wording was far from -- I guess maybe Arthur can help because I'm not sure exactly what the letter of the law is. We should word it in the context of that original law.

MR. LEVIN: A draftsperson and I may be able to do this.

"The Secretary Review." This is Part E of the title. "Not later than January 1, 2001" -- and understand that we're behind. That's what is going on now -- the Secretary of the Department of Health and Human Services shall review the status of private sector initiatives designed to achieve the goals of the plan described in Subsection A, and if such goals are not achieved -- that's 75 percent written useful information -- and if such goals are not achieved, the limitation in Subsection D shall not apply, and the Secretary shall seek public comment on other initiatives that may be carried out to meet such goals.

D is limitation on the authority of the Secretary. The Secretary of the Department of Health and Human Services shall have no authority to implement the proposed rule described in Subsection A or to develop any similar regulation, policy statement or other guideline specifying a uniform content or format for written information voluntarily provided to consumers about prescription drugs.

DR. GARDNER: So, Brian, might you say that having reviewed the evidence presented before us, this committee judges, has determined that the --

DR. STROM: The 2000 goals have not been --

DR. GARDNER: Have not been met.

DR. STROM: Or 75 percent availability of useful information --

DR. GARDNER: Of useful information.

DR. STROM: -- have not been met.

DR. GARDNER: And, therefore, we recommend that Subsection D be not --

DR. DAY: That the Secretary invite public comment on --

DR. GARDNER: Right, exactly, exactly.

DR. DAY: -- the other options.

DR. GARDNER: As afforded, as specified in Public Law.

DR. DAY: Right.

DR. STROM: Yes.

(Laughter.)

CHAIRMAN GROSS: Let's go around the room and see if everybody agrees. Ruth?

This is agreement that 75 percent compliance has not been achieved.

DR. DAY: I agree that the 75 percent complies with useful information has not been met, and that we should invite public comment for other options according to all of the law, regulations, yes.

CHAIRMAN GROSS: Okay, and, Jackie, your opinion?

DR. GARDNER: I agree with Ruth that 75 percent of useful information has not been met, and that we should invite public comment according to the provisions of Public Law 104.

CHAIRMAN GROSS: Bill?

DR. CAMPBELL: It's close. I agree that the goal of 75 percent of information distributed that can be classified as useful, and by the Keystone criteria of allowing the consumer to receive maximum benefit of the drug has not been met. I agree with that.

And I further agree that we should not simply invite comment, but we should provide guidance and advice on how that goal can be met.

DR. DAY: And I amend my comment accordingly now.

DR. STROM: And I would like to amend the original motion accordingly.

(Laughter.)

CHAIRMAN GROSS: Forget the motion.

Steph?

DR. CRAWFORD: I agree with everything that's been said, but I'd also like to acknowledge that I do think substantial progress has been made, though we need to do a lot more.

CHAIRMAN GROSS: Okay. John, your comment?

DR. SULLIVAN: I would certainly endorse Bill's and Stephanie's comments that technically it hasn't been met, but there has been progress made.

CHAIRMAN GROSS: Michael?

DR. COHEN: Yeah, I will so endorse it. I think, you know, we only looked at four drugs, and in each case there was significant problems with the information missing in our particular area, risk management and drug safety.

So I couldn't see it any other way but not extending this until 2006 or voting as you have.

CHAIRMAN GROSS: Brian?

DR. STROM: I agree.

CHAIRMAN GROSS: I agree also.

Okay. The next is make a recommendation or make some suggestions to the FDA, some options. The first thing we talked about was a workshop, getting all of the stakeholders together. Why don't we go around the group and comment on that?

DR. GARDNER: May I ask a question, Peter?

CHAIRMAN GROSS: Yes.

DR. GARDNER: Given that this committee was convened for the purpose of looking at this question, can the committee invite -- convene such a meeting to gather further information? Because there seems to be a venue issue.

And although I agree about the Secretary, maybe that is the best place for it. It seems to me to yet introduce another organizational element.

CHAIRMAN GROSS: Yeah. I think this is up to the FDA. We're just making some suggestions and they'll make the final decision.

So this time I'll start at the other side of the table. John.

DR. SULLIVAN: Could you --

CHAIRMAN GROSS: As far as do you want to discuss what options you think are worthwhile as far as gathering a group together or workshop of the stakeholders?

DR. SULLIVAN: I would concur with your previous suggestions that probably a workshop would be the first step.

CHAIRMAN GROSS: Okay.

DR. SULLIVAN: And go from there.

CHAIRMAN GROSS: Michael?

DR. COHEN: Are you talking about a public workshop, an FDA public workshop? Is that what you're talking about?

CHAIRMAN GROSS: Yes.

DR. COHEN: And would that allow us to give recommendations or provide information?

CHAIRMAN GROSS: I think you can make that as a suggestion that that should be done there.

DR. COHEN: Well, I think there are some specific recommendations that we could make that have come out of this committee meeting today. So I would like to see that as part of this workshop.

CHAIRMAN GROSS: Okay.

DR. COHEN: That we would be involved with it, that is.

CHAIRMAN GROSS: Good. Okay.

Brian.

DR. STROM: I agree with the idea of having a workshop as a logical next step. I think it's important that it be clear that the workshop is not informational. It is a workshop in order to decide on what the next logical activities would be that have, again, both carrot and stick as part of it, how the organization of the current system should be changed, not just informational, and if, in fact, there isn't anything concrete that emerges from the workshop that changes the system, the existing private system, then it would be considered a failure, and it would be up to the Secretary then to be more proactive and to follow it.

That should be understood going into the workshop.

CHAIRMAN GROSS: Steph?

DR. CRAWFORD: I agree with the outcome being a useful action plan.

CHAIRMAN GROSS: Bill?

DR. CAMPBELL: I agree. I would make it a practitioner organized and driven effort, and I think the major change that has occurred in seven or five years --

CHAIRMAN GROSS: What do you mean by practitioners?

DR. CAMPBELL: Is that the practitioner organizations, and I specifically refer to AMA, APhA, ASHP and the white paper group on safety and risk management, have committed themselves as professionals in a way that did not exist five or six years ago, and while all stakeholders may be present, I think the onus needs to be on the practitioner groups.

CHAIRMAN GROSS: Okay. Jackie?

DR. GARDNER: That sounds like an unfunded mandate to me, and I'm not sure that it works, although it would certainly be nice. I think I would go back to what Brian's suggestion was, that a meeting be convened with an understanding that an action plan needs to come out of it or it gets kicked back to the Secretary.

CHAIRMAN GROSS: Okay. Yes?

DR. CAMPBELL: I didn't mean anything different. I mean, you know, the conveners, I think -- the same sort of thing, an action plan agenda for education and training. All of that has to be there. I'm just thinking that it is best handled at this point for the practitioners to have ownership of this because that's where implementation will have to occur, not the vendor level.

DR. GARDNER: How would you effect that out of this group? I mean if we recommend that and go away today, you would expect them to pick it up or you would expect the FDA to? Tell me.

DR. CAMPBELL: It will fall in the FDA's agenda to do it. I'm just saying that the mechanism for convening and implementing should be through them. It's simply a recommendation.

CHAIRMAN GROSS: Okay. Ruth?

DR. DAY: I support a workshop which has an educational component directed towards an action plan outcome, and I think it should be sponsored by the FDA with participation of the professional organizations in developing it.

CHAIRMAN GROSS: Good. I agree with all that's been said.

Would any of our guests and presenters like to say anything?

MR. LEVIN: I was out of the room for half of this. So apologies.

I just want to understand why, Bill, you think the professional groups should have ownership of this issue.

DR. CAMPBELL: I think the professional group needs to implement it, and if they're going to implement it, they really need to own it at the very beginning. I don't mean own it, taking it out of FDA, not at all, but they really have to be driving it from the very beginning.

MR. LEVIN: I'm still puzzled. How do professionals drive the information vendor process? Explain that to me. How do they? I don't understand.

DR. CAMPBELL: I think that's the challenge for professionals to develop, whether it's professional practice standards through their regulatory boards, through whatever. I don't think the professional groups at this point have owned this process.

MR. LEVIN: I remain confused. There are two participants in the information business as I understand it, although there are far fewer than I used to understand. I think a remaining professional organization is the Association of -- what are they now? They used to be Hospital Pharmacists.

PARTICIPANT: Health System Pharmacists.

MR. LEVIN: Health System Pharmacists that are information vendors and providers, and the rest, I believe, with USP out of the business are proprietary.

So, again, I don't get the connection with professionals and the vending of information, which is what this process is about.

I mean, the origin, unless we talk about the scientific origin, but the origin of the material, the product is with the vendor. The vendor is either an organization, a not for profit organization, or it's a for profit organization.

DR. CAMPBELL: And who's the customer?

CHAIRMAN GROSS: I think at this point --

MR. LEVIN: The customer is the patient.

CHAIRMAN GROSS: At this point we're really just giving our opinions.

DR. COHEN: Peter, could I just ask one thing?

CHAIRMAN GROSS: Yeah.

DR. COHEN: The time frame.

CHAIRMAN GROSS: Staff first and then Michael.

DR. COHEN: Oh, I'm sorry.

DR. CRAWFORD: For Arthur I can just give two examples of how the profession could help in the process, one through educational programming, articles, et cetera, but also in the absence of regulation or guidances, what people look for but they want are professional standards, and the pharmacy organizations do provide professional standards on the use of information technologies and other things because I still think part of the problem is that the end professional users are not necessarily aware of all of these criteria.

So I think it's very critical that we involve the profession.

CHAIRMAN GROSS: Paul?

DR. SELIGMAN: I'd be interested in Bonnie's thoughts on this, but clearly the pharmacies are purchasers. I mean they're buying the information, and to that degree, I think they have clearly a stake, you know, in terms of what it is they're buy and why they're buying it and in some cases why they're not buying it, from whom they're buying it from, and the quality that they demand from that purchase.

CHAIRMAN GROSS: Why don't we go on to the last part of this? And that is that following that meeting or maybe during the meeting, at some point an oversight group will be appointed. Exactly what it will be called I'm not sure, but they will have some -- they will develop some enforcement measures to try to assure that the Keystone criteria are met.

Arthur.

MR. LEVIN: I would like to speak in favor of this committee being the group. I don't think this committee as constituted can do it, but I think there are models in other advisory committees for handling where one committee ends up with a very complicated and not overlapping issues.

And the example that comes to mind is the Food Advisory Committee of the FDA, which is now in six subcommittees, dealing with things like natural toxicants and contaminants and infant formula, two very different issues, and biotechnology, a very different issue, but with the subcommittees all reporting back to the full committee.

It seems to me we were what we had in mind, those of us who talked about this in the Keystone Steering Committee process, to have an FDA advisory committee as sort of the umbrella. I think there are ways to operationalize it, given that we're small. We have a lot of other things on our plate.

But I think there's a lot to be said to vesting the responsibility in an advisory committee process. Since we're the ones making these recommendations, I think we're responsible for making sure they go forward.

CHAIRMAN GROSS: Okay. Let's go around the room. Remember we don't have to all agree on exactly what the oversight group should be and what the enforcement measures and methods would be. We just need to come to see if we have a sense that that's a direction we would like to suggest to the FDA that be pursued.

Ruth.

DR. DAY: I would like to hear the language of what we are all agreeing to before I make a comment.

In a way it's good. It changes as we go, but if someone could make an initial stab, we agree there should be an oversight committee to --

CHAIRMAN GROSS: That's basically what I said.

DR. DAY: But to -- does that mean to periodically review the materials and do sanctions and so on? I just want to know how much of a task is being recommended.

CHAIRMAN GROSS: Yeah, the concepts were there be an oversight group and that there be some measures and methods of enforcing without being any more specific.

DR. DAY: In a nonspecific way I agree.

(Laughter.)

CHAIRMAN GROSS: That's all we need.

DR. GARDNER: I could agree with that concept as proposed.

DR. CAMPBELL: Yes, agree.

DR. CRAWFORD: I agree, although I'm a little confused. Is this an oversight group that would be separate from the FDA? I'm a little confused.

CHAIRMAN GROSS: It would be part of the FDA.

DR. CRAWFORD: It would be part. Thank you. Then yes.

DR. STROM: I agree.

DR. COHEN: I agree, and if you think about it, we do have most of the components that would be necessary. The way the committee is constituted right now, the individuals who are on it have various backgrounds that would fit just perfectly if you were going to design a committee. I think most of us would fit in.

CHAIRMAN GROSS: Okay. John?

DR. SULLIVAN: I would also agree, but I think we have to remember that we're purely an advisory committee, aren't we?

CHAIRMAN GROSS: Exactly. Okay. Are there any other burning issues or comments before this group?

If not, the meeting --

DR. COHEN: Peter?

CHAIRMAN GROSS: Michael.

DR. COHEN: We need a time frame for that meeting.

CHAIRMAN GROSS: You've got to be serious.

DR. COHEN: Not have it a year from now. I'd like to see it happen pretty quick.

CHAIRMAN GROSS: Okay. Makes sense.

Okay. The meeting is adjourned. Thank you all.

(Whereupon, at 4:42 p.m., the Advisory Committee meeting was adjourned.)