DEPARTMENT OF HEALTH AND HUMAN SERVICES

FOOD AND DRUG ADMINISTRATION

CENTER FOR DRUG EVALUATION AND RESEARCH

Transcript of a Public Meeting:

Wednesday, March 1, 2000

9:00 a.m.

Double Tree Hotel

1750 Rockville Pike

Rockville, Maryland

C O N T E N T S

Welcome:     Nancy M. Ostrove, Ph.D.

Opening Remarks:     Janet Woodcock, M.D.

Small Group Assignments:     Nancy M. Ostrove, Ph.D.

Results from the Breakout Sessions:    Nancy M. Ostrove, Ph.D.

Discussion of Questions:

Question One

Question Two

Question Six

Question Five

Question Seven

Question Four

Adjounrnment

P R O C E E D I N G S

DR. OSTROVE: Good morning, everyone. Welcome back. I know that you all probably have very big heads from having to process all that information yesterday. I have to tell you my brain is definitely tired. In fact, my husband said that I sat up in bed a couple of times over the course of the night. I am sure that it was revelations, you know, about how we should best do the Year 2000 assessment. The most distressing part of it, of course, is did I remember any of them? I am afraid not. So, we are depending on you to bring those back.

We are all sure that you have many thoughts to share and today is the day that we want to focus on that. Before we do that just for a little bit of time, we would like for you to hear from the person who for years has been supporting the need for all stakeholders, and especially patients, to receive information that is critical to their understanding of how to use prescription medicines safely and effectively in a way that will minimize the risks, while maximizing the benefits that they receive from the medicines. So, I would like you to welcome someone that I am sure you all know and respect, Dr. Janet Woodcock, who is the Director of the Center for Drug Evaluation and Research.

[Applause]

DR. WOODCOCK: Thanks and good morning to everyone. It is wonderful to be able to address an audience of people who are really committed to this idea of patient information. Many of you have really spent your careers and much of your effort around this information.

I want to talk to you about the recent ideas that have been developing in FDA around risk management and how this whole idea of patient information feeds into risk management, and what this means.

As I said, you all are probably a more receptive audience for this type of thinking than possibly a lot of the scientists at FDA. I was talking to Nancy right before this session about this. Every time we talk about patient information at the FDA I think more and more people begin to understand the concepts and buy into the concepts, and there is quite a bit of momentum around this, but there is an aspect of -- what shall I say? -- it is hard to say this tactfully, but professional elitism or something like that, or professionalism, the people who own the information or are technically trained in scientific information or medical information -- there is some reluctance for those folks to share that information, I think, in a way that is understandable for other people. Some of that is a concern that they won't understand. Some of that is the fact that knowledge is power, and those who have the knowledge have the power. Some of it is really failure of imagination, I would say, the inability to step into that other person's shoes and into themselves and understand why they need to know that stuff just as much as the professional does.

The idea of risk management at FDA really surfaced about a year ago when a number of drugs had been withdrawn from the market. Some of them were drugs that had been approved for a very long time. Some of them were drugs that had been recently approved. And, there was a lot of concern amongst the world that FDA was reviewing drugs too fast and putting them out onto the market, and so forth, with little regard or less regard than previously for safety of the public.

So, we went back and looked at our processes and looked at the numbers of withdrawals, and so forth and so on, to see if, in fact, any of that might be the case. Our conclusions in that regard were that no, that wasn't the case; that actually the withdrawal rate was lower than in the past, and the number of very serious new events discovered after the drug was on the market was somewhat lower in the past. But, the point is those risks are still there even though they are discovered before drug is marketed. Those risks still pertain to the drug. Even if they are in the label, people are still exposed to the risk.

So, we went a step further and we looked at the system in this country for managing the risks of pharmaceuticals. How is that done? So, we tried to step back and look at the big picture and think how in this country do we make sure that benefits are maximized for patients and risks are minimized for patients with respect to taking medicine.

And, what we found, and we weren't the original authors of this -- these thoughts had all existed out there in the world, is that over the last hundred years or so we have operated under a professional model, and that professional model has been that the healthcare professionals -- the prescribers, pharmacists, the nurses, and so forth, will be trained and credentialed and will know everything that is needed to be known, and they will make sure that the patient gets the correct drug, and at the correct dose, and they will monitor the side effects, and they will adequately instruct the patient.

As has been pointed out by the National Patient Safety Foundation and others, within that model then, if anything goes wrong, the professional model calls for blaming somebody -- calls for blaming usually the professional who is closest to the problem and that person might be the pharmacist who dispensed something incorrectly because he couldn't read the handwriting. It might be the physician who wrote the wrong dose of a drug, or the nurse who formulated the drug wrong, or it might be the FDA that is blamed because the drug is causing more side effects than were anticipated.

In other industries though what has been discovered is that most of these type of ill effects are, in fact, system problems. What does that mean? Well, that means that this whole thing is a big system and you can't just pull out one piece of the system and blame it when something goes wrong. For example, say if a patient's wrong leg is cut off -- that is a notorious event. That is a terrible event that gets in the paper, and everything, and catastrophic for the patient, and everything. And, the immediate response, they will blame the surgeon. Right? Of course. When you read that article you blame the surgeon. How could the surgeon do such a terrible thing? That is a terrible thing. But when you go back and you try to reconstruct what happened and how did this all come to happen, what you find is, well, on the x-ray machine the technician put the wrong label on for the x-ray. So right and left were switched. And then, perhaps when the leg was prepped, the prep team prepped the wrong leg. And then, when the patient was put in the operating room, they exposed the incorrect leg, and the patient was all covered up and put to sleep, and everything, and there was the leg sticking out. The surgical team comes in, does the surgery.

It is a comedy of errors; it is cumulative mistakes that occur all through the system. If one piece of this series had been caught, this whole thing wouldn't have happened. That is what occurs, in fact, with most errors that occur, preventable problems that occur within healthcare. So, there is a whole cascade of events that lead to a bad outcome.

With medication errors -- you know, problems in the dispensing of drugs, and so forth, most of them actually never reach the patient. There are enough fail-safe mechanisms in place that somebody catches the problem and the medication error is stopped before it actually impacts on the patient. But because there are so many prescriptions in the world written today, 2.5 billion I think prescriptions written, even a tiny fraction of medication errors can lead to a significant number of problems for people.

So, my point in all this is that we stepped back and we said, okay, we, at FDA, approve medicines. We put them out there. Our attitude has always been, all right, we have studied them and the good things are in the label and the problems are in the label and now the system out there -- it is their problem. They have to handle this. They can read the label. They can understand the problems. The professional system is in charge; that is the practice of medicine. I have heard that many times from the people at the FDA. That is the practice of medicine; we don't really have an impact on that.

But if you look at a system's approach, if you step back and you think these outcomes, whatever they are, out in the real world are a result not just of one professional and their failure to read the label, or whatever, it is the result of a whole series of events and structures and processes that have been set up for healthcare, and you can see that every part of that chain has responsibility for the events that occur, whether they are for good or for ill for the patient, for the ultimate patient.

This is where patient information comes in. I don't really need to talk to you all about the role of the patient because I suspect everyone here has a very good appreciation of that. But if you look at emergency room visits or admissions, you see that a tremendous number of visits, hospitalizations, and so on, are from simple non-compliance with medications. All right? And, if you wanted to take this professionalism analogy a little bit further you would say, well, that is the patient's fault. They didn't take the medicine. Right? And, then we can just blame somebody else -- that is their fault.

Well, the question is did they know the consequences of not taking their medicine at a level of true understanding of that? Did they know the consequences of drug-drug interactions? Did anybody tell them? Did they know they should come to their doctor if they experience whatever side effect it might be? Not were they informed in some way; did they know? And, we really can't blame people if the system has failed them, if it has failed to allow them to arm themselves sufficiently with information to protect themselves.

This is the same with doctors I think. I am a physician so you might say but I feel that way about nurses and pharmacists and everyone else in the chain. If the system fails them, is it really right to blame them for that one event that occurred that was the result of a system failure?

Now, I regard patient information as a real spectrum of stuff, and what I think what is being talked about here is just one facet of that, which is the written information that is given to patients, for example. Obviously, reinforcement is so important in learning that the prescriber should inform the patient. The pharmacist should then inform the patient, and that could be reinforced with written information -- this is the ideal, in the language that the patient is fluent in, of course, and if they have trouble absorbing written language, which many people do in our society, it would be really nice to have videos and other things, and most people in our society seem to be able to manage to watch television. I have been saying this for years. I have kids. I have never seen a medium with such attraction as television. You know, it sucks people over to watch it.

So, we shouldn't limit ourselves if we are really trying to address the actual problem, which is people are being harmed because of ignorance about the medicines they take, and their are failing to achieve the promise or the benefit of those medicines because they don't understand. Now, at some level if people are fully informed and they still choose not to take their medicines, or they still choose not to deal with the side effects, or to take interacting drugs, or something, then I would say at that time, well, that is their responsibility. They have to be accountable for that because in a free society those are the choices they have made.

But most of the people who roll into the emergency room with their non-compliance, or they are using their drugs incorrectly, or they are waiting so long to come in while they are experiencing terrible side effects -- it is out of ignorance. They do not truly understand what the consequences will be to them, and they have not been truly informed. So, I think what this is all about, as well as many other efforts I hope the FDA and others will be taking, is to really think about what are all the ways we can help maximize the benefits of the medicines that are available to people, and really improve their health, and because drugs always have the potential for harm -- every one of the drugs -- how can we minimize that harm and manage it? That is the risk management part -- manage it appropriately.

Now, effective communication is the key to this. It isn't enough, as I said earlier, to just give people the information. It has to be reinforced and it must be in a form that they actually can absorb and use, not just a paper handed out to them.

I remain concerned about the quality of information that people are getting, and I guess we have been hearing that at this workshop. It isn't easy. We are not only trying to overcome the barriers that were identified during this whole process, the logistical barriers of getting this information out, and the type size, and everything, we are trying to overcome a century or more of the professional model that really people don't need this information and probably shouldn't have it because they might be worried, or something like that, or they wouldn't be able to understand it.

I think getting technical and scientific stuff into a form that people can truly absorb and understand is a tremendous challenge. We have been trying to do this at the FDA. When we approve a new drug we put information up on the web. We try to make some of the information in a form that consumers can understand, and I can tell you it is a tremendous challenge.

The reason I have so much passion about this, sometimes patient inserts are a part of the unit of use packaging of newly approved drugs, and so I get to see those when they come through and uniformly they are pretty terrible. They are terrible. I have seen ones that you would lie down on the floor laughing at them; you would think who is this directed at? You would have to be, you know, a Ph.D. graduate to even begin to access some of the information in these things that are submitted. And, our scientists are, well, you don't have enough of the information in here; you ought to add this piece, and you ought to -- you know, this isn't quite right technically. You ought to put longer words in there.

So, we aren't up against just some political difficulties or logistical difficulties. We are up against a huge societal barrier here about technology information and really the bias of experts that ordinary people can't understand this information. That is what I think. For me, that has been totally borne out in my experience over the past five years with this.

So, I think the FDA needs all the advice we can get. That is why we have had this workshop because we need the input from the people in the field who are working on this, the people who care about this as to how to do these surveys, how to move this process forward. How can we determine, as best as possible, the quality of this information in the sense of is it really useful? Does it give the information that is needed to people to protect themselves and to manage their medicines as best as possible?

So, I appreciate everyone's attendance here. I hope you are really on task today as you provide us input. I don't think any of us can overestimate the magnitude of the task that is involved and, as I said, I think the written insert or the written patient material is only one facet of this whole problem. There are a lot of people to whom you give written information that will not allow them to learn what they need to know. So, we really need to think about that as well.

But there are many people in this country who do read and who do use the written word to get their information, and who can refer to that when they are at home or when they are experiencing a possible side effect, and so forth. That is something that people can go and look at, or when they are considering not taking a medicine anymore, for example, and being non-compliant with their regimen, having that information there and why you should take it and why you should finish it, and so forth, I think could be very helpful in many of these areas.

So, we really are dedicated to moving the risk management system forward in as many ways as possible, and one crucial link in making sure that medicines are used as safely as possible is making sure that that information gets right down to the level of the person who has to take the medicine, or the parent who has to mange the child's medicine.

So, this is crucial for us. I think it is crucial for the manufacturers of pharmaceuticals -- and I see a number of people from the manufacturing sector here -- because these drugs are only going to be as safe as they are used. We can study them extensively premarket, and at the FDA we can review them meticulously and we will write all the risks down right on that package insert, and we can write the benefits down, and so forth, and if that information is not used correctly out in the world, if it isn't maximized then the medicines are not as going to be as safe as promised and they are also not going to be as useful as promised. So, in order to really make drugs as safe and effective as we can this is part of the whole management chain. I can only promise you that the Center for Drugs at least will continue to make this a high priority. We are very interested in input on this survey and any other ideas about what can be done to improve this information.

Thank you very much. Also, I am willing to take any questions people might have for a few minutes. Any questions? No? Okay, thanks.

[Applause]

DR. OSTROVE: Thank you, Dr. Woodcock. We really appreciate it.

Now, all of you have been patiently waiting your turn to get down to work. So, I am going to give you instructions about how we are going to do that this morning. Remember I told you, you had to bring your badges. Did everybody bring their badges back today? There is a number on your badge. That indicates the room or place assignment for you. This was done in a random fashion. We didn't try to stack any of the groups, or anything. So, any cases where you have lots of one particular perspective, again, that is part of the whole random process and you ended up out here, you know, on one of the ends of the distribution.

So, if your badge says group one or group six, you will be in the Twinbrook Room. If your badge says group three or group four, you will be in the Montrose Room. For both of those, when you go out the door, you make a right and then you make a right again. The rooms are along that corridor leading toward the front of the building. Group five will be in the front of this room near the door. If you look at that flip chart, you will see it says group five over there. Group two will be in the front of the room on the far wall, right over there. Group seven will be in the back of the room near the door, in that corner. Group eight will be in the back of the room on the far wall.

Clearly, you will probably have to take some chairs and kind of move them around so that you are in a comfortable position. The facilitators that we have assigned to each of these areas will direct the process that we have recommended the groups follow, just basically to ensure that all the members have a chance to express their opinions.

I just want to give you a few points first, at the risk of frustrating you; I am sure you are all itching to get started. We won't have any breaks. We will simply kind of move to the areas and start from there. We just decided that it would be better from a process standpoint to not have a set break. So, you will be in those areas from about 9:45 to noon. If you need to take a bio-break, to right ahead. If you need to stand up and walk around, don't feel constrained. I am sure you all don't need to be told that, but just in case.

Now, one thing I wanted to point out, this is clearly not the Keystone process but, just as Judy O'Brien mentioned yesterday, I think it is important for everyone to realize that everyone in this room is trying to do the right thing and has reasons behind their perspectives, and I just think we want to keep that in mind. I heard someone this morning say something about the Keystone headache, that this person had that had gone away about three years ago, came back after yesterday's meeting. I felt bad about that, but this is not the Keystone process. This is not formally a consensus-building exercise. There is not enough time here for people to try to persuade each other of the rightness or any particular position. We want all of your thoughts and your recommendations to the extent that we have the time to get those out.

However, we also would like some sense of degree of agreement with different positions. So, don't be surprised if the facilitators ask for this kind of information, just so that in terms of the facilitators presenting the results of the group conversations after lunch we can get a sense of where these different ideas come from because some of them may, in fact, be contradictory.

Now, we are interested in specifics to the extent possible. If your specifics are really specific, remember that the docket is going to be open until April 28 and if you don't get everything out that you want to get out today, or if you want to have it in a written form, please submit your comments to the docket. The number is 00N0352, and it will be open until April 28. So, I just wanted to remind you of that.

Now, the way we want to set it up is we would like all groups to address the first two questions. In your packet there is a single sheet that has a list of seven questions. The first one regards the minimum standard or threshold that should be determined for a piece of information to be considered useful -- well, in general actually for the goal to be met.

The second is should certain criteria be weighted more than others? If so, which ones and why?

So, we would like everyone to address those two questions. We have raised a number of other issues, five other issues at least, that express some of the concerns that we have had as part of this process of the interim study. We hope, but we don't really expect, that we have addressed all of your issues, and we are sure that if there are oversights you will let us know about them.

What we kind of figured, again from a process standpoint, is that we have time for each group to address two additional questions, in addition to the first two on that list. So, what we have suggested is that the facilitators take a vote at the beginning of the session and, just with a hand vote, determine which other questions the group will address. If you can get to more, more power to you. You know, we are not saying you can't do that but we feel that another two is a reasonable way to go.

Now, to seem to contradict myself with regard to the detail issue -- you know, I said don't get too specific; we want specifics but not too specific -- if that isn't ambiguous enough for you, let me suggest some details that you might want to address in looking especially at question one.

Do you see other ways of combining these data to obtain a final score that would represent the standard, or the minimum criteria that should be used to reach this goal of usefulness? If so, what are they? If you go for something similar to the one-point method that Dr. Svarstad talked about yesterday and that is in the report, please consider carefully the issue of whether credit should be given for full versus partial adherence to the sub-criteria under each of the general criteria. This is an issue that I heard raised. I don't know whether it was in the public forum or if it was someone I spoke to afterwards, but I heard it raised yesterday.

We obtained information related to three drugs, and we selected those drugs for a number of reasons, one of the major ones being logistics in that how much could we afford to buy and pay for out of pocket, and which ones could you pick up at the same time and not appear to be doing something really bizarre? We had to be able to set up a scenario where people could pick up three drugs at the same time.

How would you combine these scores to create some kind of a single score? As you can see from all of the tables that Dr. Svarstad presented, they are all separate -- ibuprofen, paroxetine, amoxicillin. How do you combine those?

If you address the question of consumer involvement, which we certainly heard about loud and clear yesterday, please try to address what kind of involvement; how much involvement; what degree; on what particular criteria should consumers be involved?

So, those are just some initial thought that I had, and you may decide not to deal with any of those and that is your prerogative. I have about 9:35. Does anyone have any questions before we move? No? Good. Why don't we move to our rooms. Some of us will be floating around but you will have facilitators, and please, be open.

A F T E R N O O N    P R O C E E D I N G S

DR. OSTROVE: If I could have all the facilitators who will be doing the speaking up at the lead table here, and the facilitators who will be helping out over at the overhead station, that would be great.

Let me give you kind of the layout of how we are going to handle the rest of the afternoon in terms of reporting back on the results of your discussions which, having kind of sat in here and there on all of them, were definitely very interesting and there is definitely a lot of information there. This will certainly be a large part of getting out on the public record. We are all here and, of course, we will have all the acetates, all the overheads, everything will be looked at.

We have kind of decided on a particular format for doing this. Who knows if it will work! If anyone comes across a way of doing this kind of reporting out that you have found works especially, let us know. Let me know, in any case. What we have decided to do essentially is to have each one of the pairs of facilitators in the different groups, except one group that had three, sit at the lead table. Each one will take a lead on a particular question. What I mean by that is rather than have us go through what every group said on every single question that they addressed, which can get a little bit redundant after a while, we will have one group start out and discuss everything that they addressed for one particular question and then, if other groups want to add to that because they have something different, then we will kind of go down the line and add to those. Then a different group will take the lead on the next question.

Believe me, I had all kinds of little slips of paper, you know, deciding who was going to take the lead on what question depending on which groups had addressed which questions. Just for your information, of course, everyone addressed questions one and two, no one really addressed question three in any detail. So, no one is really going to take the lead on that. I will just kind of open it up at one point and ask if anyone has anything to say about this.

Question four, two groups discussed. Question five, six groups discussed. Question six, all groups addressed, and question seven, three groups discussed. So, just in terms of what people found interesting and something that they wanted to discuss today, that is a little bit of data.

So, what I would like to do is start with question one. But before I do that, let me thank all the facilitators because you guys have done a great job under a lot of pressure and I really appreciate it, Nancy -- we all really appreciate it, especially those of us who didn't have to do it and got to kind of walk around. That is the best part. Starting with group two, to start the discussion on question one, start the feedback, that was Nancy Smith and Gina Choy. Nancy is up here; Gina is down there.

DR. SMITH: As we get started, I wanted to share a cartoon that was given to me a few minutes ago because I think it is extremely appropriate for this. This is from yesterday's paper. It is Beatle Baily, and Beatle is talking to Zero and most of you who look at the comics at all know that Zero is not your most intelligent person. This is the sergeant talking to Zero, and the sergeant says, "follow these directions exactly and you won't get lost." Zero says, "yes." And later on you see the sergeant says, "why didn't you show up?" And, Zero is actually crawling down the middle of the road and he says, "I'm looking for the fork in the road."

[Laughter]

I think part of what we are doing right now is people not understanding what we mean by the fork in the road. You know, it is not what we just ate lunch with, and that is what he is looking for. So, determining what is useful is really going to be dependent on the audience, much more so than the health professionals, I think, and that is why I think it is very interesting that every single group addressed question six. They had to choose two and everyone included that one. So, we know that consumers are important.

What should be the minimum standards? This one was an interesting discussion and, to begin with our group, after some discussion, decided to distinguish between what were the criteria that the study used and the components of the Keystone report. Many of the criteria were components but some of the criteria were not components.

So, Gina, could you put that up first? The Keystone components, what the Keystone report says it should include are the name, warnings, indications, contraindications, precautions, adverse events, risk of dependence, proper use, storage, general information and a disclaimer. Now, the things within these criteria that are really more criteria -- they are not components, that what it is be scientifically accurate; that what it is, whichever of those, be unbiased, be legible, be up to date and be in enough detail. Each of those categories can be applied to each of these components, and this is sort of a distinction. It is not just one part of it, and it is not enough to say the list of adverse events is scientifically accurate; all of those need to be scientifically accurate. All of them need to be up to date, and so forth.

So in terms of the first, we felt that a minimum standard should be that it addresses 100 percent of those components. Everything that was considered a component should be included. And, they all must be compared with respect to scientific accuracy, the unbiasness, and so forth.

Our group could not come to consensus. We ranged from at least one individual who felt that anything less than 100 percent was failure; that the Keystone report stated what should be included and how it should be and it was either that or nothing, and had major problems with evaluations, like yesterday, where partial or complete was considered complete in some of the analyses.

I think there was probably a larger group that felt that you have to be realistic, that these were subjective and, without really mandating exactly what little each part of it had to say, there had to be some flexibility here but, at least within some reasonable range, it certainly needs to meet all of these and, clearly, if any component is totally left out then it doesn't meet those. Now, that doesn't mean two of them couldn't be combined in the same thing or something like that, but they would all have to be included.

We also felt very strongly that we needed to check with consumer groups ahead of time in order to see what they really felt was necessary for calling information useful, and the suggestion was made that we use focus groups to really get a wide variety of people to discuss what they felt would be useful information, and what would make them read the information not just, you know, throw it away in their trash can when they get home.

We had a minority opinion that said patients should receive all of the information, the same information as the healthcare professionals do. This was a minority opinion. The majority of the group felt that there is information that health professionals need to know about that the individuals within the group do not.

But, basically, our group said for number one that you have to address all of the components, and you have to do a reasonably good job, although we didn't define reasonably good job, of covering all of them to be considered.

We felt that much of what we saw yesterday did not meet our standards for being acceptable.

DR. OSTROVE: Why don't we, for this particular time, start with Tom and move down? Tom, do you have anything to add from your group?

DR. ABRAMS: Sure, just a couple of things. We found in our discussion a lot of things that Nancy Smith went over. A couple of things that we also pointed out was that standards should be tied to existing research. There was pretty much a consensus in the group that FDA or an expert panel should not make up new criteria. The information is out there for standards and we should seek that existing thing rather than reinventing the wheel. Also, the feeling was that probably it should be more objective.

The second point is that the criteria should be weighted, and I think we go into that in question number two so I won't spend any time on that.

Then, one thing that was interesting that came up is that we need to consider the outcomes of this patient information. What is it going to do? For example, behavior modification, what is the goal of it -- to make the patient more compliant taking the medication, for example, or if a patient should have an adverse event, the patient should know what to do -- call the pharmacist or physician or how to act with it.

Also, under the same thought, the group thought it would be helpful if this prompted some dialogue between the patient and healthcare professional, whether it is the pharmacist or physician. We recalled a comment from somebody who questioned how the pharmacists were so isolated from giving advice so the group thought it would be useful if this information could prompt a discussion as well.

MS. ACKER: Yes, we had some discussions around some different points on this one. We talked about whether the threshold should be full adherence to the criteria or partial adherence, whether that would be good enough. The group felt that the goal should be full adherence, that criteria should be met fully according to the panel's definition of full adherence, but that the minimum would be greater than 80 percent. So, that would correspond to level five, the high adherence to criteria.

We discussed the fact that how you define what partial adherence is, is an issue. The point was brought up that if you were given credit for partially adhering to a criteria that, you know, the majority of the side effects that were listed in the package insert were also listed in your patient information, whether that was the cut-off or whether it was, you know, just some side effect information present that would meet the full adherence measure.

The group thought -- this sort of speaks to question two as well in terms of whether certain criteria should be weighted more than others, that maybe there are some criteria where partial adherence would be acceptable but that for other criteria we would want to see full adherence.

There was a lot of discussion that depending on the drug there may be different thresholds that you could accept. If there was a drug that had a very significant side effect profile or there were significant warnings that needed to be heeded, you would want a higher threshold for information on that drug versus something that would have a more benign profile.

MS. WHEELOCK: Our group got into some more details. I am looking at consumer input needed for the forms, such as legibility, readability, printability -- also to the function which, Nancy, you addressed earlier, considering looking at the individualized content tailored to the patient. We also looked at whether the content was useful and practical, identification of disease, symptoms, side effects and management, making the scientific terms translatable to the patient and clinically relevant; also looking at cultural diversity -- written in the language of the consumer and appropriate for geographical area; and also recommending sticking to the action plan.

MS. LECHER: Greetings from group five. We had some of the same issues that were already brought up but two additional things that have not been covered yet are that we made a distinction between items that could be scored as yes/no or present/not present, and those were fairly objectively scored. Others, however, were not as objectively scored. We agree with one of the previous groups which said that some of these are mandatory and others are desirable but not mandatory.

Also, we had a suggestion that we, on the Internet or otherwise, publish what the expectations would be in the scoring system so that the public would be informed about how we are going to evaluate these patient information leaflets on the next go around.

MS. STIFANO: Actually, group six had a hard time separating question one from question two since they seem to be so intertwined with each other. Basically, we are going to echo what has already been said, and we too got specific in saying that the minimum standard would be met if the following criteria were adhered to and had the information, like the name of the drug, and the benefits of taking it, and who should or should not take the drug, drug-drug interactions, drug-food, and so on, and also weighted the amount of information that is expressed as far as risks. We felt that as far as side effects, patients should know that this is not a done deal, that you may or may not, and the bottom line is that the information and how it is presented should encourage a dialogue between the patient and the healthcare provider, and that they just don't take it on themselves to read this and think this will happen to me or this won't happen to me.

MS. NORDEN: In group three we had a lot of similar comments. One of that that was not brought up so far -- I will just say a couple of them that aren't on the overhead -- was that in order for the information to be useful it should be not too long. Somebody said limited to one page and printed out at the site. Others said that it could be more than one page as long as it wasn't excessively lengthy. This point, in terms of certain sub-criteria, couldn't be met by the use of a disclaimer, and the way the information is presented may be drug specific but certain contraindications and precautions should be communicated in a way that it really draws attention to them. That would be formatted in a question like who should not get this drug.

The other point that was brought up in our group, and I am not sure if this falls under number one or not but I don't have it anywhere else so I will bring it up here, was in terms of what happens, and how can we evaluate when patients get more than one piece of patient information when they get their prescription? For example, there is certain FDA-required patient labeling that must be disseminated with particular drugs and often patients are getting that labeling at the same time they are getting the leaflet that is generated by the pharmacy, and a way to look at if that is too much information or whether they are consistent with each other in terms of the information that is in the two different pieces.

MS. MEASE: I am representing group four, and we had a lively and lengthy discussion around this question but never really addressed the question at issue here. There were some interesting ideas brought up but in trying to address the question here, the group felt that an expert panel would be necessary to decide the information that answer needed, whereas the consumer should determine whether the information is comprehensible.

We did identify and believe that consumer research is needed, and we did have one member who provided her input that a level five, as defined in the study as greater than or equal to 80 percent adherence to the study criteria, would be the minimum threshold.

MS. BUSH: [Comment away from microphone]

MS. MEASE: We had a very lengthy discussion on what the definition or whether useful should be part of this question.

DR. OSTROVE: That is a lot of information, thank you. Is there anyone else who wanted to add from the audience? One of the reasons we wanted to have the facilitators do this is that, believe it or not, the facilitators have no vested interest in any of this. So, we felt it would be best, from a political perspective, to have a nice, neutral perspective. But if there are others who feel that something may not have been communicated, if you could come up to the microphone?

MS. LEUNG-VEGA: I just want to make the comment on Tom's feedback from this group regarding the addition of outcomes information and compliance. I think that needs to be discussed further before something like that is put into the assessment of what information is currently out there because I think it will change the whole written information piece that will make it a little bit more complicated. We are actually in the middle of something right now and probably should continue with the initial intent before we add something else to it that is controversial. That is the only comment I have to make.

DR. OSTROVE: That is a good point. Thank you very much.

So, moving on to question number 2, should certain criteria derived from the action plan recommendations be given more weight than others? If so, which should be weighted more strongly, and why? I would like to start with group eight, which was facilitated by Tom Abrams and Branda Kiliany.

MR. ABRAMS: The group agreed unanimously that the criteria should be weighted, and that is the approach that should be used. The group then discussed how we should approach this, and thought it would be impossible to rank them since a lot of these things are very important. So initially the group decided to discuss making high, moderate and low and thought that would be too much distinction, so went to high and then low, and then changed it to high and moderate because all these things are pretty important. So, that is the process.

[Laughter]

That took about five minutes. So, we don't have as much to discuss on this one. That went very quickly but, you know, I think it is important to note because what we kind of found was that nothing was really insignificant, and I think it is interesting to note that.

So, as the group classified these different criteria into high and moderate, they found that almost everything was high. So, what we decided to do is to note where it was going to be moderate.

DR. OSTROVE: Is our giggling bothering you?

MR. ABRAMS: I just don't know if I am doing something here.

[Laughter]

At division meetings I make jokes and nobody laughs, and when I am not trying to make a joke people laugh. Anyway -- no, you can giggle.

So, what we did was say, hey, what are high, what are things that are not high. And, we said storage instructions would be considered moderate unless the products were storage sensitive, for example, you have to store in the refrigerator or there was something special about the storage of that product.

The group, except for one member, thought contraindication information should be high. The one member who thought it should be moderate said that this is really the physician's responsibility to communicate, the contraindication information and kind of decide if it is a contraindication that the patient should not be taking this drug.

The next point was the self-monitoring on drug interaction information, and the group was not real clear as far as what this meant. They thought if it was for the patient to report back to the physician about a possible adverse event, that was very important; that should be high. However, if it was just to get data back to FDA, which is the physician's responsibility to report adverse events and to MedWatch, that would be lower. That would be moderate. They thought the function of this patient information should be patient specific to that patient to help them know what to do in case something happened to that particular patient.

Another thing that was a little unclear was up to date and timely information. The group specified if this is concerning safety and efficacy information, it should be high. They were a little concerned about what might be considered new and important information -- if you got so much trivia it is not that important. One of the comments was if you put too much information in things get lost.

Lastly, if you look at, I guess, the eighth criteria it includes storage instructions and general information. The group wasn't really sure what that general information was. So, they thought it was important to have a number eleven criteria. Number eleven criteria would be general information that refers people to get additional information. It was recognized that if this is a one-page piece there may be patients who may want to get additional information, and it should be referenced where they can go for more information that they might be seeking.

DR. OSTROVE: Thank you, Tom. Mary, why don't we start at your end and come back up this time around? Okay?

MS. MEASE: We did actually think that the criteria should be weighted. We actually went through and each individual in our group weighted the criteria as they thought was appropriate, which was very interesting. We saw generally that one and four were ranked the highest and eight and ten were ranked the lowest. Someone did offer the opinion that the ranking depends on the drug.

We also ran into some differences on the interpretation of the various ten criteria. We generally thought that the negatives should be ranked higher than the positives, i.e., the contraindications, the adverse events versus storage and handling instructions. There were members that felt that nine and ten should be given, and then there were a couple of people that offered the opinion that if we thought they should be given, they should be ranked the highest.

Again, on number seven, specifically we ran into some differences of opinion on the interpretation and also thought that it was an ambiguous criterion.

MS. NORDEN: Our group thought that, first of all, the criteria should be reworked and more closely track the action plan, which is on page 21 of your handouts, where there were eleven categories instead of the ten, and to follow the report in terms of the ordering and the weighting that was recommended in the report.

Also, there was discussion concerning the risks and benefits, and that it is really drug specific to determine how much of that there should be. Also, this kind of comes back to what was previously said, that having information presented is really important and some members felt that there was a tendency to try to present a lot of risk information, which I think was also brought up yesterday. Patients need to know the benefits of the drug as well, but there should be a balance between the benefits and the risks, and if there were significant risks associated with the drug, they should be presented up front.

As far as the ADR's and monitoring adverse events, the group didn't think it was useful to present a laundry list of all the side effects associated with the drug.

MS. STIFANO: We also weighted everything. You know, we placed, like, in order of their appearance the criteria, one being identifying the drug and its benefits, and the second one being kind of a combination of identifying the contraindications and what to do, as well as specific precautions and how to avoid harm. So, that would encompass who the should not take and any drug-drug interactions or any drug-food interactions.

Then, how to take and receive, the third and, again, if it is to be taken with food or without food. The fourth would be addressing the proper monitoring of adverse events. Again, we felt that a laundry list would only be frightening to the patient. So, you would have those adverse events with a certain level of significance. For example, if the drug carries a black box warning, then most certainly you would want to address those concerns. Or, you know, if we had to have a numerical value, it would be perhaps greater than five percent experience but, again, that would be dependent on the class of the drug and the population taking it. So, for example, if the top five percent are headache, general malaise and everything else, that could be expected patient of the disease itself. That is not necessarily useful; that is what they are being treated for. It would be more out of the norm, out of the ordinary. So, if they are being treated for a headache and their toenails hurt ferociously, then put that information in there, not something that they don't already know.

The others just kind of fell -- you should weight them but they fell into more of an equal layering below those pieces.

MS. LECHER: We had some overlap with what has already been said but we also had some additional items. At this point, I just want to go back to question one and say that we had some difficulty in answering question one. We didn't know whether we were to answer the question in terms of the data set that we had been presented with, a data set that might be generated using new criteria that we would develop, or just in general. That caused some confusion on the part of members of our group about specifying what we were really talking about.

Now getting to question two, we did believe that some criteria and sub-criteria should be weighted more. The criteria that we thought should be weighted more especially included how to avoid risks and comprehensibility. It was then suggested that of the criteria on the list, criteria one, two, nine and ten could be applied in considering all the other criteria.

There was also a discussion, as there has been in the other groups, about how to prevent adverse events. That is something else -- never mind, I am reading my notes incorrectly.

We called the points under the criteria sub-criteria and they must each meet a predetermined percentage, but this may vary for each of the sub-criteria.

On the second page, we say weigh these sub-criteria differently then add or delete sub-criteria, and I do believe there were members of our group who had suggestions that I hope they will pass along as to what additional criteria there should be or what should be changed or deleted. Then, that there should be certain standard information in all of these. For example, the fact that the leaflet is important and you should read it carefully; storage information; sources for additional information about the product; and how the information was generated, where did this information come from.

Then, on our last overhead we had a suggestion that terminology be in lay terms whenever possible. There was some lengthy discussion about the term scientifically accurate. Some of the members felt that the word scientifically suggests that the language would be scientific and perhaps not accessible to lay people. I think there was a consensus that when possible terminology should be simple.

MS. WHEELOCK: Our group obviously is the group that doesn't agree with the rest of the groups here. We decided, unanimously actually, not to weight and we had a lot of reasons: that different drugs require different weights. That implies a priority. It was felt in the group that there would be a subliminal influence if you actually prioritized and what would be actually the priority because it may be different based upon the different drugs. Different patients require maybe different weights, and new information as it is known may alter the weights, such as adverse events and interactions. Again, our action plan suggestion was that four items are of importance and those are the ones to actually follow.

MS. ACKER: Our group had a lot of the same points of view as has already been discussed. Most of our criteria came out as being highly weighted or heavily weighted. As you can see up there at the top, the legibility being very important.

We had a discussion about maybe there is a tension between information being scientifically accurate but also concise and readable, and that if scientifically accurate means complete, it may be a challenge to present that in a concise manner.

The group brought up the point that it looked like from the interim results that the criteria on being unbiased in content had already been pretty well met, and so maybe that is of a lesser concern and that would hold a medium weight because that didn't appear to be an issue in the interim report. Although they felt it was important that information be unbiased but that if that is not something that you are seeing in the sample maybe you could weight that lower. That is all we have.

DR. SMITH: Our group agreed with Leslie's group. We took the easy way out and said they should not be weighted. One of the reasons given, and probably the most important reason -- and, again, we separated the components from the ones that were more descriptions of how the components should be analyzed, and the Keystone report did not weight the components, so our group felt that it wasn't our place to weight the components and, you know, it probably would differ for different drugs if you tried to do it.

DR. OSTROVE: Any additions?

[No response]

In that case, what I would like to do is actually skip around a little bit. Since all of the groups also addressed question six, I would like to skip to question six. That is, should the evaluation panel include consumers with varying educational backgrounds? If so, how should they be involved in the evaluation process? I would like to start with group six, Tony Stifano and Spencer Salis -- you notice there is an abundance of women doing this? There need to be more men doing facilitation. If you guys would start, that would be great.

MS. STIFANO: Obviously the answer to this question to our group was a resounding yes, that scientists absolutely should not be involved in assessing the usefulness or understandability of information that is going to the patients. Basically, what we did was to suggest that we keep the existing process in place -- the writing on the slide is a little out of order since we really just had so many ideas going around, but to keep the existing process in place but then also to establish a protocol to evaluate the usefulness of information by getting input from consumers, and vendors, and industries and pharmacies so that you really could come up with everybody's point of view on what they consider to be useful, and then put that information to use in how you would evaluate it.

One of the other ideas we came up with on getting patients more involved in looking at the actual piece of paper that they are handed when they get to the pharmacy is to perhaps do, in a non-threatening sort of way, a mini-comprehension test after they have had a chance to look at the information, and ask them so that you really do get an idea of did they comprehend it or not, in a group situation the chances of a person just nodding in agreement with everyone else because they don't want to admit that they didn't understand it are very high.

One of the other things we suggested is to perhaps conduct focus groups to get some answers, and to have these focus groups be comprised of consumers, and vendors, and pharmacies and the industry. There was a lot of interest on the part of the industry to make sure that patients get good information, and to publish the results of the outcomes of these focus groups and then ask the question of ourselves, well, what do we do that we find that information is, in fact, getting in the hands of consumers, and we see that it is; the numbers are rising, but if the consumers are getting it and they are not reading it or, worse yet, they don't understand it, then what good is it?

So, basically, the bottom line that we came up with is that, you know, if the information being disseminated is not understood or used, then how useful is it? So, we think that the more consumers you can get involved, not only from different educational levels and backgrounds but different language skills since the two are very different, the better off we will all be.

DR. OSTROVE: Thank you. Leslie, we are going to start with you and then move this way.

MS. WHEELOCK: We had four major areas about the consumer and my group also agreed that the consumer should be involved. In fact, in all the questions input of the consumer was very important.

The first thing was involving the consumer just by asking the consumer, starting right out and, you know, assessing or asking the questions, things such as what is really their method for preferred information dissemination? We have been speaking about written but perhaps the spoken word, the oral communication, as well as considering future technologies where things may be on the web. Also, asking them questions about will they take the med? Looking at assessing the lifestyle issues, and also what is their initial source of information? Is it the written; is it the oral?

The second major area was what types of consumer input would be important? Actually, the group decided a consumer would be a very valuable asset to the expert panel with input into the design and actual evaluation of studies, helping the expert panel develop questions and also analyzing those questions, looking at giving feedback about format and content, as well as efficacy of oral counseling, and also oral information and feedback about the oral reinforcement of the written. Also to help evaluation the sample and differentiating it -- the sample form, differentiating it and comparing it to the expert panel and what they came up with. Then, also looking at evaluating outcomes -- is there really a return on investment with all of this?

We also looked at the consumer in recommending the various characteristics about the consumers that would be involved in this process, and really wanting a diverse sample, looking at the largest user, which is right now the non-college elderly, and also to include diverse socioeconomic, cultural, ethnic and educational so that the literacy levels, interpretation and comprehension are actually addressed.

One of the suggestions was to refer a model, the over-the-counter. Studies, again mentioning focus groups, be involved and inclusion in those focus groups would be chronic users, first-time users and also new users because their feedback maybe is very different. Then, looking at the types of consumer, expanding from more than just the end-user who is the patient but also to include end-users such as the mail- order people and companies that are involved and also the chain store pharmacies.

DR. OSTROVE: Thank you, Leslie. Tracy?

MS. ACKER: Our group also undoubtedly thought that consumers should be part of the process here, also that the panel include, you know, people from various groups; that it be multi-cultural, mixed demographics according to age, rural versus urban, etc., different educational levels.

The suggestion was brought up that consumer advocacy groups could be a source to both spread information but also as a source of volunteers to participate in this process.

We also felt that rather than, as was done with the expert panel where they had sort of a questionnaire to check off did it meet certain criteria, that the consumer actually be asked about comprehension of that information and to read the patient leaflet but then be asked questions about how do you take this medicine; when should you take it, etc. That would be more meaningful and wed speak to many of the criteria there in terms of how easy is it for the patients to actually use these leaflets.

The suggestion was also brought up that consumers be involved in the process through focus groups, also as individual consumers being asked to rate these leaflets but also to do a survey among people at the point of service at their pharmacy when they get the prescription -- to actually get a leaflet but a survey as well to fill out and mail back.

MR. ABRAMS: Just to emphasize what Tracy and Tony said, the group thought it should go beyond education. Tony mentioned language and Tracy mentioned demographics such as age. Just to add to that, the group said economics and comprehension levels should really be addressed. So, you really want varied backgrounds.

Leslie referred to her group saying that we should look at other models, such as OTC models. My group said the same thing, and they pointed out to the Australian model and the European model to look at how consumers are involved.

Lastly, they stated that panel members should evaluate different sections of the patient information.

DR. SMITH: Our group also had a resounding yes, that consumers need to be involved and, you know, a random sample which means all cultural, all educational levels, and so forth.

We felt that consumers needed to look at all the components but that first, you know, the experts would do an evaluation, similar to what was done in the study that Bonnie described. That would come first and if all of the components weren't there, in our estimation it had failed already. But, if all of the components were there, then the consumers would look at it.

One suggestion was made, that the consumers and the experts actually look at it together and have some discussions, which might be very interesting, but the consumer would have primary input on is it useful or understandable. We talked a lot about whether the best way was focus groups or something like a one-on-one interview, something other than the study that Bonnie described, the brief study which seemed to require a certain education and comprehension level in order to even participate very much. So, we were trying to get ways so that people with lower education levels could be involved because, you know, we feel that there are a lot of people who probably would not necessarily be able to even do an evaluation as it was described.

An interesting point was brought up, that maybe we ought to target, if possible, consumers with the target disease, whatever the drug is indicated for. There were some people who felt that maybe that wasn't a good idea because those people would have more knowledge than someone else but, on the other hand, if you are looking at people that are just getting the new prescription for the very first time they probably wouldn't have that much knowledge and they would obviously be more interested in it since they would need the drug than just someone off the street. So, we felt it might be interesting to try to target them.

MS. MEASE: We did believe that consumers should be involved, and we grappled with defining what a consumer is. Somebody offered the definition that a consumer is anyone who takes medicine independently or gives medicine to others. We thought that was an important point to make.

We believe that we should have a large number of consumers on these panels, and we generally felt that we should start with people that would represent the majority of the population and then, after establishing those criteria, we could move on to subpopulations -- multi-cultural, age groups and also health status and the blind as well. We felt that we should involve consumers in the comprehension and testing portions of this, but we did feel, again, that the experts should tell us what consumers need to know.

We felt that keeping an eye on technology for the future would be very important as that may be the venue in which consumers are going to receive more and more information over time.

USP could also be a resource for setting the standards because they already do that for product so maybe the could do that for information as well.

MS. NORDEN: Our group also strongly believed that the consumers should be involved. Our points are almost exactly the same. These were kind of a little different concerning involvement of consumer groups and consumers in the whole process -- our group believed that an outside group should be the ones that conduct the study; maybe get competitive bids; that we should ask Congress for enough money, and to mention the IOM and medication errors; and that before the protocol or the evaluation process is implemented it should be shared with the stakeholders for comment before the implementation process.

MS. LECHER: We have two main issues. They were primarily covered by other groups but we have a slightly different slant. We also all agreed that consumers should be involved. There was some interest among some members that their participation be limited only to issues on which they are experts, which is things that would affect informed consent decisions, legibility and comprehension.

Also, we should utilize consumers as a valuable resource in conducting studies with these labels, such as label comprehension studies or cognition studies. We also mentioned actual use studies where they would take the product home and see if they followed the directions.

I would also mention that someone in our group did mention Australia as another source of information for us. Apparently, they have a history of evaluating patient information and perhaps we can learn something from that.

DR. OSTROVE: Does anyone in the audience have anything to add to their groups? If you could give your name and affiliation, that would be great.

MR. FOSSEN: I am Merle Fossen, with the Academy of Managed Care Pharmacy. In looking at question six and looking at how consumers should be involved in the education group, I would like to suggest that maybe there be a pilot study where the study focuses on a return on investment in terms of the outcomes, be they economical -- dollars and cents, clinical -- does the patient get better or worse, or humanistically -- does the patient improve as far as the quality of life goes.

MS. HARE: Doris Hare, American Foundation for Maternal and Child Health. I don't think that consumer involvement should be limited to informed consent and that type of thing. There are many consumer groups who are extremely knowledgeable on the drugs that their group would be likely to take.

MS. DAY: Ruth Day, Duke University. I just wanted to comment on what is meant by cognition studies. This group is well aware of comprehensibility, the ease with which individuals understand information. That is really important. There are other cognitive functions that are important that can be tested with these leaflets as well -- the extent to which they perceive and know that the information is there; whether they can remember it because, after all, in the real world they don't always have the leaflet handy; and then the most important thing is problem solving -- can they do what needs to be done when circumstances arise. So, the call there for cognition studies is for comprehension but more broadly-based cognitive processes.

DR. OSTROVE: Thank you, all, for your additions. We appreciate it. I would like to move to question five, should there be more detailed assessment of factors affecting readability and legibility for consumers, for example, type size, style, spacing and contrast? I would like to have group four, which was facilitated by Linda Brophy, Kathryn Aikin and Mary Mease, take the lead. Mary?

MS. MEASE: We did believe that there should be a more detailed assessment of these factors, and generally the overall format was a very important thing to consider.

This is very difficult for me to read. Let me walk up here. There are several models that we might be able to use as a resource that were presented. Someone mentioned some OTC consumer studies in the Final Rule, as well as some FTC proceedings on OTC advertising in the 1970s, and maybe we don't have to reinvent the wheel here; maybe we can look at some other resources that have some well-established processes in place.

One person did offer the idea of looking at a recent studied that was commissioned by the USP, entitled, "Optimizing Patient Comprehension Through Medicine Information Leaflets," which really seems to be very applicable here and might be very helpful.

Another important point that we had was we keep hearing about these one-page patient information leaflets and maybe we really need to get past this mind set. This is an assumption perhaps that has been made, and patients really do want more information, and in doing this we recognize that the one-page mind set may have come from, say, the technology limitations, i.e., the printers or the software, and that perhaps this is a systems issue and maybe the software companies need to be involved in some of these discussions to iron out this detail.

DR. OSTROVE: Thank you, Mary. Karen, can we start with your group and then move in that direction?

MS. LECHER: We had some lively discussion on some of these issues and where you see question marks we had especially lively disagreements.

There was a suggestion that the length of the leaflets be limited to avoid information overload. However, some other members thought that this would not be a good idea. If there were a limitation on length there would be clear information about where to get more information about the product.

Then, there was a suggestion that there be a summary of the most important information up front. Again there was disagreement because some thought that this might raise questions of liability or patients may not read further into the information if they had a summary up front.

There was an interesting suggestion about having gradations of information. The pharmacy would be capable of producing, for example, three different forms of information that would vary in terms of the amount of detail and that consumers could be given samples of each of these and then they would choose which type they would want to have generated for them for the product they were purchasing.

There was also a suggestion to add some additional factors to the analysis, for example, cognitive factors. We had room only to put a couple of these examples here. These would include chunking or grouping information that is similar and coding, which is providing appropriate headings for the information.

There were some other suggestions too. If we have time perhaps those could be discussed further today.

MS. STIFANO: Our group came had a lively discussion too about the suggestion of having vendor standards, vendor/pharmacy standards, since we did find that apparently not only are the vendors providing information to pharmacies but pharmacies can, in fact, themselves pound out whatever information they wish. So, the suggestion was to have industry play an active role in setting these standards, to work together with these vendors that are providing the information to pharmacies. The specific suggestions included the 10-12 point Aerial font, lots of white space, preferably one page but not at the expense of skimping on certain information, and we also batted around the idea of having printer quality standards for the pharmacies so that you don't have these God-awful dot matrix printers that are chopping things off or are obliterating parts of letters.

Part of the rationale for keeping companies involved is that one of the other issues was that the most updated and current information be made accessible, and this would be one way of assuring the most updated and current information be made accessible.

The only other point that had little to do with this but something that might need clarification is it seems there is some confusion right now on terminology in terms of med guides versus a PPI versus what the printout is called from the pharmacy. So, perhaps we could start with something very simple and clarify what all those mean.

MS. NORDEN: In terms of evaluating for readability, our group felt that it should be sort of two-pronged, that there should be both a consumer evaluation as well as an expert evaluation and the experts should be experts in communication and readability.

Basically, they thought that the factors in Appendix G of the action plan were very good to follow in terms of laying out criteria for fonts and things like that.

As far as the evaluation that the consumer should be doing, they thought that they should look at actual leaflets -- well, also maybe at prototypes too but at actual leaflets that are being passed out, looking at them for format and readability as well as comprehensibility and also usefulness. This came up a lot of times, that these are really not that helpful if they are not actually useful to the consumer, and ask the questions like what kinds of information do you want included, and is the amount of content adequate. Not have the consumer necessarily evaluate the scientific content of it, but is the amount of information presented adequate.

MR. ABRAMS: Group eight thought readability was sufficient as far as assessment of factors. They, however, pointed out that readability needs to be more detailed, and pointed out two situations which are very related, situational dependency -- it depends on how complex the disease state it is and how complex the drug is, the mechanism of action and things, how hard that is to translate. Going along with that, they point out it depends who you are communicating to. We can't just group consumers into one big group.

One group member related an interesting story where they designed an HIV brochure which was at a lower educational level, grade level, and they thought it was pretty good but, in fact, a couple of consumers evaluated it in their focus group and they became really very annoyed with it because they felt it was demeaning, like insulting to their intelligence. So, it is a hard situation that they pointed out.

Also, just a comment -- and I don't know if the group made this remark during this question but Mary mentioned that we shouldn't be restricted to one page because there might be a lot of information. Group eight made a point that if you have one page, no matter how much you have, how short or long it is, you really need to always have a reference where you can get more information because some people will seek more information and it should be made easier for them.

DR. OSTROVE: Did any of the other groups discuss this?

DR. SMITH: I just wanted to make a comment.

DR. OSTROVE: Please.

DR. ACKER: One of the members of our group is from Sweden and he shared with us that their country has some experience with evaluating the readability and legibility of patient information as well, and that may serve sort of as a model. I just wanted to point out to the group that Sweden may be a model as well.

DR. OSTROVE: Thank you. Is there anyone in the audience who would like to add?

[No response]

Moving right along, three groups I believe addressed question seven. Question seven is this report collected patient information from United States retail pharmacies. Are there ways to expand sampling to include mail-order or other non-retail pharmacies? Tracy Acker and Mary Kremszner, I would like you to start.

MS. ACKER: We had several members in our group who are from the non-retail world, from mail-order pharmacies. So, we had some interesting discussion around this point. The group felt that, yes, some other settings would be important to look at in terms of information patients are getting.

Some ways to sample that universe, of course, would be to pose patients and mail a prescription to a mail-order pharmacy or send a prescription to an Internet pharmacy, through those mechanism. The issue came up though for some of these mail-order pharmacies that they only fill prescriptions for their clients, for the insured members. So, in terms of study methodology that may be a barrier, to have a fictitious patient. You wouldn't get the prescription filled and billed through the insurance company. Some patients who get their prescriptions filled in that manner, they pay cash sometimes if it is a non-formulary but in terms of using that as a setting for a study, that might be a barrier.

The point was brought up that through Internet pharmacies that may be easier to use in a study because you could pay cash that way for a prescription.

The idea was brought up to select some of the test sites through working with their trade organizations for mail-order pharmacies, that they might serve as a source for pointing out the sites that work well for this study.

Then, the settings we came up with for what might be interesting to look at would, of course, include mail-order and Internet but also institutional settings, such as outpatient when a patient might be started on a new medication in a hospital but then be discharged on it. That might be the first time that they are on that medicine and that would be a way for them to get the patient information, and we should include that. Also, HMO clinics, if the HMO requires the patient to fill a prescription at their pharmacy, that would be a setting. And, also not to forget the setting of physician dispensing. There is a small minority there but we should look at all the places where patients might be getting an introduction to a new prescription and that would also include physician offices. Also, the idea was brought up for the weight loss clinics, for instance, that are prescribing diet drugs. That would be a setting as well.

DR. SMITH: Our group also discussed this fairly extensively, and we felt that they must be included because they are increasing. More and more people are buying their prescriptions, especially over the Internet. We thought the Internet part would be relatively easy because there are currently only four Internet sites that are approved by NABP.

We just had an extensive discussion and nobody in our group had ordered a drug from the Internet, and I am curious how many people in this room have. Could we have a show of hands? We have a few; great.

The question was, does the information that you get -- when you send in an order and you get an e-mail confirmation of your order, do you get the information then or do you get it with the drug, or both or, of course, there is always the possibility of neither. But we felt it would be good if you would get it right after. Perhaps both was the ideal way since it would be so easy here. If you got it with your confirmation, maybe you would read it because you still didn't have the drug but you would have a little bit of time to read the information before you got the drug. So, perhaps you might be a little more apt to read it. But we did think we would not be sampling from these sites, that all of these sites would have someone buy the drugs we are particularly interested in.

Managed care -- we discussed the problems because in most cases you have to be a member. We also talked about the acute care setting, patients starting drugs in the acute care setting, and we felt that we needed to check on what percent of new prescriptions are given for the first time within a hospital or other acute care setting, and if it is a high enough percentage we would need to include them, or if it is not too high the other alternative would be to put the caveat that those weren't included in the discussion. But we felt that, no matter what, the Internet and the managed care needed to be included because even though at this stage they may be in the 15 percent range of all prescriptions, they are rapidly rising.

MS. MEASE: Like the other two groups that looked at this question, we did believe sampling should be expanded to the non-traditional pharmacies -- the Internet, mail-order, managed care. When looking at what type of patient information should be included, we went back to the Rule 104-180 and thought that the term useful written information to 75 percent of individuals the term individuals should be defined because we do have children that do take their own medication, especially if it is a chronic condition. So, we thought that individual should be defined in the Rule.

DR. OSTROVE: Did any of the other groups address this?

MS. NORDEN: We didn't specifically address this question but almost the exact same kind of points came up when we were talking about another issue and, basically, there was a need to sample other venues besides just what was sampled in this, and look at more drugs also, besides just the three.

DR. OSTROVE: Thanks. Is there anyone from the audience that would like to add -- okay, name and affiliation.

MR. HOEY: I am Doug Hoey, from the National Community Pharmacists Association, and we are a trade association for retail pharmacies. Just as an added note on what Tracy presented, on using trade associations to examine any of the managed care settings, I guess I would just like to caution that there is a randomized sample. Just as there weren't trade associations used to identify retail pharmacies, it introduces a certain amount of bias. So, I would recommend that trade associations be advised but not used to select the pharmacies.

DR. OSTROVE: Thank you. Actually, none of the groups addressed the last question, at least specifically. Did I miss question four? I skipped question four. Let's go to question four. Question four is, should the assessment include additional criteria or types of in and, if so, what? I would like to have Leslie Wheelock and Dorothy Ballmann start with that.

MS. WHEELOCK: Our group looked at two major areas, the study considerations and evaluation of the leaflet. With regard to the study evaluations, a larger sample size was actually discussed, and large meaning not only in size but also in its diversity as well. Also, asking some questions to assess are there some differences with drug classifications, and also looking at patient comprehension, and some more detailed criteria evaluation, and also how patients are using and interpreting the data they are getting.

With regard to the evaluation of the leaflets, were there any pharmacy modifications, comparing the FDA's approved patient package insert with what is distributed by the vendors; also looking at who and how the leaflets are developed; and also does the information in the leaflets provide sufficient information. So, that is what our group came up with.

DR. SMITH: We basically just had one comment but I think it was important, and this was that the Keystone criteria makes the distinction that bolded boxed warnings need to go up front but there was a strong feeling that all bolded warnings, anything that had to be bolded as a warning, really needed to go up front and get more play right from the beginning.

DR. OSTROVE: Did any of the other groups have additions to this kind of area, or any comments from the audience? Okay.

MS. DARCY: My name is Mary Lynn Darcy and I am with Micromedex, a vendor. On the last slide that was just put up there, there was a comment on the boldness and the black box warnings, which we didn't specifically address as an issue in our group but several people, as the discussion went around, did discuss this and maybe there were about four of us in the group who felt that this was the one thing we differed on with Keystone, and some of that was due to the USP research that had previously been done that talked about chunking data. Of course, we could have more research, however, prior research does support that when you separate out the black box warnings section you then have warnings in two different areas. You no longer have chunked the data. We felt that wherever the precautions section would be, that is where the black box warnings should be. They should be incorporated in that area.

DR. OSTROVE: Thank you.

MS. COHEN: My name is Susan Cohen, and I think that in conjunction with this we should encourage, particularly HMO's where the physicians are allowed ten cents -- ten minutes -- that is interesting, and that was subliminal --

[Laughter]

-- that they have someone who can assist the patients as they come through the office. After all, this will be as good as a physician in the long run and people need the physician. I would like to encourage industry to have some enlightened self-interest in doing some interesting things in terms of labeling. As I understood, the label that I would have loved to have seen on bottles is not possible but if you use the black box on a piece of paper why can't you use something on a bottle too?

I think whatever program is developed, it should be reevaluated in a certain number of years to see if it is really working, and go back to the consumers -- that is what it is about -- and find out if it is really working for them, or are there things that they really would like to have.

I am delighted that you have had this, and I hope that we can go forward and we can all get our medicine and understand what we are getting.

DR. OSTROVE: I think we all agree with you on that. As I was starting to say before, none of the groups formally addressed question three so we are going to wait and see what comes in with the written comments for that. I know that group three didn't get to be the lead on any of these, which is Janet Norden and Melissa Moncavage. So, I just want to acknowledge that they are here, and Karen Lecher and Rebecca Redman also did not get to be the lead -- not that they were unhappy about it, but I want to acknowledge their presence here and their contributions.

In fact, Karen sent me a note and said that their group had some additional comments that didn't seem to fit clearly into any of the questions addressed formally. So, I figure we have a little bit of time, Karen, if you would like to present those as a part of your group's process, I think this is a good time to do it.

MS. LECHER: We didn't put these on the overheads because we didn't know if they would be appropriate for presentation. The first few that I am going to give you are primarily about the content of the information that members of our group suggested be in the leaflets and, therefore, they would then be reflected in the evaluation criteria.

There was a suggestion that there be more balance of the benefits and the risks; that current leaflets often emphasize the risk information and don't give enough information about the benefits so that patients might be scared off from taking it or may not understand the implications of they don't take it appropriately.

There was also a suggestion that the leaflets tell when the product was approved to give the consumer a sense of how long it has been in use. Related to that is information about the number of patients who have taken the medication.

There was a suggestion that giving percentages of adverse events may be misleading to consumers. There may be under-reporting of adverse events, or they may not accurately reflect the rate of adverse events. There was not consensus on any of these items that I am telling you about, this one in particular.

There was a suggestion to group the adverse events as more common, less common and rare, and define what those terms mean, perhaps giving numerical ranges rather than giving percentages associated with each of the adverse events. And, also to tell whether the adverse events are being reported from clinical trials or from postmarketing experience.

There was one other suggestion that doesn't deal with the content of the leaflet. If there are low ratings, and one of our members has looked at each of the ten criteria to see which ones seem to be rated higher and which are lower -- if there are lower ratings, those particular criteria should be examined in more detail, perhaps looking at the inter-rater reliability and the sub-criteria that were used, and perhaps those could be worked on before the next study.

Finally, ideally each of the leaflets would be individualized to the condition for which the medication is being dispensed so that information that may not be applicable for that use would not be on the leaflet. Thank you.

DR. OSTROVE: I said that my brain was already full yesterday -- yes, Dr. Svarstad?

DR. SVARSTAD: May I make one comment?

DR. OSTROVE: If you could come up to the mike and give your name and affiliation, yes.

DR. SVARSTAD: Bonnie Svarstad, University of Wisconsin. Several people made a comment so I want to raise a question here and get your feedback on this because I struggled with how to answer it. One suggestion that was made a couple of times was that the actual evaluation form with the sub-criteria be circulated in advance for comment. I can see all kinds of good reasons for that -- to give the public, vendors, pharmacy, etc. a chance to have input into that. I wonder if those people that made that suggestion have thought of the downside of that?

The downside of that is one that a university, a high school or a teacher often faces when exams start to circulate. The one possibility here, for example, is that you do circulate the ibuprofen, the amoxicillin and the paroxetine evaluation form and then whoever is responsible for that information makes sure that those three were done well but that all of the other medications were not attended to. Does anybody have -- do you get my concern?

My concern is that, on the one hand, we want feedback, as much feedback as possible -- we, meaning whoever is responsible for evaluating. On the other hand, the concern is that by circulating a particular evaluation form for a particular medication may lead to some attempt to improve only those sheets rather than the sheets that are necessary for all drugs. And, I don't know the answer to that.

DR. SMITH: I would to address that. To some degree, those will probably get the most change, but I also think a lot of groups may use them as samples. So, it could be useful. I also think, and correct me if I am wrong, Nancy, that they will not be the same drugs in the next. So, these three drugs, I guess, are the only ones that will not be included in the next go around because they have been done. So, if anyone is trying to correct a few labels in order to maybe score higher next time, those are the absolute worst ones to correct right now.

DR. OSTROVE: And, I think it is possible that Dr. Svarstad's point, in regard to some of the comments in the groups, is that before we use it for the next time that they be sent around.

DR. SVARSTAD: What I thought I heard was several suggestions to circulate the evaluation forms [comments away from the microphone].

DR. OSTROVE: To repeat what Dr. Svarstad said, what she had heard in several of the groups was that, in fact, the test forms that will be used in the next evaluation be circulated, and that may have been a mis-perception on her part but that is what she was responding to with regard to this particular comment.

DR. SMITH: I think certainly if the criteria are somewhat different, then it would be good that those be circulated but not the specific details for the specific drugs probably.

DR. OSTROVE: Again, I think it is important for everyone to understand that we, today, have heard a wide range of suggestions, and I am not going to make this suggestion because actually legally we are not allowed to do this but if I wasn't constrained I would say go lobby your congress person for money because a lot of, you know, what we are hearing today would constitute a fairly expensive undertaking. I mean, we will have funds but I don't know if we will have those kinds of funds.

So, we are going to have to sift through all the information that we have heard today and all the information that is going to be submitted to the docket within the next couple of months and decide what we want to focus on. Where obviously there seems to be some consensus, clearly we need to go in that direction. In the many areas where there isn't complete consensus, then we have to make some decisions as to what the right thing is to do given the constraints that we are working under in terms of resources and funding. So, I think, you know, that is something for everyone to take back. It is invaluable to have had you all come today and to give us your feedback, and it will all be taken into account.

I think that we also need to consider the whole issue of continuing, even before we do the next evaluation, getting some more feedback and if we make any drastic changes to what we have here, certainly getting back to the Keystone participants and the people who are involved in the process and letting them know what those changes are. Of course, the biggest question comes down to, well, what I think is a substantive change may not be what someone else thinks is a substantive change but we can deal with that. We have dealt with those things before.

None of that undermines the importance of what we have done here today and the importance of your submitting your comments because we will be considering all of them. I can't stress the importance -- I feel like I am repeating myself but I think it is okay to sound like a broken record in this particular instance because I am really being sincere about it, and really do want to hear from you. Yes, ma'am?

MS. BUSH: Pat Bush, USP. I have a question. It must be that you considered not buying the drugs. That, of course, would make the study vastly less expensive. So, why did you turn down doing exit surveys or observational survey type work, which has been done in the past, I gather, rather successfully?

DR. OSTROVE: If you go to a pharmacy and simply say give me what you are going to give -- you mean exit surveys --

MS. BUSH: Yes, you stand and you watch what happens a patient and --

DR. OSTROVE: And you ask them to give you the piece of information that they have gotten?

MS. BUSH: And, then when they exit you ask them, yes, and then you look at it and talk to them but you see do they get something.

DR. OSTROVE: You are going to have a huge range of drugs to do something like that.

MS. BUSH: Yes, but you ought to have general criteria.

DR. OSTROVE: But the general criteria need the drug-specific sub-criteria in order to be assessed. So, you would have to have drug-specific sub-criteria for who knows how many products. We did think about other ways of doing this, and we really felt that from a methodological perspective that this was the best possible. If you have other ideas, please send them in.

MS. BUSH: Bonnie says the Hawthorne effect is so big there.

DR. SVARSTAD: [Comment away from microphone].

DR. OSTROVE: This effect arose from some very, very old studies that show that when people know that they are being observed, that alone changes their behavior.

MS. BUSH: We all know that but there have been a lot of this kind of studies. Are they all then terribly biased? They meet the people outside after they leave the store, or they go to their houses, or other ways.

DR. OSTROVE: I hear what you are saying and, as I said, we are listening and that is why we are here.

MS. BUSH: You are right, the pharmacists would probably pretty quickly catch on when the patient says, guess what happened to me after I left last time. So, okay.

DR. OSTROVE: Thanks.

MS. RUCKER: Lee Rucker, National Council on Patient Information. Having participated in the action Keystone committee a few years ago, what is exciting about today, revisiting this, is that I think we have a real opportunity with the timing for two factors. One is the fact that just a few months ago we had the Institute of Medicine report on medical errors, which I think did a great deal for raising the public consciousness about the problem and solutions. So, I think we can capitalize on perhaps that raised public consciousness for the short term.

The other action that is coming down the pike, and every day it seems to be more imminent, is the expansion of an outpatient Medicare drug benefit, which certainly affects the people who use the medicines the most. However, opening the access floodgates could bring with it, of course, more problems.

DR. OSTROVE: Sure.

MS. RUCKER: And, therefore, more practicality and appropriateness and importance of our effort here over the past two days, and down the road in terms of research. So, I think that for each of our organizations that we are representing we need to embrace this opportunity and see it as an obligation to communicate to our respective stakeholders what has transpired here, where we are going, and how we can all do our part.

DR. OSTROVE: Work together. Absolutely. Absolutely. I second that.

MS. COHEN: Here comes a grade school question. If you are going to spend money on all these wonderful suggestions, why does it have to be drug specific? That is what I don't understand. It is expensive to do these things, and I think every bit of information would be valuable to consumers, of course, depending on the medication and how you tailor it to the medication. So, I am a little puzzled why you have to do specifically and just not do it in terms of what is best for everybody and that gets the myriad of medications that are available to consumers. I have missed something here. Help me; I really need it.

DR. OSTROVE: Do you have an answer to that in the back? Sandy, could you come to the mike?

MS. ESKIN: Susan, in answer to your question, the whole plan is an attempt to improve the actual information that consumers get. What Congress asked is that the Secretary has to evaluate compliance with this voluntary plan. She has to assess what is out there in the marketplace. So, obviously there can be differences of opinions on how many drugs should be looked at and how many leaflets should be looked at, but there needs to be evaluation of actually what is out there, otherwise you are not doing what is being asked.

MS. COHEN: [Comment away from microphone].

MS. ESKIN: Well, of course, but there has to be some way. There was a methodology applied in this study and, hopefully, all of us who can comment on the specifics of the methodology can, hopefully, improve the methodology of the next set. You can't look at every single pamphlet; you can't look at every single drug. So, you sit down perhaps and look at the whole marketplace and look at perhaps the most widely prescribed drugs, or you take a whole range of drugs -- one that has a black box warning, one -- these are all possibilities. You may wind up at the same place you are right now, which is three drugs. We don't know but I think you clearly have to do real pamphlets involving real drugs. I don't think you need to buy them. That is another issue to discuss. Maybe I am thinking something that is obviously faulty but what if you asked every pharmacy to submit samples of what they actually put out?

DR. OSTROVE: I think that this is a discussion that could go on for half an hour, hour easily, and we only have a few minutes left. I would suggest that any of you who have suggestions concerning the methodology, please submit them. Believe it or not, we actually have reasons for why we do things and so we can respond, and we can give you the reasons for why we make a certain decision. But if you have a good rationale and if it is something that we haven't thought about, we are absolutely happy to consider it and see whether that would work better. We are not plugged in to any particular way of doing anything. Nothing is set in concrete. So, please keep that in mind but I don't think that this is really the place for a discussion about methodology right now. So, with that, Ruth, if you are not going to talk about methodology.

MS. DAY: A very brief and important thing, there is a small point that may cause a lot of confusion, and that is, we have been using criterion numbers and I wanted to point out that the criterion numbers you may have in your notes from yesterday are from Dr. Svarstad's report where, for example, number nine is scientific accuracy. In all the criteria numbers that have been on the screen today, they have been based on the handout that we got here where scientific accuracy, for example, is number one. So, if you have in your notes criterion number something or other, make sure you keep track of whether it was the first day from Dr. Svarstad's report or from today from the handout so that you know but, also, if you submit something to the docket don't use the criterion numbers or at least identify that they are from the report. Use their names rather than the numbers.

DR. OSTROVE: That is a very good point, and I really appreciate that. Excellent. Thank you.

Well, I think we are at a good point, at this point, to break. Again, I want to thank all the people who have worked so hard in putting this together, and especially Ellen Tabak and Marcia Trenter, and the Office of Training and Communications, all the people from the Office of Training and Communications and from the Division of Drug Marketing, Advertising and Communications, and all of the rest of you who came -- Dr. Svarstad, who flew all the way in from Wisconsin. Thank you, all, and submit your comments, please. Thank you very much for coming.

[Whereupon, at 3:00 p.m. the proceedings were adjourned.]

   Back to Patient Labeling