UNITED STATES OF AMERICA

P-R-O-C-E-E-D-I-N-G-S

(8:41 a.m.)

CHAIRPERSON BOEHLERT: Good morning, everyone. We'll start by going around the room and introducing ourselves. David Horowitz, could you start by introducing yourself and your affiliation?

MR. HOROWITZ: I'm David Horowitz. I'm the Director of CDER's Office of Compliance.

MS. WINKLE: Helen Winkle, Director of Office of Pharmaceutical Science, CDER.

DR. HUSSAIN: Ajaz Hussain, Deputy Director, Office of Pharmaceutical Science, CDER.

DR. CLAYCAMP: Gregg Claycamp. I'm Director of Scientific Support Staff at CVM.

DR. GOLD: I'm Dan Gold. I'm not director of any agency. I'm with D.H. Gold Associates.

DR. PECK: Garnet Peck, Purdue University.

MS. SCHAREN: Hilda Scharen. I'm the Executive Secretary of the Advisory Committee for Pharmaceutical Science, FDA.

CHAIRPERSON BOEHLERT: Judy Boehlert, Boehlert Associates, LLC.

DR. MORRIS: Ken Morris, Purdue University.

DR. DeLUCA: Pat DeLuca, University of Kentucky.

DR. RAJU: G.K. Raju, MIT Pharmaceutical Manufacturing, NSU.

MR. PHILLIPS: Joe Phillips, International Regulatory Affairs Advisor, International Society for Pharmaceutical Engineering.

DR. SINGPURWALLA: Nozer Singpurwalla, George Washington University.

MR. MIGLIACCIO: Gerry Migliaccio, Pfizer, representing innovator companies.

DR. FACKLER: Paul Fackler, Teva Pharmaceuticals, representing the generic industry.

CHAIRPERSON BOEHLERT: And Joe, do you want to?

MR. FAMULARE: Joe Famulare, Director, Division of Manufacturing and Product Quality, CDER Office of Compliance.

CHAIRPERSON BOEHLERT: Okay. Thank you, everyone, and once again, welcome to today's session.

Hilda Scharen will now read the conflict of interest statement.

MS. SCHAREN: Good morning. The following announcement addresses the issue of conflict of interest with respect to this meeting and is made a part of the record to preclude even the appearance of such at this meeting.

Based on the agenda, it has been determined that the topics of today's meeting are issues of broad applicability and there are no products being approved at this meeting. Unlike issues before a committee in which a particular product is discussed, issues of broader applicability involve many industrial sponsors and academic institutions.

All special government employees have been screened for their financial interests as they may apply to the general topics at hand. To determine if any conflict of interest existed, the agency has reviewed the agenda and all relevant financial interests reported by the meeting participants.

The Food and Drug Administration has granted general matters waivers to the special government employees participating in this meeting who require a waiver under Title 18, United States Code, Section 208.

A copy of the waiver statements may be obtained by submitting a written request to the agency's Freedom of Information Office, Room 12A-30 of the Parklawn Building.

Because general topics impact so many entities, it is not prudent to recite all potential conflicts of interest as they apply to each member and consultant and guest speaker. FDA acknowledges that there may be potential conflicts of interest, but because of the general nature of the discussion before the committee, these potential conflicts are mitigated.

With respect to FDA's invited industry representative, we would like to disclose that Gerald Migliaccio is participating in this meeting as an industry representative acting on behalf of regulated industry. Mr. Migliaccio is employed by Pfizer.

Dr. Paul Fackler is participating in this meeting as an acting industry representative. Dr. Fackler is employed by Teva Pharmaceuticals.

In the event that the discussion involves any other products or firms not already on the agenda for which FDA participants have a financial interest, the participant's involvement and their exclusion will be noted for the record.

With respect to all other participants, we ask in the interest of fairness that they address any current or previous financial involvement with any firm whose product they may wish to comment upon.

Thank you.

CHAIRPERSON BOEHLERT: Thank you, Hilda.

We will be addressing two topics this morning, the pharmaceutical industry practices research study and pilot model for prioritizing selection of manufacturing sites for GMP inspections. And David Horowitz is going to introduce us to these topics.

MR. HOROWITZ: Okay. We're going to start off with the studies that --

CHAIRPERSON BOEHLERT: Can you turn on your mic.

MR. HOROWITZ: Okay. We're going to start off with the two studies that Jeffrey Macher and Jackson Nickerson will be presenting.

Jeffrey Macher is a professor at Georgetown's Business School, and Jackson Nickerson is a professor at Washington University in St. Louis's Business School. They both have M.B.A.s and doctorates in business. In addition to that, I believe Jackson Nickerson has a Master's degree in mechanical engineering, which is also an interesting complement.

They have both done extensive work prior to focusing on the pharmaceutical industry on the semiconductor industry and produced a very highly regarded and participated in a very highly regarded and successful study of that industry that has been very helpful to that industry.

And they're going to be using some of the same techniques and approaches in examining the pharmaceutical industry, but also the regulatory side of this industry that's somewhat unique from the semiconductor and many other industries.

They's be discussing two closely related studies of pharmaceutical manufacturing and regulation, the first of which focuses on FDA's regulatory oversight of drug manufacturing, and they'll analyze various FDA databases using econometric techniques to identify factors that are predictive of FDA oversight and regulatory outcomes.

Now, this is of interest to us as well sa it is to industry presumably. We hope that the study will facilitate our ongoing efforts as part of the GMP initiative to enhance our regulatory oversight, including aspects of coordination and consistency which we are trying to address in the GMP initiative and aspects of increasing the risk based focus of our programs.

The second study will investigate the relationship between FDA's regulatory oversight and the resulting production and regulatory performance of the drug manufacturers, and it will also look at the effective of pharmaceutical manufacturers' organizational variables on production as well as regulatory performance.

This, of course, is of great interest to the pharmaceutical industry. It's also of interest to us for a wide variety of reasons, one of which is that we hope to be able to incorporate some of the learning and some of these results into the model that we'll be discussing, which is a work in progress trying to help us prioritize manufacturing sites for GMP inspections.

The connection between the two you can probably see, is that factors associated with strong regulatory performance and production may support reduced frequency or scope of inspectional oversight.

We also hope generally to gain more insight into how FDA policies and actions affect industry performance and behavior to better tailor and adjust our actions to achieve the desired results.

After Professors Macher and Nickerson speak, there will be four speakers, including myself who will discuss the use of a technique known as risk ranking and filtering, as we are attempting to apply it to FDA's efforts to prioritize manufacturing sites for GMP inspections.

Starting will be Gregg Claycamp, who will discuss risk ranking and filtering as a risk management tool and putting it in the context and comparing it with certain other types of risk management tools.

I'll follow by providing some context and a little bit of an introduction to this first iteration of our site selection model.

After that I'll be followed by Nga Tran and Brian Hasselbalch who will discuss in more detail how FDA went about designing the model, including many of the data limitations and hurdles that we face in seeking comment and assistance, and also discussing a technique that we have begun using called expert elicitation. But it's only the beginning, and one of the reasons we're here is because we want more input on that model and we hope in the future to expand it publicly.

So with that I'll ask Professor and Macher to begin.

Thank you.

DR. NICKERSON: Madam Chairperson, committee members, attendees, good morning. David did such a great job I don't think I need to stand up and give you any presentation. He gave you a very good summary of what we're doing.

Yesterday you heard quite a few words around science, pharmaceutical manufacturing knowledge, and we also heard some words about management, organization, incentives. That last category, that latter category of words falls squarely in the domain of management and business.

It's interesting to look around the committee because a necessary condition to make all of the changes that have been described both on the FDA side, as well as the industry side, is this notion of management and change. Yet I don't notice anyone from a business school on the committee. So hopefully the approach that we're taking might be new and different and useful, and we think the FDA is going to find it useful.

We'll also tell you about the manufacturing study and the manufacturers believe it's useful because a large number of them have participated.

So what I'm going to do is tell you about two studies. One we call the FDA research project, and some of you have heard of these projects before. Some have not. So we're going to walk a fine line between giving you some introduction and hopefully a little depth, but not too much depth.

Jeff will stand up and talk about the pharmaceutical manufacturing research project. Let me just give you a little history about the FDA project, as we call it.

It began when Jeff and I had a phone call back in the fall of 2001. We had read some recent press reports that there was an increase in the number of FDA actions against manufacturers, and this was interesting to us because we had both participated in a Sloan semiconductor foundation grant where we studied the semiconductor industry, looking at best manufacturing practices, and we thought that sort of methodology might be useful in the pharmaceutical industry. So over the next year and a half we pursued this topic with the FDA and with manufacturers and ultimately got the project off the ground.

Let me tell you what the goals of the project are. There are three. We believe we can develop a risk based assessment of GMP outcomes, that is, trying to understand why and when we see various outcomes.

In order to do this, we have to identify those attributes that are correlated with those inspection outcomes, and I'll tell you a little bit about how we're going about doing those correlations.

And finally, what we learn we hope to transfer to the FDA. So this is both in terms of our analysis, some data, and analyzing that data, but also the methodology or framework that could be used as we move forward in time.

So let me tell you about the approach. We spent a lot of time interacting with various people in the FDA in order to identify what data sets already exist in the FDA. We weren't going to create a new data. We're going to leverage off existing databases.

We're going to look at and estimate the likelihood of various types of outcome. You're all familiar with the inspectional outcomes and some other outcomes I'll talk about in a few minutes.

Well, in order to estimate the likelihood of these outcomes, we have to look at a number of factors, and I'll review all of those factors. The factors are about the product of the compound, the plant, the firm, but also factors about the FDA and the investigators and the amount of resources allocated and the likelihood of an inspection being chosen.

And out of this, we believe we can allocate or investigate the allocation of resource and perhaps develop a model to provide some estimate of what the risk is for either delaying inspection or accelerating inspection. In other words, how do we optimally allocate the FDA's resources?

And finally, we think we can provide some feedback to the FDA about how they manage and train their investigator work force and also some information about the different districts and hopefully some of that will come through as I talk about the data and the analysis.

Well, we found a number of databases in the FDA. Unfortunately they all don't talk to each other. so part of the big task is once we get all of this data, we have to combine it and, in essence, clean it so that we can match it up.

There's a database called COMIS, which deals with supplement filings, DQRS which deals with field alerts. There's some outsourcing information in something called EES. FACTS is the database that is largely in the ORA and deals with inspections. Product listing, product recalls, product shortages, those are fairly straightforward. Registration, which is an annual database. Warning letters, and we helped construct a training database so that we know at what point in time the level of the training, the type of course that the different investigators had before they went out on inspection.

Now, this is all collected, and we're trying to integrate all of the data in order to develop these statistical models.

What are the important outcomes? Well, I already mentioned the inspection outcomes. On action, voluntary action or ordered action indicated. Those are the standard outcomes from each investigation. Beyond that, you might get a warning letter, but there are also other outcomes, perhaps more real outcomes in terms of field reports, product recalls, and product availability. So we're going to use those outcomes in our analysis.

Some of the key factors that we're looking at that's already collected by the FDA include what type of compound is it. Is it an NDA or ANDA? Is it prescription versus nonprescription? Some information about the product class, product subclass, process indicator code. Those are somewhat rough measures, but measures nonetheless.

We have supplement history, the extent of vertical integration. At least for certain aspects do you produce the API and formulate? Is your testing outsourced or done internally?

We can also assemble the history of regulatory outcomes for the product at least to 1990. It's very difficult to go back before 1990. There was a major computer system change, and it would be rather difficult to integrate data before 1990.

And then, of course, we can look at the history of regulatory actions not only for the product, but also for the plant and also for the firm to see how that affects the likelihood of inspection or the likelihood of various outcomes.

Other factors. Well, in terms of the facility, we'd like to know how old it is, its size, what's produced there, the number and the variety of products. That may impact the quality manufacturing, if you will, or it may impact the likelihood of the FDA choosing to inspect. Hopefully we can tease apart those different motivations.

We can look at the change over time in terms of the number of products or the diversity in products. Importantly, we can look at ownership changes. That is recorded in the database, and when you have an ownership change often systems change, and the question is: is that for the better, for the worse? What are the issues? Does it encourage the FDA to inspect? We don't know, but we'll be able to figure that out from the data.

And, of course, this regulatory history that I mentioned.

Firm level variables. Again, age and size of the firm. There are a number of manufacturing locations. What's the breadth of product that they produce, both in terms of number and variety? We can look at things like number of pass introductions because that may affect the amount of human resources that are allocated to fixing deviations versus introducing their products.

We can look at the number of past regulatory decisions. So, for instance, we have heard some stories that if one plant gets a negative review, then other plants might get reviewed shortly thereafter, and we can identify if there are these spillovers or reputation effects that manifest either within a firm or for a particular compound.

If a particular type of compound, let's just say aspirin, if something was found amiss at a plant, then maybe all other aspirin plants are inspected right away, and we can identify these sort of behavioral reactions.

Now, so far I have focused on manufacturer variables, but of course, FDA variables matter also. So we can identify FDA district, not just domestically but internationally. We have some estimates on the inspections, the amount of time allocated, the amount of manpower allocated to these inspections. We have the number of investigators, the reason for inspection, who's on the team, and the time since the last inspection.

In terms of the investigators, we can look at some very key issues that the FDA has already moved to try to correct, and you heard that yesterday, which is, say, in New England one day an FDA inspector might be out looking at a blueberry packing facility, a fish packing facility the next day, and the third day they're at a biotech firm. How does that accumulation of experience matter and translate into the outcomes that we see? We can evaluate that.

Also, there are different stages of training for these investigators, and we've collected information on who has received what training by when, and we can ask questions about how that impacts either the likelihood of a facility being investigated or the likelihood of a given outcome.

And I'm using likelihood and probability interchangeably from our talk, even though they may not be exactly the same.

To preempt that, we do teach Bayesian economics in the business school.

DR. SINGPURWALLA: But maybe you're doing it wrong.

(Laughter.)

DR. NICKERSON: Also, we can assess various policy shifts like the SUPACs when they were introduced and how that impacted not only when firms were inspected, but also the outcomes of those inspections.

So once we have all of this data and it's all integrated together, what are we going to do with it?

Well, we want to undertake a statistical analysis to estimate the probability of the various outcomes that we've described. Now, it's a particularly difficult issue because you can't use sort of standard statistical tools, the big workhorses, something called "ordinary re-squares" (phonetic).

It turns out the FDA chooses to inspect for particular reasons and manufacturers may choose to place certain compounds in particular plants for certain reasons, and so we have to account for those choices, which makes the analysis a little bit more difficult, although there are a number of good techniques to account for these difficulties.

Once we estimate the model we can use it to ask kind of factual questions, "what if" questions. What is the risk of delaying inspection on this particular compound or this particular facility or this particular plant.

We can ask questions "what if we insure that all investigators had the full complement of training before they went into the facility" and ask a wide variety of "what if" questions that we believe can help tease out the risk of either accelerating or providing some backing off of regulatory scrutiny.

It should also provide some insight in terms of what sort of things should be monitored as we move forward, what matters, what are the critical variables and parameters.

So ultimately we think this analysis will improve our understanding, FDA's understanding, and industry's understanding of inspection outcomes and how they relate to the various attributes that we can measure.

This risk assessment will be used to inform FDA oversight choices. Now, this is retrospective data,but again, the framework is something that can be used also moving forward, and fundamentally it tells us something about particular processes, particular plans, particular manufacturers, as well as tells us something about particular district offices and possibly particular investigators, although we don't have investigator names that are all hidden from us with some sort of ID code so that we can't do that matching.

Well, what's the status? We've been working on this for a while now. We completed what we called a pilot study, which involved interviewing lots of people in the FDA and, as Jeff will tell you in a few minutes, a lot of people in industry.

We wanted to identify from both sides of the coin what was important, what was problematic, what the good stories were, what the negative stories were in order to shape our analysis.

Phase 2 is collecting data. I'm happy to report that all the data for CDER at least, all of those data sets, have been assembled, compiled or sitting in CD-ROMs on a desk somewhere. We're waiting for them to be released to us, and we anticipate that will happen this month.

Once we have it released to us and we're still working with CBER, they have a different set of data sets, and they integrate a little differently. So we're still working there.

Once the data is in our hands, it will probably take a while to go through and, as I say, clean the data, typos, data entry mismatches, and resolve as I understand it there are some 13,000 observations, 13,000 plant visits over this time, maybe even more.

In any event, it will take some time to clean that data, and then there are actually a variety of statistical techniques that we're going to be using depending on what the particular question is. So that might take anywhere from three to six months once we have the data in our hands.

That's the FDA project, and what I'd like to do is turn the lectern over to my colleague, Jeff Macher, who is at Georgetown University, and he'll review what we call the pharmaceutical manufacturing research project.

Thank you.

DR. MACHER: Okay. Thanks, everybody.

This is pretty much the same presentation, just on the manufacturing side versus the FDA side now.

This research project emerged at the same time when we were discussing the increase in severity and number of CJ&P violations, but we are asking another question. We wondered, based upon what we learned in a study, a Sloan funded study in the semiconductor industry, specifically on semiconductor manufacturing, whether these violations were related to managerial, organizational, and technical practices that we found to be the case in the semiconductor industry.

We learned a lot from the semiconductor industry, and the benefits that we gave to firms in reshaping their managerial organizational and technical practices were demonstrable. Most firms improved significantly their manufacturing performance, and we wondered if we could do the same thing here based upon a large scale analysis of the number of pharmaceutical manufacturers that we could get convinced to participate.

So we began interviewing manufacturers in the spring of 2002 and we literally traveled around the U.S. and to Europe interviewing dozens of manufacturers. We tried to be as broad as we could. We interviewed many pharmaceutical manufacturers, biologics, APIs, contracts and generics. Generics aren't listed there.

Really there was two reasons to do that. One, so that we could come up to speed on this industry. There are some nuances that we didn't really understand and admittedly we're still coming up to speed with it.

And then, secondly, we wanted to ask questions that were important to the participating firms. So there was a good deal of dialogue and give-and-take in developing a questionnaire that most firms found to be pretty effective.

We went live with an Internet based questionnaire in the fall of 2003, in November, and since then I have principally been engaged in marketing and soliciting participation.

We expect to close the first round of the survey shortly, and shortly should be in quotes. We don't know when that will be, but shortly.

The goals, very similar to the goals that we had in the semiconductor manufacturing industry. We wanted to develop a standard set of benchmarks for measuring, manufacturing, and regulatory performance, and this in itself is an heroic endeavor. We want to identify the managerial, the organizational, and technical practices that underlie good and poor manufacturing and regulatory performance and then provide a confidential score card -- and this is one of the reasons why we think it would be beneficial to the firms that participate -- to specific manufacturing facilities on how they perform against anonymous others so that we can compare API manufacturers to API manufacturers. We'll identify who you are against a set of anonymous others, against a set of peer groups, and I think that's beneficial in and of itself.

Our approach, as I mentioned, we developed this focus questionnaire of potential factors that we thought and based upon input from industry influenced manufacturing and regulatory performance. We administered over a secure Web site via the Internet. We assign a unique user name and password to each participating manufacturing facility. That user name and password is used by the individuals within each facility to fill out the data. It's completely secure.

We then collect the data. One of the nice things about this is it dumps the data that's collected on the Internet into a relational database. We can then analyze the data using a variety of econometric techniques very similar to what Jackson had already presented to you, and then provide a summary of our findings.

We'll write a couple of white papers, make industry presentations such as this to industry overall, and as well FDA and industry meetings.

The database. We've secured participation from a cross-section of U.S. and European manufacturers. We've stayed strictly to U.S. and European manufacturers. Right now 21 firms and 60 manufacturing facilities that have either finished the completion of the survey or are actively completing the survey, and it's my job to sort of push these people through.

One of the difficulties obviously is pharmaceutical manufacturing is crazy enough. We're coming into these facilities and asking them, "Oh, by the way, can you do a little more work?"

It has been trying but usually successful to get these people to commit to it. It's just a process that takes some time.

The survey is, as I mentioned, on line, and each manufacturing facility provides detailed data on between one and five compounds. We ask for all of the compounds that are manufactured within the facility, but then we ask each firm to choose or each facility to choose the top five, where the top five is defined somewhat loose. It can either be in terms of volume or it can be in terms of the importance of those compounds to the facility, where importance could be defined in different dimensions.

What we're really asking is what are those top five compounds that you would change your manufacturing, your technical and organizational practices if we presented data that showed how you can improve? Okay?

The performance outcomes, instead of the semiconductor industry where we just looked at manufacturing performance, now we're looking at both manufacturing and regulatory performance. In terms of manufacturing performance, theoretical and actual yields, batches started and failed, and then a cycle time measure.

Regulatory performance, failed alerts and biologic deviation reports, and then warning letters, consent decrees, deviations and supplements. Where we think we're going to make one of the biggest impacts is in deviation and supplement management.

The related key factors that we're asking for in the survey, it's nine sections. Actually it's 11 sections, but we sneak two extra sections in by calling them A and B. The company -- that's a joke, by the way.

(Laughter.)

DR. MACHER: The company in the Strategic Business Unit, we asked for just some simple financial information as well as some demographic information, things like facility size, facility age, facility location, things of that nature.

We ask for some brief financial information on each facility that's participating if they have it, revenues, employee sales, R&D expenses, property, plant, and equipment, some demographic information, number of employees, age, size, location. I mentioned a few of these already.

Product information, the number of products or compounds manufactured and their type, and then regulatory inspection information outside of FDA. So Brazil, EMEA, Japan, things of that nature.

And then questions on the extent of outsourcing within the manufacturing facilities, development, process development outsourced. Is any part of manufacturing outsourced? Are APIs done internal to the manufacturing facility, internal to the firm, or external?

Product and process development. We do pretty big sections here. Most of my research investigates new process development. It's one of the things that I've gotten into when I was studying semiconductor manufacturing.

We look at information on where was product and process development done in terms of its location relative to the manufacturing facility. How was it organized? Were engineers from the pilot plant collocated with the manufacturing facility?

This is really a learning before versus learning by doing approach.

And then the timing. How long did it take? How long did process development take for the specific compound versus other compounds in this facility, versus other firms, speed and new process development?

Human resource management, another thing that's been one of the things that we learned from the semiconductor industry, was the importance of incentives related to human resource management. So we're looking at things like employee appraisal, employee promotion, the mobility and demographics of employees. How much are they trained? What types of training?

So we're asking for data on things as diverse as SPC controls, all the way up to a variety of different dimensions.

The extent and use of teams within the manufacturing facility. So we're gathering data on whether they employed quality function deployment teams, cycle time reduction teams. What's the team make-up and composition? Is it just engineers or are there the lowest level operators involved with technicians, involved with engineers?

Deviation and supplement management. We look at whether the firm employs an information technology system to track deviations and supplements. The extent of process analytic technology, that we've taken information, taken, borrowed, used information from Ajaz in a section of the survey to look at deviation and supplement management.

And then finally, how is it organized? Who has responsibility for a deviation correction once it has been in place? How many people have authority or need a check-off on that? A variety of questions we ask in deviation and supplement management.

Where are we right now? As I mentioned, Phase 1 was an exploratory pilot study which was completed in the summer of 2003, which led to the development of an Internet based questionnaire.

Phase 2, we're nearing the end of it, is data collection. We've been fairly successful with convincing firms to participate, and a multitude of firms within manufacturing or a multitude of manufacturing facilities within a given firm.

We'll conclude the first round shortly, but we will most likely continue to market the survey to other pharmaceutical manufacturers, and then similar to the FDA study, we're going to need some time to go over the data.

So we imagine the analysis will require three to six month of work where we'll do similar, again, to the FDA study some statistical and econometric analysis and begin writing final reports.

What's not included is, depending on our money, Jackson and I have not taken any money from FDA or industry. So we are funded through grants from our respective universities and then economic think tanks.

Depending on the amount of money that we have left, we'll either visit a number of the participating firms to make sure that the data that they've entered and the results that we show are sensible, or we'll hold conferences either at our respective universities or at a location to be determined. I'm thinking Hawaii, but that's just me.

(Laughter.)

DR. MACHER: I guess that's it, and I think now we have questions, unless you want to end.

CHAIRPERSON BOEHLERT: No, we do. Thank you, Jeff and Jackson.

We have time for questions for either of those speakers. Yes, Gerry.

MR. MIGLIACCIO: Jeff, the regulatory performance, when you're talking field alerts and deviations, are you looking at or are you looking at the resolution process?

DR. MACHER: Both. We're looking at it, for instance, let's say for deviation management, we're looking at the number of deviations within three separate areas: raw materials, process, and equipment. So we're looking at number. We're looking at time to deviation correct, and then we're looking at a separate number, whether it's a repeat deviation.

MR. MIGLIACCIO: All right. My concern about deviations is deviations can be cultural. Some of our facilities write very detailed SOPs. So any deviation from that is a deviation that's reported, although at another site with a much more general write-up, perfectly acceptable write-up SOP, it wouldn't be a deviation. So it's cultural.

So we have to normalize for those cultural differences. The same thing with field alerts. Many field alerts for an OOS will be closed out as not having been an issue after it's fully investigated. So using numbers, I'm a little concerned about just using numbers.

DR. NICKERSON: A couple of comments. First of all, deviations is the trickiest part of the whole survey just because of this. There are different parameters in the manufacturing processes that will identify something as a deviation or not.

The way we deal with this, there are a couple of things. One, we look for whether it's recurring deviation by your own definition or a new deviation.

Second is when we do our analysis across all of the firms or all of the facilities, we use something called fixed effects, and the idea is to, in essence, take out the intercept, if you will. That takes out the -- it adjusts for the different width of these SOPs. What we look for is the rate of change. Do we see a decline over time in all of these parameters? And that's the key thing we're looking for in deviations.

I'd also point out that in terms of regulatory performance, we also look at supplements, and we're collecting information on how costly it is to firms to assemble the information, file the supplements, and what is the success in filing those supplements in terms of timing, but also approval rates.

So that's another dimension of regulatory performance that Jeff hadn't mentioned.

CHAIRPERSON BOEHLERT: Ken and then G.K.

DR. MORRIS: Actually two things. One is that there actually is a business school person. Granted it's not much of a business school. It's Sloan, but you know.

DR. NICKERSON: Who's that?

DR. MORRIS: G.K., yeah.

DR. NICKERSON: You teach in the business school? Okay. Well, I didn't see that on your Web site.

(Laughter.)

DR. MORRIS: But you're right. It' snot much of a business school.

DR. NICKERSON: Yeah, right.

(Laughter.)

DR. MORRIS: I just want to make that clear, but the other question is when you did the API, when you included the API sites in the evaluation, were these API sites that were always associated with the innovator company or were these independent API production sites that service more than just one customer?

DR. MACHER: These would be independent API sites. Now, within the innovators, they would also have some API compounds, obviously.

DR. MORRIS: Right. No, I guess that's my question. Did you both --

DR. NICKERSON: Yes, the answer is both. So some of the firms have API collocated with formulation. Some have API distinct, separate, separately located from formulation, and then there are API firms that are separate, and so we have all of those in our sample right now.

DR. MORRIS: And do you distinguish between them in your analysis?

DR. MACHER: Yeah, The analysis would then compare API manufacturers, distinct API manufacturers to API manufacturers, biologic manufacturers to biologic manufacturers, and then we could even further granulate on the chemical firms.

We could break up the granularity of the analysis into finer increments, and it's important to know that it's not just identifying those types, but the management processes within those firms that will be able to identify how they differ also or if they're the same.

CHAIRPERSON BOEHLERT: G.K. and then Dan.

DR. RAJU: I had two questions for either of you. One is general and the other is more specific. So I'll ask the general one first.

The history as we got here was that you had experience in the semiconductor industry and you were going to look at the pharmaceuticals, and you've reached a point where you've collected the data and you've begun or you will begin to do analysis and you will have results shortly.

Yes, we will have some results from it, but you've learned something in all of your discussions at the sites and the FDA. What was the surprise? What did you learn qualitatively in terms of your experience at semiconductors, which is what you've done so far over the last year or two?

What was the surprise?

DR. NICKERSON: I think what we've learned is that the two projects should add a lot of value. That's what we've learned, and I don't think there's one --

(Laughter.)

DR. NICKERSON: Bayesian analysis is important, but this --

DR. RAJU: I actually thought your project fits nicely into the Bayesian framework. I really thought so. I'm not sure if Jeff does, but --

DR. NICKERSON: In fact, there are many different techniques for analysis, and we're fortunate at Wash. U. to have one of the world's experts in Bayesian econometrics, Sid Chip (phonetic).

DR. SINGPURWALLA: Yeah, I know him.

DR. NICKERSON: So Sid --

DR. SINGPURWALLA: (Speaking from an unmicked location.)

(Laughter.)

DR. NICKERSON: I'll tell Sid he's rather flat and see what he says about that.

But so there are a number of different techniques we're going to be using in order to analyze the data. It depends on what the particular question is.

DR. RAJU: Sure, okay. And then I had a second question that's more specific around a couple of things you had here. You asked in the survey for people who talked about between one to five compounds, and you said that was somewhat flexible. Is that a good idea for somebody like that to be flexible if that's the basis for you to discriminate and evaluate performance?

DR. MACHER: I actually don't know if I said flexible. If I did say flexible, I was in error. Okay? So here's the idea.

We're giving a survey, an Internet-based survey which is going to take anywhere from two to three weeks to each manufacturing facility. These manufacturing firms are taxed in terms of what they can provide us. So we want to make it as easy as we can for them.

For instance, we learned about the generics yesterday. They manufacture hundreds of compounds. So, in fact, do contract manufacturers. We can't ask them to input information on 100 different compounds. So we have to be specific in asking them to do their top five, the five that they deem the most important in the facility.

Almost every facility has given us five compounds per facility. Now, there are some facilities that don't operate. They're single compound focused, but that tends to be the minority.

In terms of flexibility, we're asking them to give us those top five that they deem most important in terms of whether our results would change the way they go about doing business, whether that would change the way that they manage organize and implement technology.

So I don't know if I've answered your question.

DR. NICKERSON: So let me add on to this. There's a sample selection issue, and that's your question.

DR. RAJU: Yeah. It's not really that you're asking for five, but I haven't heard how you're asking them to decide on those.

DR. NICKERSON: So let me tell you what the parameters are. We have a set of parameters we asked them. We're looking for compounds that are at least two years old, but were introduced in less than ten years.

We asked them for those compounds that are materially significant to them, where that material significance could be volume or revenue.

We also have a number of characteristics about the processes in terms of when they're introduced, how much total cumulative production has occurred so that in our analyses we can fully characterize the sample selection that's involved.

So we do have these rather strict guidelines. With just about every facility we've had a discussion. So we're pretty comfortable in knowing what they've selected versus what they haven't selected. So we have a pretty good idea of the full scope.

Obviously a compound that's been out there for 20 years, you're not going to see a lot of improvement in any of the production performance measures that we're looking for, and we're not going to look at those compounds.

It makes no sense to look at a compound that just came out last year because we don't have enough accumulated history. So that's the sample selection that we've decided on, and we do know what the parameters are pretty well.

DR. GOLD: Is it my turn now?

CHAIRPERSON BOEHLERT: Dan, it's your turn.

DR. GOLD: Thank you very much.

I have two questions. Number one --

DR. NICKERSON: Do you teach at a business school, too?

DR. GOLD: No.

DR. NICKERSON: Okay, sure.

DR. GOLD: No. In fact, I've never even gone to a business school. Is that beneficial for me?

(Laughter.)

DR. NICKERSON: I don't know. We have some programs that I could interest you in perhaps.

(Laughter.)

DR. GOLD: Deviations are looked at differently by different companies. Now, you talked about deviations as a general category. Have you defined deviations for these various companies in a way that enables you to say, "Yes, I am going to be able to judge or look at the deviations at A, B, C, D and E companies in a meaningful way so that I can really understand how they're handling the same deviations differently"?

DR. NICKERSON: An excellent question. As I mentioned before, deviations is the toughest part of this, in part because as you mentioned and as Gerry mentioned firms and even plans within firms will define deviations differently.

DR. GOLD: Of course.

DR. NICKERSON: Right. So what we've done is we've provided standard deviation -- standard definitions on different classes of deviations to all of the participants, and we've asked them to define their deviations in accordance with our definitions.

That said, we still expect there to be plant specific differences in these measures. So the best we can do from the statistical perspective is to put in what I call a fixed effect. That is, identify that there's a different plant and that, in fact, they may have different definitions or different thresholds, but then look at the rates of change over time of the different classes of deviations and the amount of resources allocated to how you respond to those deviations and compare that to the way they're organized to manage the deviations.

As you probably know, in some facilities the group that identifies the deviation manages its resolution. In other plants, there's a cross-functional team.

DR. GOLD: Yes.

DR. NICKERSON: In other plants still it gets shoved over to one group who is supposed to deal with it.

So we believe that we can analyze the different ways in which the firm is organized to handle deviation and assess the rates of change of the different parameters we're measuring.

DR. GOLD: Yes. There are some firms that include major deviations as well as minor deviations as part of their deviations list. Are you segregating these into just the major deviations?

DR. NICKERSON: Largely to the major deviations, yes.

DR. GOLD: Yes, okay. A second item. Another apsect, very significant aspect of management, facility management, is change control. Now, you have not mentioned at all the issue of change control and the monitoring of change control techniques and application of change control and the drive that change control may have on supplements, on validation and revalidation and so on.

Are you neglecting that entirely?

DR. NICKERSON: Excellent question. The answer is, no, we're not neglecting it entirely. In the survey, it is hard to give you the full survey because it's so large. In the survey, we pay attention to where certain decisions are made in the organization. So we know if decisions are made at the low level, two levels up, three levels up.

And we also look at where conflicts are resolved when there are conflicts between and among different entities within the manufacturing facility, and those questions we believe get at basically the issue you're describing.

CHAIRPERSON BOEHLERT: Garnet.

DR. GOLD: Yes, all right. The final question I have is related to, if I may, API facilities. It is reported -- I don't know whether this is actually the case -- but is reported that approximately 80 percent of the APIs that are used in the U.S. for dosage forms originate from overseas, and a lot of them are from independent API producers.

What percentage of the API facilities that you've included in your study are independent API producers and from what range of countries are you going to be obtaining the data from?

Can you just give us an idea?

DR. NICKERSON: Sure, I can give you an idea. We have maybe three or four independents in Europe, and we have another four from the United States. Those are independent API producers.

DR. GOLD: None from Asia?

DR. NICKERSON: In our study we have only focused on Europe and the United States, in part, because in order to get the study going, we felt it was important not to take any money from either the FDA or from industry. The net result is we applied to a number of academic centers at Georgetown and Washington University.

Well, fortunately we were able to get some money, but not enough to include either India or China in our study. If we had a larger budget, we would more than happily include them in the study, but it was just not economically feasible to do so.

DR. GOLD: But even in Europe there are a very large number of API producers, independent API producers, including four which seem to me to be a rather modest number.

DR. MACHER: Well, the participation is voluntary. We have done our best job of marketing this as best we can, and there are only certain, I guess -- so many ways in which we can go forward.

I guess the other alternative is to do nothing and not do the study at all. And what I'll also add is this is just the first phase. The second phase and subsequent phases will add to the end.

But you know, we can't swallow the cow. We need to sort of take a little bit off as we go.

DR. NICKERSON: The other thing to realize is you asked specifically for independent API manufacturers. We have a much larger number of API manufacturers that are in larger firms in Europe. Some of them also sell out into the market. So, in fact, we may have more apparent API manufacturers in Europe than the four independents.

DR. GOLD: But the ones you're talking about, the larger ones in Europe, are they affiliated with U.S. or multi-national firms?

DR. NICKERSON: Some are and some aren't.

DR. GOLD: Some are and some aren't. One of my major concerns are the ones that are truly independent and not very large and not controlled by multi-nationals.

DR. NICKERSON: If you can give us a few more names to participate, we'll include them.

DR. MACHER: And actually since I am in charge of marketing right now, for any of you pharmaceutical manufacturers that aren't participating, please come see me.

DR. GOLD: Yes. Well, thank you very much.

CHAIRPERSON BOEHLERT: Okay. Garnet, your turn.

DR. PECK: Yes. Within the 21 firms, do you have any sampling of the so-called contract manufacturers, in particular, non-prescription drug manufacturers?

A lot of these are very large volume operations. I just wonder if there is a sample.

DR. MACHER: Yes. Yes, we do, but we're trying to avoid some firms that, for instance, make products like skin lotions that are still under some FDA approval. We're looking for products that have a pharmacokinetic benefit. Things like toothpaste or skin lotion we're avoiding.

We do have contract manufacturers in the sample that do prescription and non-prescription drugs within the U.S. and within Europe.

DR. PECK: It's the solid dosage form that I was specifically --

DR. MACHER: Solid dosage, yes.

DR. PECK: -- questioning.

DR. MACHER: Yes.CHAIRPERSON BOEHLERT: Others? Nozer?

DR. SINGPURWALLA: Well, as you know, I don't teach in a business school, but some of my weaker students have received positions in business schools.

(Laughter.)

DR. SINGPURWALLA: Now, I'm not going to criticize what you have done, but I'm going to make a comment. I think the parallel between semiconductor manufacturing and drug manufacturing is not quite the same because a semiconductor doesn't cause damage to an individual. It may, but most semiconductors are like little light bulbs. You can throw them away.

What I would like to suggest is there are some manufacturing functions which involve great risks, and you may want to look at those. Now, I don't know whether you'll have access to them or not, but the Sandia labs, for example, does manufacture components for nuclear devices. They carry great risks, and they have come up with a system for manufacturing under highly risky conditions for risky components.

You may want to look at that, and there may be a better parallel between drug manufacturing and what they are manufacturing. So what I'm suggesting is you may want to look at manufacturing activities that involve risky elements both in terms of handling the elements and also in terms of the consequences of bad manufacture.

That's just a suggestion, and it's not a criticism.

DR. MACHER: I'm actually going to address your concern. The drug products that pharmaceutical manufacturers make are safe. They are. There's no question, and I think you're misunderstanding what we're doing.

We're looking at the process by which drugs are manufactured, given that there's a level of safety that already exceeds any expectation, all expectations. What we're trying to do is improve the efficiency of the existing manufacturing process. Okay? That's what we're trying to do. We're trying to make it so firms can improve their yields and their cycle time, and so that they can solve problems more quickly.

That's our objective. That's our goal. There are a lot of parallels between semiconductor manufacturing and pharmaceutical manufacturing, and you and I maybe can talk on flying about those. I've been in 30 semiconductor manufacturing facilities and about 15 to 20 pharmaceutical facilities. So I think I have a pretty good idea of the similarities, and they are there. They are there.

The products that they're making, yes, are different. The manufacturing processes, the way you organize, the way you manage, and the technology that's put in place have corollaries.

DR. SINGPURWALLA: I think you're becoming on the defensive, and I'm glad you are because that gives me an opportunity to come back.

(Laughter.)

DR. SINGPURWALLA: All I'm suggesting is look also elsewhere, and I said I'm not criticizing what you have done. All I'm saying is maybe there are other avenues that may give you more insights and more information than what you have been doing.

So maybe you misunderstood my intent.

DR. NICKERSON: That's fine. Thank you.

CHAIRPERSON BOEHLERT: Any other questions or comments from committee members?

Ajaz?

DR. HUSSAIN: I think I didn't clearly understand the coverage or how many generic forms would be part of this because my concern is simply that if we don't have, for example, API manufacturers from Asia and so forth, the survey might not reflect the generic industry, and that's a concern also.

DR. NICKERSON: It certainly is a concern because at this point we don't have any of the Asian manufacturers.

DR. HUSSAIN: But how many generic manufacturers are in the product manufacturers?

DR. NICKERSON: I don't have an exact number for you because there are some firms that are strictly generic manufacturers, but there are others that have a little of both, and so I just don't have that exact number for you. Okay?

Clearly, there will be some sample selection issues. No doubt about it. If we go back to the semiconductor industry, we studied a total of 36 manufacturing plants which if you looked at the number of the firms involved, the firms represented about 80 percent of the industry. The plants didn't but the firms did.

And I don't think we have firms that represent 80 percent of the industry. We still have firms that represent a substantial share of the industry.

So there is this tradeoff in terms of getting all of the little firms, and we're certainly under sampling on the little firms mainly because they're the ones that have the fewest resources to contribute.

To fill a survey, just for people to get a sense of this, it takes two to three person-weeks, which is very costly for the firm, and we're very sensitive to that. We have been ecstatic at the participation we have received so far.

I'd love to have more of the smaller firms, but as long as we understand what the sample selection is, as G.K. was pointing out, then we can interpret the results accordingly.

CHAIRPERSON BOEHLERT: Ken.

DR. MORRIS: Yeah, just a quick comment. Perhaps the way forward is because you're at the stage of getting the Phase 1 results, maybe after that it will facilitate expanding it to cover some of these concerns, but having worked with the same monetary constraints, I know you can't swallow the cow, although certainly we'll try.

So it may be the best way forward is to categorize this the same way we're talking about examples that we need. So if we lump this, if you will, not to do any violence to the study's benefits, but if we lump this in the same category as creating examples, then the first stage may be just to disseminate the results of Phase 1 and then hopefully resolve the issues of recruiting as well, some more funding so that you can do this without having to fly coach.

DR. NICKERSON: That's exactly right. We have been flying coach and staying in coach also. Once we're done hopefully the value -- Howard Johnson's. No -- once the study is done, hopefully it will demonstrate the value that we believe is in the study, and as the manufacturers perceived the value, then perhaps there will be other people signing up, and perhaps once we have demonstrated our ability to maintain confidentiality both with the FDA with respect to the FDA data -- I'll point this way because the industry reps. are over here -- with respect to the industry data, then that will also provide a little more legitimacy, and that may allow us to advance to a second stage.

CHAIRPERSON BOEHLERT: Any other questions or comments from committee members, FDA?

(No response.)

CHAIRPERSON BOEHLERT: If not, thank you, gentlemen.

DR. NICKERSON: Thank you.

(Applause.)

CHAIRPERSON BOEHLERT: We are slightly ahead of schedule, more than slightly ahead of schedule. What I propose is we take our break now for 15 -- well, you don't have to break Nozer.

(Laughter.)

DR. SINGPURWALLA: But then you won't break when I want to.

CHAIRPERSON BOEHLERT: Well, that is a problem. We'll allow you an individual absence.

DR. HUSSAIN: Madam.

CHAIRPERSON BOEHLERT: Yes, Ajaz.

DR. HUSSAIN: We probably are behind.

CHAIRPERSON BOEHLERT: Oh, we're behind?

DR. HUSSAIN: Yes.

CHAIRPERSON BOEHLERT: Oh, we've got one more speaker.

DR. HUSSAIN: Well, the next topic was supposed to have started.

CHAIRPERSON BOEHLERT: Okay. I'm sorry. We're not going to break. Nozer, you're correct. I looked at it rapidly. Yeah, I've been away too much. I'm thinking about vacation on Friday.

But okay. Our next speaker is Gregg Claycamp. Sorry about that.

DR. CLAYCAMP: That's all right.

Good morning, ladies and gentlemen. My father taught in a business school, and actually started at the Sloan School, and I mention that in that -- let me see if I can keep this started -- that risk analysis borrows a lot from many disciplines, including business management, economics and statistics and engineering, et cetera.

And, indeed, my father is a Ph.D. in economics and had gone on to advise corporate boards basically in the business strategic management, risk management area, and even as short as a year ago, we were discussing how do we advise in my case on risk end points and in his case on market penetration and percent share and so forth, and suddenly the light bulbs went off and we realized after all of this time our careers had merged and we do exactly the same thing. We just had a different lexicon.

And so just setting that, I think my role in these talks here is to set a philosophical background for what our team has been working on, and so I just thought I'd start with that little personal observation.

Risk is an intuitive and familiar concept to everyone. If I polled each one of you, you would have your own -- I seem to be on auto pilot here -- if I polled each one of you, you would have an idea of what risk meant to you and what it meant to the organizations you work in, and they might differ. At least on first blush, they might differ from one definition to the next, and they're probably all correct in that we can tease out the elements of risk in everyone's definition, although they may seem a bit different.

And the trick is when you have such a conceptual basis, rather than something that's more concrete and exacting to everyone, it ends up being a difficult challenge for a large and complex organization to settle on one definition of what risk means to them.

And that has been a large part of this process, is getting everybody at the table to say, "Okay. What do we think is risk in these terms?"

Well, risk assessment, which you'll hear about a lot in this process -- my show is on auto pilot here, I think. Okay. It's still flying on its own.

Okay. Burt risk assessment is not a single process, but a -- okay. Borrowing from the National Research Council, risk assessment is not a single process itself, but it's just really a systematic approach to organizing and analyzing scientific knowledge and information, and moreover, this information is directed at supporting a risk decision.

Risk management can be viewed as a systematic process for identification, assessment, control and communications of risks to life property or other things of value, including you may actually want to consider the risk of losing a view if there's construction across a bay from your summer place or something. I mean, anything can be set in that framework, things of value.

As a broad concept, we have as I've stated many possible meetings, depending on the individual or the organization or even parts of the organization. This effort is complex in scope and requires thinking about risk in many different contextual levels, and I believe that we can do that without departure from our overall mission to reduce, manage, and control risk to public health.

So that's where I'm starting from, and now I'll try to paint a little broad brush stroke picture of where these processes are in thinking of hierarchical levels of risk management.

As used here, we'll refer to high level as the broadly based general and principal driven approaches. These are the ones that are more qualitative and are based on the principals that are shared among all fields of risk management.

The low level approaches refer to very specific modeling and discipline driven approaches. You can view this as a hierarchy in processes and systems that high levels can generate a number of different low level approaches and utilize those approaches in an organizational problem of dealing with many types of risks, many types of hazards, et cetera.

Risk ranking and filtering that we'll talk about here is a high level approach or process, if you wish. So, for example, in looking at the pharmaceutical area, in particular, I borrowed this from an FDA report on managing risks for medical product use just showing us that there are known side effects that come out in the pre-market review of the safety and efficacy of the drug or the device.

There's actual medication or device errors that occur once there's practice so that the missed medication errors in hospital settings, for example, and device errors, and there's this area called product defects. The product defects are one area tha this particular effort has been focused on.

There's also these unexpected consequences, and that is so that we can't be all knowing, and essentially it has been called Phase 4. We see things happen when there's larger populations using pharmaceutical products, that they were unanticipated consequences.

Well, the drug quality in one view of this is that drug quality is really focused on those product defects, and the public health side is what we're trying to link up with and improve that linkage in this initiative.

So quality systems, one way to view that is that it's really focused on decreasing the likelihood that you'll experience probability defects and also will decrease the chances that given that some would occur anyway even at a low risk, it reduces the chances that those will ever make it to the patient.

But there's a variety of risk tools that support quality systems directly, and these are, you know, ongoing and lots of discussions between the ICH Q8 and Q9 efforts, and these tools that I've listed here are things such as failure mode and effects analysis, FMEA, and fault tree analysis, hazard analysis and critical control points, probabilistic risk assessment, root cause analysis, and many others and many others that are being invented as we speak that typically are combinations of processes and models that have already been developed. They are just new hybrids and slightly changed from the historical models.

And these tools are very helpful for focusing on assessing and managing risk, given a specific product or product class. It's when you can get down to the low level detail levels that you want tools that can address very specifically these issues.

On the other hand, at a high level, the FDA and organizations, manufacturing organizations, are also faced with dealing with a lot of different issues and yet hopefully bringing them into some prioritization in their work planning for their business or regulatory frame.

So, in other words, you're trying to put on the same table all of the apples and oranges and mix that with the beans and the potatoes and everything else. We deal with a lot of complex issues and a lot of issues that have different health endpoints. They have different hazards and so forth.

So how do we make sense of that at the high level?

And so one way to view this is that you have a series of these on the pharmaceutical side, a series of these models shown in the previous slide and the tools that might be used to do the high level prioritization among many different types of products are things such as hierarchical holographic modeling, which has been written a lot about by Yackov Haimes, a systems engineer. It comes from engineering.

Risk ranking and filtering is also one that he spent a lot of time on and that has a history in aerospace, as well as manufacturing processes.

And risk matrices, and I put the ellipsis at the bottom of that to indicate that there's many high level processes that are being discussed in the risk management side.

Okay. So questions will change and tools will change with the level of analysis. At the low level our risk questions might focus on identifying and characterizing risk to drug quality for a specific product or within perhaps a specific product glass.

And we can hopefully in many cases start to see quantitative measures and quantitative analyses, and these analyses will be driven by those.

At the high level risk questions focus on how things compare with each other. Risk ranking is really you can think of it as a series of decisions to start to prioritize or rank within a given class and then across classes as well. And these are essentially tools that are customized for each application, and so this is a little bit different and relies on committees willing to be creative and put their best thinking forward to borrow from every applicable area they can think of and customize an approach.

And it's really driven by principles more than calculational endpoints. Okay. So just as one low level example, I took a slide that I think many of you have seen before, and I take a fault tree analysis, and that's kind of a favorite of mine because I come from a radiological health engineering background, and this was a favorite of getting licensing for nuclear power, was to do very highly quantitative fault tree analysis, which is starting with we've got a failure at the top.

If we take a light bulb failing and just for a second think about when that light bulb fails what goes through your mind. Well, if love analysis like some of us do, a whole lot of things go off, like, well, there's no electricity. There's a thought, and the glass might be broken. The filament might be broken. There might be a vacuum leak, and so that first gate just below bulb fails is my PowerPoint representation of an or gate. It's either/or on those first four boxes there.

But you can take no electricity on the left side of the slide. You can take that back another step and say, well, you might have no electricity because either the power plant failed or the power line failed or the connector was corroded, et cetera.

And you can take that even farther down another step. The power line fails and wind broke the line or a tree breaks the line. Just an old tree falls on it, et cetera.

Well, this shows how complicated right away a very simple failure can become, and this is quite minimal to probabilistic modeling. It has been used, again, in safety analysis many times, and there's one challenge, and that's that if you take even a simple manufacturing line and try to do this, you'll quickly find that you've got an enormously complicated problem at the first glance. You can break down every piece of equipment into its various faults, and the sources of those faults, and right away you're into a very complicated subject.

And this has been done for things like process chemical manufacturing where there are significant safety issues in terms of, you know, if you mix a couple of chemicals you get a very unwanted reaction from toxic gas release to explosions, et cetera. And so there's very elaborate modeling on the chemical manufacturing side to try to do risk projections for faults in the manufacturing.

Well, some of these low level tools, they have another hazard that we always need to think about in these contexts, and that's the philosophical or communication type side of these. When you develop a highly quantitative risk model which may be built on initial parameter estimates, whether they're flat priors or Jeffrey priors (phonetic) or whatever, they're put together, and they come up with some risk estimate, and they come up with some uncertainty at the end of that.

That itself may communicate to the audience that the audience may hear that you have a lot more precision and knowledge about your model than you actually do. You have to be very careful that on the quantitative side, it starts to look more impressive than the data that may be supporting it.

So we're very cognizant of that, and we work very carefully to avoid looking like we know more quantitatively about a system than we actually do.

Well, that's one possible hazard in a fault tree. The other problem is that you start with that fault, and you may miss the whole picture. You can go down this fault path, and you miss the whole picture, and the example I like to use does come from the radiation field, and that's the Brown's Ferry nuclear accident in the mid-1970s roughly.

It had, of course, in its licensing process, had very elaborate fault trees and used a lot of reliability analysis in its history in building. But what it didn't capture is that a couple of plumbers insulating some duct work would check for a breeze and check that there's penetration of this duct work with the lighted candle, which caught some foam insulation on fire. The fire spread because there was a breeze going through the penetration, and it turned out redundant safety system cabling, and so everything went wrong, and it came very close to meltdown status.

And you know, that wasn't in the fault tree that these would share penetrations and so forth. So we have to be aware that in any type of modeling that we do at the low and high level of all sorts of ramifications of what it's communicating, what it can really tell us, and be very aware of the uncertainty in our modeling itself. What about other models and other views of the world?

So why use high level systems methods in risk management? Well, as I mentioned, low level approaches are, indeed, elegant and capture many details, but they miss interactions and relevance across systems. Complex quantitative models, as I mentioned, may convey a level of precision and understanding about a system that's unjustified. Different levels of understanding and quantification may exist for each subcomponent, but a high level seeks optimal use of diverse kinds of information to inform risk decisions.

So quantitative risk assessment models are only one thing on the risk manager's tables. There's lots of other inputs as we all know going from the values of the stakeholders, the public, the political issues, the legal issues, you name it. It's all on the table, and these are only one of the issues.

High level models really have their source and systems approaches in thinking, and we can have the chicken and the egg discussion on whose field, business, engineering or whoever started this all, but nevertheless, it's all shared at this point and is useful for our work.

The risk management of complex systems is multi-objective. It has got multiple decision makers. It's hierarchical. There's hierarchies and there's lots of overlap, and sometimes there's conflicting objectives and endpoints.

And generally these exceed our human capacity to put everything in a simple model. So to just go over again kind of the broad brush stroke philosophy of where we are with this, we look at using the one I mentioned, hierarchical holographic modeling, which refers to the fact that it's multi-dimensional and it's hierarchical.

And basically this slide and the next couple show that it just starts with an organization of information. Recall I said risk analysis, risk management is a systematic organization of the information, and so that's kind of the common sense issue. What are the things that we think are related to risk and given that we can identify the risk endpoints that are in our interest frame.

And so those may fall within areas of health, compliance, resource, social, political, geopolitical. You can go on and on and just put everything on the page.

So how do you make sense of that in high level approaches? We'll talk about one here, which is risk ranking and filtering, and that's to drill down beyond that highest level and start to flesh out a model with what factors we think may be important in predicting risk.

And those may fall into classes of product and process and whatever that are at a more detailed level than in our initial chart.

There may be a variety of endpoints where we can start to get closer to that low level and maybe eve envision having some quantitative models in form what impact does loss of sterility have on risk, and you know, that's our pipedream thinking for risk analysts is, gee, when can we get to this and get some real quantitative tools going, and that's a ways off in many of our areas right now.

You systematically develop the low level details. So, for example, you could break down into what are the things going on by process that might affect sterility, and actually get into the fault tree analysis and failure modes and effects analysis that are at the low level.

So low level analysis can be quantitative, relying on these other tools, but data gaps may need to be filled with estimates from expert solicitation, and there's a lot of intelligence out there that is accumulated experience of doing this for years, and how can we tap that information because it might not be existing in a database or in a quantitative tool? How can we tap that and use it to inform our risk based decision making, and that's where expert elicitation comes in. It's tapping the mental models that are already in existence.

Sometimes only qualitative information is available for specific processes. So perhaps we might have a qualitative scale such as low, medium, and high, and I just showed one example of severity scale and a probability scale because in many of the high level definitions of risk, risk will be placed in terms of probability of occurrence and the severity of occurrence. And so that's just an example of what that kind of qualitative scoring might look like.

Now, of course, this can mature over the years, and very low could eventually defined as one in a million and low as one in ten to the fifth and whatever. You can think of this as a beginning, and it can improve as more information comes to the problem.

And this just follows up on it that there is some reciprocity that in this concept of combinations of severity and probability, that you may have something that is of high occurrence probability and lower severity, and that may fall in the same range as something that has the inverse, the high severity and lower occurrence probability.

Eventually, risk ranking and filtering will take whatever information that can be identified and looked at as helpful in informing the goal of ranking our risks and pooling those in some form, usually very simple mathematical processes to average and weight can be used, and try to come up with some ranking by combinations of the data that we have and the expert elicitation data, et cetera.

Now, the question is what is the filter. Well, you know, these are not classical, empirically driven models which have random sampling and so forth. We just don't have the information and the ability to set that kind of thing up.

So your best intentions to try to capture models of risk in a given process or given product and so forth, you may come out that everything ranks the same at the end, and so filter is a nice way to say you can go back and say we're going to put a policy on that that can, for one thing, expand the scale and deal with those issues of do we have enough range to be able to rank in the first place, and it can also be that the filter is the policy driven aspect, and that's -- in other words, if we have resources that can only cover some percent of all of the things that we'd see as being work that needs to be done, you know, what would that top n percent look like, or X percent across all organizational units

And these are very difficult policy issues sometimes because the worst n could be looked at as across the entire organization or across units of organizations. Filters may have risk, resource or another basis, and they may have differential effects on the final ranking. So those may need to be compared.

So, for example, if you had some kind of risk score and all of these organizational units just labeled A, B, C through S, you might have a natural scoring that fell out of that risk ranking, and filtering, and you might use a risk based filter that says, well, if anybody exceeds this overall risk score of whatever, then that organizational unit is prioritized, and so they all did in this case the way I've drawn that line arbitrarily there.

The other way might be to take a more Perito (phonetic) type approach and say we're going to get the most of the risk score in that top level A through H, or whatever it is, and have it driven by the resources of available to do that.

So those are the types of questions that the risk ranking and filtering leads to once you actually finally get the ranking out of model.

Where does it fit in the overall cycle of risk analysis or risk management in some writings? Well, you start somewhere, of course, and our belief is that starting to look at the potential for risk management models is better than having nothing at all, and it's better than relying on purely historical information locked in people's heads. We want to tease that out into something that's workable for now and the future.

Start with assessments, databases. You know, come up with some multi-factorial risk model which is on the assessment side, and that then is information that goes into the risk management side.

And as I mentioned, the not only risk ranking and filtering goes into prioritizing work, but other factors are always at the risk management table.

Data sources, including quality systems and manufacturing science, in my view they really inform the risk modeling at that side and, therefore inform the risk ranking and filtering, but they are really at the heart of the detailed information, and this is all as shown as a cycle that goes on. It's iterative and hopefully improves with new information in each cycle.

Well, I hope I've conveyed that on the high level thinking and the philosophical thinking, that we're at a challenging area where we do get some real quantitative information here and there, and we have a lot of qualitative information from experts who have been doing this work, who have in their head a model that is working perhaps. And it's as Bernstein said, that risk management decision making are about where we confront probabilities, and it's a balance between the measurement and the gut because risk management is a judgment, and it uses any kind of information to make the best judgment possible.

Okay. Thanks.

(Applause.)

CHAIRPERSON BOEHLERT: Thank you, Gregg.

And I think we have one more speaker before we take a break. We're going to hold questions until we've had the four speakers on this topic.

MR. HOROWITZ: Let's see if I have more luck with this. Okay. So far so good.

Okay. What I'm going to try to do is take up where Gregg left off and transition to discussing how some of the concepts that Greg discussed that have been used in other contexts relate to our specific question at hand, which is: how can we be sure we get the most bang for our buck with GMP inspections?

Now, that question is even broader than what I'm going to be focusing on and what we'll be focusing on. We're not going to be discussing all of the different aspects of the GMP program. We're not going to be discussing how to make the GMP program or GMPs themselves more risk based.

But what we're going to be focusing on is putting aside those other questions now with the program that we have, with GMP regulations and thinking the way it is currently now. How should we best allocate our very limited inspectional resources? Where should we go first so that we don't run out of GMP inspectional oversight resources before we get to some of the most important sites to look at.

So let me go back to the start of the GMP initiative. In almost two years go, in August of 2002, which I look back at the concept paper periodically, and I'm sort of surprised that there are as many things in there that are sort of predictive of where we ended up because I think at the time a lot of people viewed those as pipe dreams and just words that FDA was saying, but I think we have taken some important strides.

And this model, our effort, we're really just getting off the ground on it, is an effort to pout into practice some of those words that we put forth in August of 2002.

One of the reasons we said we were undertaking the initiative, and these were three quotations here is that we wanted to evaluate the currency of our drug quality programs given that it had been 25 years since anyone had closely looked at GMPs and drug quality closely as we are now. But we wanted to, among other things, look at determining whether FDA resources are being used most effectively and efficiently to address the most significant public health risks, and we also said that in order to provide the most effective public health protection, we should match the level of effort against the magnitude of the risk.

Now, that's much broader than where you go for your inspections, of course, but we also said that resource limitations prevent uniformly intensive coverage of all pharmaceutical products and production. Although the agency has been implementing risk-based programs in some sense, a more systematic, rigorous risk-based approach will be developed.

Well, what we're talking about today, I think is just the first steps towards that end. This is a slide that amazes me, and it's the first time that I've presented it in public because I just couldn't believe the data, and I've presented the blue line before because there is a lot of evidence that our resources available to complete systems based inspections have declined significantly over the years.

Now, some of that decline has to do with resources being put into pre-approval inspections which have a GMP component to them, but that partly explains some of the decline, not entirely because it is quite precipitous, and I think the trend is likely to continue even though we've tried to stave off some of the decreases in the last few years.

But this green line is quite extraordinary because it shows tremendous growth in the number of domestic registered firms, and that surprised me particularly because as this industry is globalized, I though there would be not such a steep increase in domestic firms. I expected to see just a steel increase in foreign firms.

And what I think this tells is something else that's been going on in the industry for the few years, and that's more use of contract facilities, moire outsourcing and the phenomenon which is not something this group typically gets involved in of medical gas repackagers. A lot of these facilities starting in the '90s began registering with the FDA. Many of them were engaged in this activity before, but more and more started registering.

The more inspections we did, the more registered, and the more problems we found, the more inspections we did, and it got to the point where about half of our inspections were devoted to medical gas repackaging, and this is taking medical gases from larger tanks essentially and putting it into smaller tanks. It doesn't raise many of the quality issues associated with more complex drug manufacturing.

But anyway, the point of this slide is simply that it became very clear to us before we started this initiative that what made sense in 1980 and in 1978 as a strategy for inspection to meet our biennial inspection requirement no longer makes sense any longer, and we need to think about where this is going for the future. Every inspection has to count.

So we might not have perfect data. We might not have perfect knowledge, but we need to at least do the best we can to systematically use the information we have to prioritize our sites for inspection.

So what I'm going to do now is try to walk you through how we got from our sort of vague understandings of risk to try to take some of the consensus definitions out there of what risk is and how we use that to develop factors and then try to organize hierarchically as Gregg described these factors into a model that we could explain to people and that we would use for thinking about identifying risk factors, weighting them, and then prioritizing and ranking sites for inspection.

So let me start with risk. As Gregg pointed out, everybody has their own definition of risk, and they all have certain value to them, and they are all probably correct in certain contexts, but we wanted to go with a consensus definition, and ISO and a lot of other consensus definitions typically include two elements. They typically include the probability of a harm's occurring, and if it does occur, the severity of that harm.

And so I'm going to look now at the working definitions from Q9 to sort of figure out what harm is and how to apply these terms. I recognize there was a spirited discussion on Q9. It's still very much a work in progress. These definitions aren't exactly the way we in FDA would have done it, but I think for our purposes today they're illustrative of how you might go about thinking about these issues.

All right. So if risk is about the probability and severity of harm, of course, the key is risk to what. In other words, the key is how you define harm, and the Q9 definitions sort of walk you through several definitions to actually figure out what harm in the context of pharmaceutical quality might be.

And they start out by saying harm is damage to health, including the damage that can occur from the loss of product efficacy, safety, quality, and availability. Well, that, of course, begs the question, what is quality.

Yesterday we heard some discussion that Dr. Woodcock has some thoughts on quality that I want to link to these Q9 definitions. So we're going to focus on quality as the primary harm, that is, the core of the risk we're looking at.

All right. So what is quality? Well, there's a lot of literature out there on quality, nd it has to do with the degree to which a set of inherent characteristics of a product, system or process fulfills requirements. Well, that just begs the question of what are the requirements.

The needs or expectations that are stated, generally applied, or obligator by the patients or their surrogates, and I think we talked yesterday about how the regulators sometimes have to stand in for the patient to determine the needs.

So let me sort of try to combine these terms. My understanding of how those Q9 definitions and ISO definitions fit together is that risk quality is the probability and severity that a drug will fail to meet the needs and expectations of the patients and their surrogates.

Okay. So what are the needs of the patients and expectations of the surrogates? Well, that's what we heard yesterday that Dr. Woodcock has given some thoughts on, that I think link up nicely to this, and she talks about clinical performance being the key, and she said recently in May and before that several months earlier it's the delivery of efficacy and safety as described in the label derived from the clinical trials.

But I think we all know intuitively that the needs and expectations of the patients also include the availability of the drug, something we should consider in our risk matrix, and sometimes price, but that is something that consumers are more readily able to discern and are less dependent on FDA for, I think.

Okay. So Dr. Woodcock goes on and talks about how clinical performance is how the drug performs as described in the approved labeling, and that it delivers the relevant attributes of the drug and the clinical database on which the FDA approval decision was based.

So that begs the question which she answers: what are these attributes that can serve as surrogates for clinical performance? Because these then become the core to the risks that we're going to focus on.

And she identified some of the standard things that people talk about here, and this is largely true to her slide. We can all disagree about certain aspects, but I think we all intuitively know that there are certain areas that are critical quality attributes, that if there is a chance that one of those things or more of those things could be messed up, that's the kind of risk quality we're talking about.

So then risks to pharmaceutical quality can be identified based on the probability and the severity of an adverse impact on one or more of those attributes. And you could explicitly include factors that mitigate the probability and severity of those or the factors that have a positive impact in your risk model, and we tried to do that.

Okay. So let me try to summarize graphically my conceptual thinking and our conceptual thinking that underpins the model.

So we have the probability and the severity components here which make up harm, and ultimately it's the probability and severity of the adverse impact on quality attributes that are that harm. And so the quality attributes are sort of the linkage between the needs and expectations of the patient to the harm that we're seeking to evaluate risks or probability of severity of adverse impacts on.

So I know that's a lot, but really we tried to sort of go back then and say, all right, so how do we go about identifying risk factors with that conceptual framework in mind, and I think this is sort of intuitive to a lot of people. What hazards can adversely impact drug quality, attributes, and surrogates; what processes and parameters are critical for those quality attributes and surrogates; what factors may affect the identified hazards and the critical parameters and processes; and other variables that might be predictive of drug products with or without the identified quality attributes.

And that sort of, I think, goes back to Gregg's hierarchical chart. It's just sort of trying to organize our knowledge, thinking, and intuition about these factors.

Okay. So we start with from the previous chart the probability or severity of adverse impact on the quality attributes. We identify risk factors. We, of course, have significant data limitations which prevent us from including some of those in our model.

We want to build in certain incentives for developing process understanding, for doing the right thing, and for adopting the kinds of practices that are believed to be correlated with high quality manufacturing. You take those risk factors. You quantify them. You aggregate them. You rank, and then you start all over again.

And that's sort of the model that Gregg presented. Okay. Well, I'm not going to get into the details of the model during my presentation right now, but we did that, and we looked at factors, and we fried to organize them into categories.

Now, there's nothing special or unique about these categories. You could slide it ten different ways, but we felt that some of these factors are about the product. Some of them are about the process, and some of them are about the facility.

So what we tried to do is look at the risks associated with each manufacturing site and aggregate them and rank them against the risk scores for the other manufacturing site. So our goal is to systematically incorporate our current knowledge about drug quality risks in an effort to prioritize sites for periodic systems based GMP inspections.

Well, not surprisingly, we encountered some very significant data limitations, and that prevented us from capturing some of the elements that we hoped to capture this round, and I think this is a challenge obviously.

But it's also a great opportunity for us to go back and look at our data systems and start thinking about how to better capture data that will be more useful for this activity.

We also want to create the right incentives for drug manufacturers to adopt the practices that are correlated and connected with high performance and high regulatory and high efficiency performance. And I think this is an opportunity to do that as well.

Okay. So I'm going to just go through a slide each on each of those boxes. Remember there's product, process, and facility, and I'm going to just try to explain why we drew the lines for those three. It could have been done other ways, but when we were thinking about this category of factors, the product factors, we were thinking about what are the intrinsic properties of products such as the deficiencies in quality, if any, would have a more advertise health impact than others.

And we have some good recall data that's potentially useful, and among other things, it tells how the agency classified those defects associated with those products or dosage forms.

Another box was about the facilities and what we felt is there's a group of factors that really addresses the question are some manufacturing facilities or manufacturers in some cases more likely to produce a product with quality problems.

Well, we think that the effectiveness of the quality systems are predictive of that, and we believe that there is a connection between the compliance history or the inspectional record associated with the firm. Of course, not all violations are the same, but we do believe that there is some predictive aspects there.

Now, interestingly, one of the elements of risk is exposure, and I think it relates in part to severity and in part to probability, but if something goes wrong at a facility, the impact is likely to be much greater if the drugs are going to every household in the world or in America than if it's just a local facility producing a few drugs for the community.

So we felt that exposure of the drug products manufactured in a facility is a risk factor that ought to be considered by the agency in prioritizing its resources.

We also are very much looking forward to the results, preliminary and future results, from Professors Macher and Nickerson so that we can learn from and glean some additional factors that may be predictive of success that relate to the particular facility.

Okay. Then another category of factors we categorized as the process factors, and I think this is intended to answer the question are some manufacturing processes for particular product classes more likely to go wrong than others? Intuitively we sense that some processes are more complex and some were simpler, but our data is very limited on this. We didn't have any good quantitative data.

So our risk management experts suggested that we use expert elicitation. Now, we've started on this process internally within the agency. It's our hope to expand this external experts like yourselves and make sure that we're capturing the best expertise that we can get, but the Office of Pharmaceutical Science, for example, select, hand pick their best people to try to assist us in working on that survey. We have participation from field investigators who have a perspective, from compliance people, from folks across three different centers.

What we're trying to do is to use expert elicitation to identify risk factors and to assist us in this approach. They're going to look at, among other things, the risk of contamination or mix-ups and the risk of the loss of the state of control for the process for particular product classes.

There may be a potential here as well for process capability metrics and to include other quantitative factors in the future for this model, and we look forward to your input and others' on how we could do that.

I think I've been very candid with you that we recognize that this is a beginning. This cake is not baked yet, but we do believe that there's great opportunity for us to grow and to use this model to be more rigorous and systematic about our approach to selecting sites for inspection.

But inevitably the model can only be as good as the scientific or technical assumptions and the data that are used to develop the risk scores. We don't think there's anything magical about the processes we're using.

Multiple iterations and successive revisions will be necessary and we hope will reflect a growing knowledge base both within the agency, but more importantly outside the agency, and it will also reflect the extensive input from our internal, but ultimately we hope from our external experts.

So your input on prioritizing for improvement we hope will be very helpful, and we look forward to that.

Thank you very much.

(Applause.)

CHAIRPERSON BOEHLERT: Okay. Thank you, David.

Now I think we're ready for break. We will take a 15 minute break and reconvene at 10:40.

(Whereupon, the foregoing matter went off the record at 10:26 a.m. and went back on the record at 10:43 a.m.)

CHAIRPERSON BOEHLERT: Okay. We're ready to get started with the rest of our presentations.

Before we have the first presentation, I would just like to note for the record that we have no participants in the open hearing later this morning. However, there was one member of the audience that submitted some written comments. They have been distributed to the committee members.

Our next speaker is Dr. Tran.

DR. TRAN: Thank you.

Before I get started, I just want to thank David for such a good presentation about a model that I think Brian and I can just go back to our desks and continue to work.

However, we're supposed to go into the details of this model. Before I get into the detail, you've got a pretty good overview from Gregg about the theoretical framework on how we do risk filtering and holographic modeling and all of that and some of the general nature of a model.

What I'm going to do before I get into the specific is I'm going to talk to you a little bit about some of the applications that have been out there using the tool risk ranking in regulatory government, U.S. EPA, California EPA, USDA, and some of the management tools that Department of Defense had used, as well as industry using the risk ranking tool.

And the reason I want to talk about it a little bit is as Gregg mentioned, we borrow and customize the existing protocol model system out there to make it fit into what we're trying to do, and when I first met David, I was working on a project of risk ranking for DOD and that's how we kind of met, and that's how David brought me on board, I think, to help him with looking into all of this information and put something together that we just not create out of thin air, but use existing experience out there with other agencies, other industries.

So this is why this background. I'm going to go through it very quickly. I'm not going to spend too much time.

At the risk of looking very academic, I'm going to flash through some very, very busy slides. My background is environmental health risk assessment. I work a lot with EPA models, a lot with USDA type of models, and DOD models relating to chemical exposure. So a lot of this background is chemical oriented, and given that you are in the pharmaceutical industry, chemical should be something very familiar.

This busy slide is just to let you know that EPA, the European Chemical Bureau, Health Canada have gone through and developed a variety of risk ranking tools. These models are used to prioritize chemical substances. We have thousands and thousands of industrial chemicals out there.

These models are used to prioritize chemicals so that certain ones are going to be regulated based on potential for harm to the public or because of the volume that's being made up in the general commerce, so on and so forth.

So there are many, many models out there to rank risk.

This model, I'm going to flash through some more details, such as this EPA risk minimization tool. This is a regulatory decision tool, and before I start talking about these specific models, they have a variety of complexity, and they typically can range from ranking based on the pure hazard of a product. They could be based on the ranking of the potential for exposure for the listed products, or they can be ranked based on a combination of much of what David and Gregg talked about is the probability of exposure or the probability of harm, combination of the public exposure and the severity of the harm.

And this model has tremendous impact on the chemical industry. It's a very basic risk decision tool. It's the foundation for their solid waste management. It's called RCRA, Resource Conservation Recovery Act, and it's essentially prioritizing the universe of industrial chemical out there based on their persistence in the environment and will target those for specific regulations, an impact on a tremendous amount of industry out there.

And it is based on the framework of judgment really, and the term that I'm going to use a lot is "surrogate measures." Surrogate measure of exposure, surrogate measure of hazard, and surrogate measure of harm, and in this framework what they use are chemical emissions and some key physical chemical parameters to come up with some cutoff to prioritize chemicals which have tremendous regulatory impact.

And this very busy slide is like an influence diagram, and it looks very sophisticated, but it really isn't. If you look at those boxes -- and I'm going to focus on the human health concern box which is your far right -- you see the score three to nine. The reason I want to show this, you can see the scoring that we're going to be using. We talked about these as weights.

Essentially this system that has been used extensively by EPA is based on weighting human health concerns associated with chemical on a range of three to nine, and if you see those boxes that influence those scores are based on some surrogate of health effects, based on some very primitive information about cancer/non-cancer health effects, and some judgment about how to weight those effects on a scale, rankings of one, two, three.

And on the other side, you have the human exposure potential. This model, looking very sophisticated in this diagram, if you look really into the detail, it's a very simple expert judgment based on very limited information, as surrogate measure for exposure and surrogate measure for hazard and roll those factors up into a score and rank. Okay?

So this is the kind of concept that has been applied out there. The reality of it all is they have a lot of issues, a lot of chemicals. How do you prioritize which to target for regulation to pay attention to, to do research, to do more testing, so on and so forth.

And these frameworks are expert judgment based with some limited information, empirical evidence to support those judgments. And for the most part they are qualitative, high, medium or low ranking system. This one happens to be a semi-quantitative, ordinal scoring, one, two, three, four, five, six, seven, eight, nine, ten.

This is another system that EPA has used. They call it facility index system. This is to identify facility which releases that made up to the top priority list that they should pay attention to, and they look at the release information, then use a scoring system. How much is being emitted into the environment as a volume, as a surrogate for potential exposure? Those chemicals that are being emitted, what are the potential human health hazards?

Again, the surrogate measure for those is some weighting system that are put in, and some of the environmental persistence information, if the chemical has a long half-life, there's a surrogate measure they use to look at potential exposure.

A combination of those type of risk factors roll up into some scoring system to prioritize facilities. So that had been done. This was done in the '90s, and it's still being used by the agency in some fashion.

And very quickly, again, there are many different systems out there, and the complexity will go from low to high, and in this paper, Pennington and Yu (phonetic) had summarized all of the systems out there. They've looked for chemical risk ranking, and from low to high, in Group 1 essentially what I wanted to point out is you go from a very low complexity or model which is generic emission data to very complex Level 5, which is very complex information, very site specific risk assessments.

So the parallel is what we're doing -- number three is the scoring and ranking -- is middle of the road. It's not just volume of the pharmaceutical products that you make, but it's some combination, and we're not talking about a site specific risk assessment with the range of complex risk assessment that break a point, so on and so forth. We are about Level 3. Okay?

And, again, DOD has used this kind of approach to compare risk predeployment. I work on a project for them in looking at some of the chemical exposure, radiation exposure, physical hazard exposure. The troops might be exposed if they're deployed to certain areas overseas, and they can be deployed to many, many different areas all over the world.

So we have come up with a system of prioritizing based on these risk factors, a combination of some intelligence information and some expert judgment on how to bend this very qualitative information into high, medium or low as a framework to prioritize.

These tools are being used by AFMET (phonetic) to look into attachment data and where they should deploy troops, given what risk constraint they might have.

So as complex as those deployment situations may be, the data are limited, and they are forced to deploy under some very quick, straightforward risk ranking framework, to pull through that information and come up with some quick decisions. So that's been done.

I'm going to skip this. I think this is very similar to what Gregg presented earlier. The military model that I've worked with uses a combination of severity and probability of occurrence to come up with a ranking scheme to compare very disparate risks from chemical to radiation, to the bridge being blown up, so on and so forth.

Again, this slide is just meant to say you look at their interpretation of those very qualitative risk matrix of extremely high risk, from E, the red boxes, to low, the green boxes, have very critical meaning, and if you look at this risk level definition of the very last column that says unit stats, we're essentially talking about these qualitative terms translate to troops deficit. Fifty percent of the troops are going to be below unit strength. So they're talking about translating from this very qualitative term to something very quantitative, and this is not based on numerical empirical data. A lot of these are done out in the field with very limited information.

And, again, this slide is now the military in that context semi-qualitatively defined the probability of exposure. If you see the way they did it, they define unlikely as less than ten percent of the troops are going to be exposed to something, to an agent, to a hazardous situation.

Again, these scales are set up so that when they are out in the field with the limited information they may have, they can plug these in and come up with a ranking. Okay?

Another example that has been used, another example where risk ranking has been applied as a decision tool is, again, this has to do with constraint of resources. This is an industry initiative that I helped with.

It has to do with we have a lot of industrial chemicals that are in commerce, and there are a lot of chemicals that are used in high volume. They're called high production volume. For instance, they're mostly consumer products, a lot of the aliphatic alcohols, a lot of the surfactins. We use a lot of those chemicals, and they are very low toxic, but they have never really been tested for other endpoints, such as reproductive development toxin, so on and so forth.

So there's a pressure to do those kinds of testing, but we have a lot of those chemicals out there, a lot of products. We can't possibly test for everything. We need priority setting tools. Which of those products are we going to really actually test?

So this model is to help industry to do just that, and they are using these. And, again, lack of information. You can't really go out there and measure every single consumer product, every single chemical you have out there, how much you're being exposed to. So we use a very rough approximation of exposure.

This model is an exposure based risk ranking model to prioritize product that should be tested for, and this model is based on frequency of how much of a product you use, amount you use a day, percent that is retained in the skin. In this preliminary cut of the ranking, there's 100 percent absorption, 100 percent retained on skin, so on and so forth.

And as an example on one of the outputs in this model is for a chemical type, Chemical A hypothetically. This is a real chemical, but I can't keep the information. This is going to print in a hypothetical Chemical A. These are the product categories that this chemical goes into.

So based on this scheme, we would test aftershave because given the approximation of the surrogate of exposure, which of these products the public are exposed to the most that would have this Chemical A. Aftershave would be the one.

So that's the kind of very simple, straightforward strategy to come up with what product you're going to test. So you can't test all of them.

I'm going to skip the microbiological as the same idea. It's using some information to bend the hazard based on the property of microbes and score and rank.

The Ross and Sumner is a food microbe ranking system that has been developed by the Australian authors. This is being used in Australia, and the point here is this is another risk prioritization tool, and it asks a series of questions, and I'm going to just flash through a couple of questions that this model asks the user to go through.

One is the hazard severity, and again, if you look at this chart, it's again an expert based framework. The question is: how severe is this hazard?

And the user with this model is asked to put in the weight, and these are arbitrary weighting factors based on your expert knowledge. Okay?

Again, in these food risk ranking models, you tend to think about consumer, and are these the acceptable populations that are going to be exposed, and some of the susceptible populations, infants, AIDS patients, so on and so forth. So in this model they use again a weighting system to weight up the population that you should be concerned about.

And, again, this is based on your knowledge, some empirical knowledge about what percent of the population you're trying to protect, fall into these categories. So this is some empirical information, plus some judgment on how you put those weights on those percent of the population.

And this model is a look at the process. A look at the process is like to reduce the growth of the microbes and, again, this is arbitrary weighting based on the expert judgment.

One of the models that is really close to what FDA is doing is the USDA Food Safety and Inspection Service, inspector optimization system model. This is the model they use to prioritize the inspector work force. Again, they also have constraint, limited resources on how many inspectors they have and how many meat and poultry processing facilities they have to go and inspect.

And they have written this up in a report to Congress in 2001, and this model at the time was purely a hazard based risk ranking model. What they have come up with with this model is a food safety hazard coefficient that's based on the inherent hazard of the food product, which is meat and poultry, and it has the process of making these food products, and they use an expert elicitation, but there is no data. If you are working the food industry particularly, there aren't any data in terms of sampling, very limited sampling data.

So in this FSIS model of prioritizing the facility risk so that they can deploy inspector resources accordingly, they basically used three variables. One is a species variable to reflect the inherent biological, chemical, and physical hazard associated with the meat and poultry that are arriving at the inspector. The data don't exist. Expert elicitation is used to get at that.

The second variable that's a reflection of the inherent hazard is the process variable, and again, in this process they assume normal process, normal slaughtering plant, normal packaging plant processes.

And a third variable they put in there is the volume, very similar to ours. They wanted to have some surrogate that would account for the potential for the number of consumers that might be exposed should this product going out they would be exposed to. So they use a volume, the facilities' size

And a little bit about the expert elicitation. Again, they don't have any data on the species variable or the process variable. What they went through is a process of elicit opinions from known experts.

And they have two different elicitations. One is on the hazard itself, on the product or the species itself. The species are where the cows are views. And the question that they ask here is: based on your expertise, rank these; rank order these from one to ten. How hazardous are these? How likely are these going to be contaminated with microbes going into the processing plants?

And you can imagine this is a very tough elicitation because where are these animals coming from, the geography and the season when they're being brought it is going to change the answer.

So this is not an easy elicitation that they had to go through, and they had to be really careful what expert they're going to choose, and they used a combination of government, academia, and industry expert elicitation.

And they did the second elicitation on the process, and the process is the grinding of the beef as an example, the slaughtering process, you know, different kinds of processes, and again, the same series of questions were developed, series of experts were selected to elicit and rank order these.

And so that's the process they went through. Their model is hazard based with a surrogate for exposure which is the volume, and it's a coefficient score at the end to rank the sites.

And their model is also evolving. There's also a learning and evolving and the model is going to be improved over time. This is the latest presentation by Elsa Murano from SSIS.

Their next step is to put in, to change, to modify to a hazard control coefficient, and what that does is they can incorporate compliance history into these coefficients.

So now the first phase is the apparent hazard with surrogate for volume. The next phase, to put in the compliance history, to improve the scoring, and to rank the sites to target inspection.

So that's what's going on out there, and there are many more out there, and they are evolving, and everyone that is trying to use this kind of system to work smarter.

Okay. Good. That took me five minutes. I didn't want to spend too much time on that, but if you have any questions, you can ask me later on.

Okay. Now, let's go into CDER office compliance process. What do we do?

So having been through all of this risk ranking process with other agencies, when I met David, I said, "Please help us with this." And as you know, you works in risk assessment. It's easy to talk about concept in terms. It's very hard to operationalize anything. So that's the challenge.

We began all of this a year ago, and David and people at CDER, CVN, CBER, and ORA have an internal expert working group. I think Gregg and Brian were all members on that working group, and they have gone through with their expert in house, gone through and generated a list of what they think is relevant risk factors that we should consider for site risk ranking and that we should consider in developing this model.

And they have gone through a process of generating those risk factors and assign them values, high, medium, and low risk, and this is an example.

When I first showed up, I was given a paper about five pages long. It's a spreadsheet of factors, a just listing of factors and risk descriptor, high, medium or low, as you see here. And I looked at it, not having worked in pharmaceutical, coming from a very different background. I said, "I don't understand. How do you come up with risk, high, medium, or low? What's the context? Risk to what? Risk to whom, and what is risk?"

And I was asking a lot of dumb questions because I just didn't know what all of this was coming from, and by asking some very basic questions, it became to emerge -- well, back up to what Gregg said earlier. As a risk assessor, we like to systematically organize things. So when I saw these lists of five pages of factors, I wanted to organize them. I had to put them in context.

So we began a process of coming up for air. We have too many details. We need to come up for air. We need to get back into the high level organization, into somehow all of these factors have to fit in certain categories so that we can systematically organize them, manage them, and combine them.

And that's how the three components are derived. It's based on a process of discussion, of me asking a lot of questions of what are you thinking. Why do you think this is high risk? High risk to what? High risk because the product is high risk? This variable, if something goes wrong, the product is going to potentially impact the users, or if this variable goes wrong, does it have to do with the process? What does it have to do with?

And in the end, through a serious discussion, things start to fall into the natural categories. For instance, some of those factors, I'm just showing you some examples here. The dark blue, through membranes, that's a factor that has to do with the product versus cartooning and packaging has to do with process. So we go through a process of categorizing that way in the facility.

People talk a lot about approval first time. You know, that falls into the nature of the facility. What is that facility all about?

And I think David already gave us a pretty good background on this chart. So essentially we took a bunch of factors, a big list of factors, organizing them and make them sit on three legs essentially. So now we've got the three legged stool to work with.

So one of the legs is product. One of the legs is facility, and one is the process. And the idea of the framework is we're going to go back down, drill down to these boxes, to these legs, and make them walk, and in the end we can fill it all up and have the site risk potential, and that can be the score.

So as you can see, this is very similar to some of the other models that I just flashed through very quickly at the EPA what they've done, the USDA, what they've done, and what DOD has done. So this is not different from what's been done. It is just a different application.

In the next couple of minutes I'm going to talk about drilling down to those three categories. How do we select the factors given the laundry list of factors that we have categorizing into these categories? Which of those are workable? Which of those that we can actually work with? Which of those that we actually have data, empirical?

By the way, of those Bayesian, I'm a strong believer of having data before I start. I don't have any prior, but that's my bias, but then we also --

DR. SINGPURWALLA: That's a tragedy, too.

DR. TRAN: It's a tragedy, but don't forget. Once we have the empirical data, we can put in some judgment. That becomes somewhat of a prior.

DR. SINGPURWALLA: Well, we'll talk about this.

DR. TRAN: Yeah, I made a mistake. I told them I'm a frequentist. Big mistake.

Okay. And so once we select the factors, it's going to be driven by how feasible are these factors. Do they make sense? Do we have data?

And judgment has to be on some kind of avenues. We can't just be pulling out of thin air, in my opinion, and from that we develop a logical algorithm to combine and then come up with a final composite score.

I'm going to talk first about, again, this. We have three components, and we'll talk about the site product score very quickly. How do we populate that component?

And we teased that out into two more subcomponents. One is the intrinsic factors, inherent hazard associated with a product, and these categories, these factors are the intrinsic factors that David had talked about earlier, sterility or non-sterile drugs, whether they are over the counter or prescription drugs.

These are very rough approximation of intrinsic factors. We recognize that. This is something that in the long run we would add additional intrinsic factors, true intrinsic factors of potential hazard associated with a pharmaceutical product, that if something does go wrong, the consumer will be severely impacted.

So we recognize this is a very rough approximation. This is only the beginning. What we're most comfortable with is recall data. We have empirical data out there that tells us about the severity of the quality effect and how frequently that does happen.

So the bottom line is for the moment, the model, we have put a lot of emphasis on the recall data, and one of the challenges, we're using the recall data is we need to be able to link the recall information to the site because remember this whole model is to be able to somehow capture the three components, assign it to a specific facility, come up with some kind of a score and rank them, rank order them, and then we can target the right one for inspection.

And our data source for site information is the fear accomplishment (phonetic) and compliance tracking system, and please don't ask me any more about the database. You have to ask Brian for that. I take the data from them, and I just use them, and I'm told this is where all of the site information are being kept.

And also in this database there are product codes, but these product codes aren't the same as the recall data code. So we have a challenge of matching data. So that's one of our challenges.

And we went through a process of grouping the recall, and I think I have a slide to talk about that. No, I don't.

Essentially what we have to do is since we cannot assign the recall data to a specific site, we stepped back and we said, okay, let's aggregate the recall data into some fashion that we can link it up to the site, and one way of doing that is in the FACTS database we kept the data based on dosage form or profile class. Some of the product classifications that the earlier presentations, so that's how we rolled the recall data into those product classifications, and then those product classifications are associated with the sites.

Again, we use the CDER recall database, and we are looking at the recall data between 1997 and 2004, all of the occurrences that we've had. This is how we are looking at in terms of putting a weight to the recall data. This is the recall weight matrix. It looks like that probability and severity matrix that Greg had showed earlier. Like I said, we borrow methods from existing literature from other agencies, and this is one of the ways that we're going to weight the recall data, and these are the weights from one to five that's going to be assigned to each dosage form and that's going to be attached to a facility.

And, again, we don't have probability. So we are looking at some surrogate percent of total recall in an HHE class, and five is the highest hazard. One is the lowest hazard if you want to interpret this directly.

I already talked about this. I'm going to skip this because I talked about the correlation.

Let's go to the facility component. So that's essentially for the timing of what the product component factors look like.

The next component, the next category, the next sets of factors are the facility, and where are the components of the facility box in the site risk potential score? At the time being we have three basic components within the site facility score.

The history of inspection. We're looking at a scaling, a weight scale for this factor, and essentially if a site has been recently inspected, it's going to get a very, very low scale, less likely to be picked up in the next year, so on and so forth, and if that site hadn't been inspected in a long time or never been inspected, it's going to have a higher scale there.

History of compliance and violation. This is the OAI, though no official action and the OAI category. We're going to pull that in here with a weighting scale, and OAI is going to have a highest score. So the 30 that had a history with OAI would have a higher score there, and the volume, again, this is a surrogate for potential impact for this facility should they have something to go around with this facility in terms of reaching the consumers. This is a really rough approximation.

And, again, for this facility site score, our data came from FACTS, field accomplishments and compliance tracking system, and we are downloading the data for the years 2000-2004, and all sites are being scored in this way. They are all foreign and domestic firms.

Last but not least is the process. I think this is the one that's the most interesting so far, is the process component factor. This is one that gave us a lot of headache because it was the toughest one.

We didn't have any data. The idea here is the factors that should be fed into the site process score are the relevant inherent process risk factors. What are those? And the relevant process controls and risk mitigating factors. What are those?

And we understand that these factors are product and facility specific. This is when we ask people to kind of come up for air and think broadly. It always goes down into the very level of detail. A very specific product, very specific facility. So this was a huge challenge, but I think the working group was successful in having a lot of discussion on how to kind of step back up and categorize products, categorize unit of operations, and come up with a process, an explicitation (phonetic) to ask people questions, to come up with some information on how we can come up with this process score.

And I'm going to turn this over to Brian since he's spent a lot of time with the expert group.

DR. HASSELBALCH: Yes. Well, it's a bit strong to say no data. We have data. It's just locked in paper files, and we have no ready way of getting at it in any time soon.

So we thought it would be nifty to query the experts in the agency. We could have gone outside the agency, but that involves some other bureaucratic hurdles we didn't feel like we wanted to deal with at the moment. So to expedite things, we stuck with experts inside the agency.

We began drafting the document with a smaller group of experts among the various centers involved with regulating medical products, but our device center, and the key questions we asked in drafting the survey, which I'll show you excerpts from in a little bit, were to ask what are the relevant process related risk factors. In other words, could we think of processes in terms of the source of variability.

Naturally, of course, we can because they not only contribute to variability,b ut when they work well, they contribute to homogeneity or lack of variability and good quality.

We also asked what, if any unit operations are more reliable to a loss of control or to risk from either environmental or product to product contamination?

We drilled down to unit operations you'll see shortly, but as you'll also notice, we don't actually allow much for the unit operations in a final aggregation because of limitations of our site identifiers for information.

Thirdly we asked should the experts or would the experts want to distinguish among products or product types. Could we categorize all products into certain groups and expect the experts to reliably distinguish between those groups of products in their opinions or judgment about risk to variability, quality and control and contamination.

Naturally, we felt we could expect that distinction from our experts. So we set about identifying mutually exclusive categories. We borrowed a bit, I should say, from ISPE's Baseline Guide. I've given the site here for soderol (phonetic) dosage forms. It's at the back. It's intended to be a tool for companies to use in building new sites as to those areas that may cause them more or less headache or difficulty or cost in constructing and making operational the new facility.

I've just taken a page out of this. It's several pages long, covers different areas, but very nicely I think it signals us that it's possible to distinguish unit operations by product types when we're talking about GNP issues like variability in terms of process and contamination.

So a big struggle was in categorizing products to get a number that wouldn't be too burdensome for a panel to ultimately answer on, but on the other hand to make it fine enough so that we could, going back to our inventory of sites, identify those sites by those kinds of products.

We code in our agency many things, and one of the things we code in many different ways for many different purposes are the kinds of products each site makes, and by "site" I mean manufacturing facility.

We found a lot of cross-correlation. I'll show you some of that in a little bit. I know the professors are being challenged by that issue as well.

Again, we chose to create families of products by their relationship to similar unit operations, so blending, mixing, tableting or compression or fill, liquid or solid.

We also distinguished high from low active weights. We felt the experts might think differently about the influence blending has on a product if that product ultimately has a lot of active percentage of its total weight or very little active.

Again, the variety of resources, including experts. Here's just a taste, if you will, of our cross-correlation. The product groups you see on the left are those groups ultimately that will influence the model. So that's the aggregation. They will -- I'm sorry -- that are in our expert elicitation survey.

The middle column are those codes that identify those kinds of products that exist in our data systems, and the description is off to the right.

Here's an excerpt from the survey just to give you an indication of the kinds of products we chose again, and here are the questions we asked the experts. These are the five questions we asked each expert to answer on a scale with respect to the various product types you just saw and the unit operations that you haven't seen yet, but that are at a smaller expert panel associated with those families or categories of product types.

Three of the questions have to do with and I think get to process control. The other two have to do with contamination. I think, you know, our feeling was in crafting the questions this way and including only these questions, that we were really capturing the essence of the GMP standard or control requirements.

This is an excerpt, just an example. Again, solid oil drugs, in this case immediate release, the five questions, the scale that the experts were asked to answer on, and you'll see here the unit operations we identified as typically occurring or used for this kind of product, and of course, it would be the same whether it was high active or low active for the most part.

After asking the experts to go through this ranking exercise for these different product types by these control contamination questions in unit operation, we then rolled it all up into a single page questionnaire about whether they felt essentially whether process control or contamination was more or less significant for those product types.

So in other words, we took out the unit operations and just asked them is process control or contamination, if you had to decide, which one would be more important to you in terms of the quality of the product being produced from that process.

We, in fact, did not deliver by E-mail. We delivered by paper. Well, we sent it by E-mail. Everybody printed it out and did it by hand, and then we consolidated the comments by hand as well.

We got 50 experts to participate from a variety of staff members. We had a 90 percent response rate. I think that may be because some offices were really heavy about getting the answers back.

The cooperation was very good, as a matter of fact, and we're still analyzing the results. Now, I don't know if you want to go into too much now, at the risk of some discussion at the moment on how we analyze or are considering analysis of the expert elicitation data.

Yes, please.

DR. TRAN: I think this is a team effort. I'm going to need Gregg to talk about the fuzzy arithmetic. We're looking at the data right now, and we did some exploring and I was just graphing some of the average answers and see if there's anything that looks like an outlier, and for the most part, the answers are pretty consistent, that there are no real outliers out there.

And we have the two different ranks. One is the product ranking, the general big picture ranking. This is a list of product, the using process, oil contamination. These are the weights as of our last survey.

We did that internal validation. We just want to make sure that the answer for the unit operation drill-down is not going to be so different. We wonder if they're going to be really different from the overall ranking and the correlation is pretty good.

And we're in the process of developing process weights based on the unit operation, drill-down survey. That's the most comprehensive way of looking at that, and as a true frequency, I'm looking at K-Ming (phonetic) cluster analysis and Gregg as somewhat of a Bayesian, he's looking at fuzzy arithmetic, and the two of us are going to come back and compare notes and see which way we want to go. I think we're going to go with the fuzzy math as soon as we can get all fuzzy about it.

Do you want to talk about that?

But the K-Ming cluster is just the five questions combined, use cluster analysis, and the weight is going to be given the highest weight for the cluster that has the highest center, and that's very straightforward. It may not be suitable for expert data, categorical data. It's just that we think the fuzzy arithmetic might be the better way to go.

Gregg, anything on that on the fuzzy stuff?

DR. CLAYCAMP: I don't think it's necessary to go into any details now other than the real objective here is, as Brian mentioned, that we don't want to lose detail in our probing of the expert mental models that have been out there and been doing this for years, but once it hits the spread sheet, all of a sudden we have a lot of information before us, and so we're asking questions. Can we collapse this into its key drivers for the sake of simplicity?

And so it's looking at principal components, for example, and you know, very, very preliminary analysis is they kind of fall into lines that the experts would have told us in the first place.

So those are the reasons that we're looking at those techniques that it would take, you know, as many as 11 measures down to hopefully a couple that would be easier to handle as weights in the model.

DR. HASSELBALCH: This is the summary chart. Again, for the model scoring purpose, we'll likely distinguish process controls from contamination and let both of those contribute to a single site score in addition to the other categories of product and facility.

Let me just summarize in plain language. At least I can do that.

The model's impact on our inspection decisions. It is simply that a site will tend to be less frequently inspected if it has been inspected recently and/or has relatively few previous violations of GMPs and/or smaller volume product. So that all contributes into the facility weight module.

It will be less frequently selected for inspection if they make non-sterile OTC drugs and there are other product types that aren't associated with a high frequency of serious recalls; contributes to the product weight of the model, and the process solicitation data largely will contribute to the third element, which is that they make products estimated to be relatively straightforward of manufacture and not vulnerable to contamination.

Of course, the converse is also therefore true. Sites will be preferentially selected for inspection on an annual basis if the opposite holds.

This also summarizes in chart fashion the scoring scheme and the contributions now into the model. I think this would be a good time for me to point out that largely we have to communicate. The difficulty or limiting factor here is largely to communicate this to our field staff.

We have 19 different district offices. Any multiple of that that are involved in program planning at the district level, we need a any to communicate to them the center's priorities for inspection in a way that will allow them to strategize or conduct their inspection to take into account those areas of production or the facility that seem to matter the most, that seem to influence the most the risk that that facility has in our marketplace.

This is not a model to predict a violative site, though it's going to have a tendency if we pick bad sites. Historically there's a preference, but it's not design for that purpose. It's largely intended to get to those sites, FDA inspectors at those sites, reliably, at a reliable frequency that seem to matter the most in our marketplace.

Of course there are things we'd like to include in here for which we presently lack data or a mechanism to account for them, but these, again, as David mentioned, we expect that this model will change over time, and we'll have to incorporate additional information as we go along.

And I think one area where we can easily include some future information would be in the area of some metric associated with process capability, whether it's a CPK or some measure of yield or success at making batches. We're hopeful that that will have a future impact on the model, perhaps drive down the score for certain sites.

Okay. There are some questions that I think we'd like the subcommittee to ask, and, David, you'll facilitate the section?

(Applause.)

MR. HOROWITZ: Okay. People may be getting hungry, and I know there are a lot of questions and comments that have been building for the hours, and so I just want to, before we start, reassure people that this is not your last opportunity to comment on this. This is just the beginning.

In particular, in September when we announce a big announcement on the GNP initiative, we'll be putting forward a small white paper that will describe some of these things. We'll be opening that up for public comment and whether it's a docket or through other forms, and we hope that you'll all bring forward the comments from today, but also other comments that may occur to you subsequently and other constructive suggestions on how to make this better.

We're hoping to pilot a rough version of this model for the coming fiscal year, but it won't consume all or even a very large portion of the field's resources, but some of the field's resources will be devoted to doing inspections that are derived from this model.

So with the permission of the chair, I could start on these questions then, and I recognize that you'll probably have comments that go beyond these questions. That's okay, too, but if I could, I'd like to start on these.

First, can you identify alternative approaches that would systematically prioritize manufacturing sites for GMP inspections?

I have a feeling that there may be some ideas out there on how we might do this completely differently, and we're all ears. We'd like to hear some other ways that we might be able to accomplish the same objective we have with the limitations that we face in data and other things like that. So, please.

DR. SINGPURWALLA: Answer to the first question is yes.

MR. HOROWITZ: Okay. Anyone else?

(Laughter.)

MR. HOROWITZ: I want to get that yes.

DR. SINGPURWALLA: Yes.

CHAIRPERSON BOEHLERT: Yeah, I'll let you recognize the committee members.

MR. HOROWITZ: Oh, okay.

DR. RAJU: David, going back to the comment that you made at the start of your presentation that this is more about inspection rather than the broader initiative, are you willing to entertain some broader initiative responses to one that connect back to inspections?

MR. HOROWITZ: Yes, I am, just recognizing that this model is not intended to go beyond its very narrow purpose, but I'd be glad to.

DR. RAJU: In the end, safety and efficacy and availability are about a product that somebody consumes, and he really doesn't care or doesn't know what site it's made at. So an alternative approach would be about a violative product and about prioritizing the manufacturing product rather than the site, given that, of course, the product has to be made at a site.

I know you've laid the foundation for it. I've seen Brian's presentation, and you've laid the foundation for it, but looking beyond, could it be about privatizing among products rather than sites as an alternative approach that your foundation might get to because the customer really doesn't start with the word "site." He starts with the word "product."

MR. HOROWITZ: Yeah, I'll start briefly, and then I'll ask the other speakers, but I think that's very plausible. Ultimately though the way the inspections work is they have to connect the product to a site because they have to decide where to go, and I think drawing that connection out would be very valuable, and I hope that the model begins to do that, but I think there is probably more opportunity for focus and knowledge to be derived and applied in that area.

Brian, I think you were.

DR. HASSELBALCH: Ditto. Exactly. I think as a start it's fine, but I think the future will have it smarter and make us capable as a bureaucracy to distinguish not just sites anymore, but processing lines at sites. Because after all, a site could be very big. It could be multi-building, huge campus, or it could be one building.

And I think in the future we'll be more capable of making those distinctions, but there are some things that have to happen internally about how we count the work we do and value that that also have to change along with that because we're now heavily driven by sites, addresses in terms of budgeting and planning.

But thank you for that comment.

MR. FAMULARE: I just think one fact to think about, G.K., is that a lot of the work we've done over the last ten years since the generic drug crisis was product and preapproval inspections, and we've seen the fault of not covering systems fully, sites fully. So in order to get back into those sites and systems, proper quality systems at a site facilitates products, changes, and continuous improvement.

So there is an emphasis back on quality systems which right now translates somewhat to sites, but as Brian brings up, being able to then drill that down to product lines' processes would be the next step.

DR. RAJU: You can go to it both ways. You probably have to do it simultaneously. The problem with going to the site and all of the paper work and the quality system and all of the tracking is, given the legal relationship between regulatory and regulated, there's such a big degree of gray area before you go to the truth with this, the physics, chemistry, and biology. That's the process that goes into somebody's body.

So there's the physics, chemistry and biology that depends on a system to do it right, and the other vocabulary is being put in place, and you always need both, but I think we probably have overemphasized the top-down too much.

MR. HOROWITZ: But before I go on to the second question, maybe I'll follow up to Nozer's answer to make sure that no one is constrained by the wording of the question and say that if you have additional or alternative approaches that you'd like to recommend and ask us to consider, now would be a good time.

DR. SINGPURWALLA: Well, I'm glad you asked because your question says can you identify, and I said yes. But now you're asking me what the alternative is.

The way I would see it is I would see the problem of inspection, of choosing a site for inspection, as a problem in making decisions. So I would draw a decision tree, and I would choose that particular site. I would prioritize my site according to the expected utilities that I would get from each decision tree.

So I would draw a decision tree and do it, which is the way one should choose sampling inspection plans and amount of sampling that needs to be done.

So I would use the standard recipe for doing it in a more formal way, and that's all I have to say on that one.

But I do have comments on the presentations. So I hope you'll give me a chance.

MR. HOROWITZ: Okay. Can we get through these next few questions?

DR. SINGPURWALLA: Yes, absolutely.

MR. HOROWITZ: Can I just ask Gregg if Gregg wants to respond to that first on the question of decision trees as an alternative approach?

If you have a comment, please share it with us, and then Paul is next.

DR. CLAYCAMP: Right. At this early stage, that was a little bit overwhelming overall, but a lot of this does fit right into that type of process, and that's my personal bent, is to set up decision trees.

DR. SINGPURWALLA: So you recognize that.

DR. CLAYCAMP: Absolutely.

DR. SINGPURWALLA: Yeah, thank you.

MR. HOROWITZ: Paul.

DR. FACKLER: I just wanted to say that I'm guessing you haven't finished this analysis so that these sites haven't been identified or prioritized, but when that has been done, I think it might be useful to look then at the distribution of the sites, recognizing that more than half the prescriptions written in the U.S. are written for generic drugs, it would be useful to look at the distribution of generic versus PhRMA site and bio versus traditional oral, small molecule sites to see if the distribution is similar to the distribution of products in the United States.

Not to say that they necessarily will correlate, but I think it would be an important thing to look at. I don't think you want to make this simply a scientific assessment or an objective assessment. I think that there are subjective reasons that might cause you to change you inspection procedures.

MR. HOROWITZ: Thank you.

Okay. I'll go on to the second question then. In what areas would additional data provide the most value added in prioritizing manufacturing sites for inspections? I mean, you could all see that our data is very limited here, and you know, one of the things we need to think about is prioritizing our efforts to improve this model.

So I'd like your thought on where we might add data to this model. I'm sure there are other improvements people can suggest as well, but for this question we're focusing on where additional data might be most valuable and improving the model for our purposes of getting the most bang for our buck.

It looks like Nozer is -- no, your red light is not.

DR. SINGPURWALLA: No.

MR. HOROWITZ: Okay.

MR. MIGLIACCIO: I'm having a little liberty with the question.

MR. HOROWITZ: Please.

MR. MIGLIACCIO: Because I'm not sure. There's one data point that I'm not sure it is going to have the right value, and that's volume. I'm very concerned about the volume factor.

First of all, it would imply that GSK and Pfizer would get most inspections, which if you look at the way some of us run our business, you will have high volume facilities that make only one or two or three products, and inherently the risk is lower in running those. There are fewer changeovers.

And then there's the dosage regimen. How much exposure is out there depending on how many patients there are for that product. Volume in itself is not a good factor to use. It has to be expanded into other -- you need to complement that with something else. Pure volume I am very concerned is going to lead you to low risk facilities when you look at it.

So I'm concerned. We have to figure out how to complement volume with something else because going to your question, you know, you're going to direct it to high sales companies, and that's a concern.

MR. HOROWITZ: Okay. Can I just briefly respond to that? And then I see Ken's light is on.

We believe that the model as written now does complement it, as you put it, with a variety of other factors. If volume were the only factor we looked at, the model would be absurd on its face, but I think there are so many other mitigating and other factors.

The weight of volume in determining frequency of inspection is actually quite low if you take out that factor and the fact that, you know, it's counterbalanced by so many things, some of which you mentioned. If the high volume site does a good job, for example, you could expect that they wouldn't have a particularly bad compliance history, and I think that would be something that would be weighted in.

If they do a good job in a high volume site because it's easier to focus on that, they might have fewer recalls associated with that product, and so forth.

So it's definitely something we need to watch for though, and I understand your concern because you don't want to create the wrong incentives. I mean, obviously we want to encourage firms to adopt those mitigating and other factors which take advantage of, for example, the good things associated with high volume manufacturing.

Anyone else from the speakers who wants to address that? Gregg.

DR. CLAYCAMP: Yeah, just to follow that up, you know, at this point if you try to look too formalistically at the details in this, you'll see things going on that in the modeling sense will look like confounding and multiple colinearities, et cetera.

So right now, the conclusion you'd come to is that it is being tempered by, for example, when we asked the experts in brainstorming what were the factors to do with processes, making the same thing all of the time was lower risk than process changes, and so that kind of works against the volume rating.

So, you know, there are competing factors in the model right now that I agree with Dave that it probably in the end isn't weighing very much.

MR. HOROWITZ: Right.

DR. DeLUCA: David, you need a volume risk index so that when you have the risk that doesn't include the volume, but then that comes in as an index because if a small firm is a medium risk and a large firm is a medium risk, then I think the large one plays a role

DR. MORRIS: Yeah, I guess it's sort of the same point said slightly differently, but you know, 100 deaths is worse than 10,000 cases of diarrhea, for instance.

MR. HOROWITZ: Absolutely.

DR. MORRIS: So even if it's local, which is what somebody else had said earlier, I think even if you have a local effect, it can be much more detrimental.

the other point I wanted to make in terms of the areas of additional data, I'm not sure quite how to do this, but there's a bit of a problem using historical expertise when you factor in where we're going, I guess, because on the face of it -- I'm not saying this can't be overcome and within the same system -- but you're bringing into question issues like, for instance, if you say that your last inspection, if it was more recent, you're at lower risk. Well, if you're controlling your process, monitoring and controlling real time so that you have gotten the regulatory relief so that you don't need as many inspections, then that ends up making you higher risk even though it is innately making you lower risk.

Similarly for things like Sterile processes being counted as higher risk than non-sterile. Historically there have been, you know, some very elaborate mechanisms for making sure the sterile products manufacturing is very reliable. So are you penalizing them in the face of being more reliable?

And finally, the controlling of a process when we're talking about the -- I'm referring now to the process of the unit operation ranking of difficulty in the historical expertise -- if you're talking about controlling to time as an endpoint, then that will give you in many cases a very different answer in terms of the reliability or risk of that unit operation than controlling to the endpoint.

That's all I have to say.

MR. HOROWITZ: Thank you.

G.K., I think you have one.

DR. RAJU: In terms of Question No. 2, I think there's a systematic -- if you went back to Janet Woodcock's definition of quality and you said safety, efficacy, and availability you said, but you were the surrogate of the customer, and then you define surrogate variable, such as identity purity that you were going to do your regulations around.

But when you made the mapping from the customer to the surrogate measurements, safety and efficacy, but presumably mapped on, but availability didn't show up in that mapping, and so the system that we have is predisposed to go after a company that might be making a very, very difficult product that nobody can ever make, a sterile product, a vaccine that would never have been on the market, but it's available.

So you would go after maybe a sterile product or a very complex process that they were the most innovative in the world to make. So how do you eliminate that bias of availability not being in your broader risk, although it could be outside this model?

MR. HOROWITZ: Yeah, I think that's an excellent point. This is why in a lot of these comments I think one of the themes is we need to be careful about the incentives we create here because it could have unintended consequences, and that's one of the reasons why we're rolling it out for input, one of the reasons why we're going to be phasing it in slowly.

But I think the particular issue that you raise with regard to availability, that might be something we could consider as a mitigating factor or a risk decreasing factor if the product is at risk of loss of availability. Perhaps that's something that we ought to take into account.

But I want to say though that just because we inspect it doesn't mean it will be taken off the market because there are other ways that we can take those factors into account.

Now, some would argue a critical lifesaving product that is a single source product that is really hard to make, we should be inspect them and working with them in trying to help them make sure they can keep manufacturing.

DR. RAJU: Right.

MR. HOROWITZ: So it doesn't necessarily need to result in reduced inspectional oversight for this model, but I take the bigger point that we really need to be very careful about the incentives that we create to make sure they're the right ones to push and encourage the industry to improve their process of understanding and to adopt the most modern technologies.

Joe.

MR. FAMULARE: You know, just going off, I second that. Very often when we're in those situations we will inspect more towards working jointly to resolve those issues and those very complex products, but also to respond to what Ken said before in terms of depending on the regulatory paradigm and the advancement of modern technology, PAG, and so froth, you're saying it may result in less inspections or it may result in a different way of looking at things.

You know, a lot of the discussion yesterday was about reducing supplements, and therefore, at some point not only will the investigator, but what we have factored in, the product specialties may want to look at that. That may be a factor that we bring in to target. Not only will we look at that at inspection. It may be at an appropriate frequency, but it will be a way of targeting when we want our product specialist there because they're looking to reduce their supplement burden, and so forth, and bring that along.

MR. HOROWITZ: Don.

DR. GOLD: There are a couple of points that I wanted to add. One is to look at or consider hard to fabricate products. I think this was already mentioned before. There are a number of products in the marketplace that are quite difficult to fabricate and where controls are very important.

And, secondly, there are some products in the marketplace where control of uniformity of dosage is extremely important, where the patient has to be titrated and the product has to be carefully controlled. And I think that has to be added to the mix as well.

Finally, I'd like to make another point. Perhaps you're getting to this a little later on in this discussion, but with the absence of a dedicated pharmaceutical inspector, there is a considerable variability in the efficiency of inspections that I have seen. I've seen this both in the United States, and I've seen this at various other parts of the world.

So when we talk about using the history of the firm or the past inspection of the firm, whether it's a VAI, they get a VAI, I'm very concerned that unless we move to a pharmaceutical inspectorate that is more uniform and better trained in their capabilities, that we may not be using the proper metric when we talk about previous inspections as affecting the frequency of the oncoming inspection.

Now, I know, Joe may not agree with this fully, but this is certainly well within my experience.

MR. HOROWITZ: Yeah, I'll yield to Joe in a moment, but I think this goes back to Ken's comments earlier about one of the problems with getting a model like this off the ground is if you rely on historical data, but it's not static data, the pharmaceutical inspectorate and the approach to GMP inspections is changing, and I think that we have created a dedicated pharmaceutical inspectorate that will now be starting the coming fiscal year be operational.

And I do think that there are a number of aspects of the GMP initiative, including the creation of the pharmaceutical inspectorate that will gradually improve the coordination and the consistency of the observations that come about as a result of GMP inspections.

And what I expect is that over time the data on which we rely, the historical data on which we rely, will be increasingly reliable and increasingly valuable to feed back into the model.

But there's no doubt that we're dealing with some of these challenges right now.

DR. GOLD: But, Don, if we talk about a pharmaceutical inspectorate starting some time later this year or next year at the earliest, and we're talking about in implementing this model within a reasonable period of time I thought you're aiming at some time later this year to start introducing this model. How will we merge the two timetables?

MR. HOROWITZ: Right. Well, that's what I'm saying. The data we're using is based on the old model, and we all understand that there are certain problems with that, and that's why we're switching over to a pharmaceutical inspectorate model, and as a result, our data will not be as good as it could be and hopefully will be in that area.

I wish that were the only data shortcoming that we were dealing with right now, but it's certainly one of them that we'll have to keep an eye on.

Joe.

MR. FAMULARE: You know, just to speak to your concern about investigator's consistency and how that influences the model, you know, a lot of this initiative is to address those inconsistencies in not only the formation of the pharmaceutical inspectorate, but in doing the expert elicitation, you know, not only were reviewers called on, but folks in the Office of Compliance of CDER and those investigators that are predominantly, if not 100 percent, although there are fewer in number now than we would like, were called upon in terms of their experience with the expert elicitation.

So we tried to overcome as many of those mitigating factors -- and Brian could chime in on that. He's most familiar -- as there could be to get that consistency in there.

I think what folks have to think about and step back for a while is we're transforming from a system where we inspected or aimed to inspect every firm every two years that registered, and for years we have not been able to do that, but we didn't have a good working model as to who we should get to first, and it's going to take a while.

We've taken some rough cuts at this. Let's do all sterile. Let's do all Rx drugs, and let's do all new registrants. But this is taking it to the next logical step, and when we hear about this in other venues, probably the most common thing is police work. You know, they've done computer based policing and so forth.

I recently read an article about a Midwestern city now that just did this type of work on convenience store robberies, and actually it helped them to catch crooks because they put a pattern about it as opposed to just putting old marks that you saw in the old movies on a map where the crimes occurred.

And even in that same article, that same city, even incorporated an element of PAT. They put sound detectors to hear gunshots so that you could go nearest to where the gunshot is and figure out that's where the crime is going on.

So you know, these are not --

PARTICIPANT: Have you told that to Ajaz?

MR. HOROWITZ: Well, people are probably getting hungry hearing the reference to convenience stores.

(Laughter.)

MR. HOROWITZ: But you will feel --

MR. FAMULARE: But I think we have to put it in perspective, that we're now really trying to put together a model of figuring out who we're going to go to first and when, and even to go to the trouble that the professors had, Jackson and so forth in getting to those overseas companies. We have to pick and choose our shots overseas even more stringently because it's difficult also.

So this is the first very organized step we're going to take in doing so.

MR. HOROWITZ: The last question or comment on this and then we'll have to hit number three because I know people are eager to move on.

Garnet, please.

DR. PECK: This is for Number 2.

MR. HOROWITZ: Okay.

DR. PECK: You explained and defined various product types, and then you also comment on unit operations. But there is no explanation about what was done with the information or the knowledge base that was gathered, and I think for two it might be interesting to take a look at the processing and what unit operations are involved and see if there is some kind of correlation coming out of this, and it may be like the policeman, you know, spotting something that could be happening with a particular series of unit ops and analyze those.

So that's my thought for Question 2.

MR. HOROWITZ: Okay. Thank you.

Now Brian on the expert elicitation. do you want to respond to what you're planning on doing or have done with that date?

DR. TRAN: Yes. That's our plan, is to drill down and analyze the data at that level, but we haven't gotten that far yet. That's our intent.

MR. HOROWITZ: Okay. Let's look at Number 3 and then depending on the discretion of the chair, there will be additional time for questions.

But this is just specifically whether there might be some metrics we ought to consider. Process capabilities come up. SPK is one measure that is talked about a lot.

If we could build in any more objective data into the system obviously we want to do it, particularly if it could be widely understood and accepted. Any thoughts on that? Any metrics of process control, which is really the heart of what we're looking to focus on for the GMP program, that we might include? Any thoughts on that?

DR. FACKLER: I'm not sure where you would get this data or if this is really an answer to this question, but facilities that have a high turnover in personnel are clearly going to be -- I shouldn't say "clearly" -- might be more at risk than facilities where you have a stable set of employees, and I don't know how you would necessarily get that data without going there and asking the question, but to me it might be a factor.

MR. HOROWITZ: Well, we may hear more also from the Nickerson and Macher study to identify some objective measures and things like that.

The other thing is some of the data we could go out and determine on inspections and add to our databases routinely. So one interesting idea that I heard would be one measure might be look at the percentage of the root cause investigations that actually get to the root cause versus the cause is undetermined. That might be an interesting surrogate for a process understanding.

That's not data we currently have in our system, but in theory that might be something we could collect. You know, there's limited resources, but if we could figure out a few good ones perhaps like that, perhaps like something else, we could improve our databases.

You know, overall I think it's just the process of beginning to think critically about these things that's very valuable for us, perhaps even more valuable than the actual reordering of the sites. And we're eager to engage in more dialogue like this to get on the same page.

So at the discretion of the chair.

CHAIRPERSON BOEHLERT: I think we could take just a few minutes if there are some burning questions. I know we probably all had questions as the speakers presented, the last four speakers. So, Gerry, did you have something?

MR. MIGLIACCIO: Yeah. I mean, is this going to be transparent? Will sites understand how they're ranked?

MR. HOROWITZ: You know, that's one of the hardest questions because, you know, we want enough transparency to get valuable feedback and input, and we want to create incentives, of course, and be transparent enough to do that. In that sense, we'd like to be able to reward sites that are doing it right.

But we can't obviously make it so transparent so that anyone could run our model and they'd know exactly where FDA is going to be at any moment because there's certain regulatory problems associated with that.

Particularly given our limited resources, there has to be a perception of greater coverage than we're actually able to achieve.

(Laughter.)

MR. MIGLIACCIO: I understand that, but this whole initiative is about both FDA and industry putting their resources in the highest risk areas. So if we from a corporate perspective understand what you consider high risk, that helps us to understand where we need to put our resources.

MR. HOROWITZ: I complete agree.

CHAIRPERSON BOEHLERT: Okay. Ken.

DR. MORRIS: Just a real brief comment. Would that not just be served by knowing what the criteria are rather than knowing the ranking though, Gerry?

MR. HOROWITZ: Yeah.

MR. MIGLIACCIO: With the number of facilities that we have that are FDA approved, I would like to understand how the FDA has ranked them. I think we may rank them somewhat differently.

MR. HOROWITZ: Yeah, I think the challenge is for us to provide enough information so that we can be transparent about the things that we think are the riskiest and the risk factors so that we can have good dialogue about that, but also so that industry can focus on this.

CHAIRPERSON BOEHLERT: Any additional questions, comments?

Pat.

DR. DeLUCA: Yeah, I'm kind of old enough to go back about 30 years, even predate your slide there with inspections in registered firms, but it seems that some of the questions that are being asked were asked then, and I don't see anything in reference here to a concept that 30 years ago was called self-inspection, and I don't see that mentioned at all in these deliberations.

And I'm wondering if this isn't something that should be incorporated into this together information that would allow you to prioritized, where the industry would have actually self-inspection programs.

MR. HOROWITZ: Gerry, do you want to talk about the first party audit program or address that question?

MR. FAMULARE: Well, you need to address it in two ways. I mean, there was a major effort to announce a first party audit program some years ago where we were promoting self-inspections on how the agency could benefit from those self-inspections to change or mitigate the amount of inspections we need to do.

It fell on a number of complications, even looking at how some of our sister agencies wound up, such as OSHA, where they told them they had to go and do a rulemaking, and we were bound by current regulations and so forth, where we weren't about to be able to offer a definitive no inspection, no warning letter, no whatever under the act.

It was a little bit easier in EPA's case because they could mitigate certain amounts of fines and so forth. So we went off that path onto the systems based inspection path to put focus on the proper places in the inspection.

Further than that, one of the elements in the September announcement will be a corollary guidance to the GMPs to try and emphasize modern elements of quality systems, and that for sure will be one of the areas of emphasis. You know, it's an area where we've always not looked particularly so you could be frank with yourself, but on the other hand, how could you translate that information to FDA in such a way that you didn't mess up the frankness of your self-audit or prejudice that, but again, be able to get some benefit from FDA that we need less scrutiny or less scrutiny in these areas from our self-inspection.

So there's certainly been a lot of thought in the various circles around this particular effort.

MR. HOROWITZ: If I could just follow up on that briefly, we completely agree that self-inspections are a crucial part of an effective quality system, and we want to create incentives for firms to do self-inspections.

We haven't been able at this point to capture how you would feed that directly into the model specifically. For example, if we went out and asked them did you do a self-inspection, you know, everyone would just say yes, and really the key is not just whether you did one, but did you do it right, did you do it well, and we don't want to be in the position of grading their self-inspections because it's been our longstanding policy that we don't generally ask to see your internal audits because we want to encourage you to do them and find whatever is buried in the closet and to be frank with yourselves about that.

So there's a real challenge for how to tap into that, and I hope that through the quality systems enhancement guidance and perhaps even through Q10 one day we can create more incentives and guidance to encourage exactly the kinds of self-inspection activities that we want to encourage.

CHAIRPERSON BOEHLERT: Any last comments before we break for lunch?

DR. SINGPURWALLA: I do have lots of comments, but I think I don't want to take up lunchtime. I'm wondering if there's a later opportunity.

CHAIRPERSON BOEHLERT: yes.

DR. SINGPURWALLA: My comments are technical.

CHAIRPERSON BOEHLERT: Okay. There may very well be. It seems to me this was a topic we could have spent the whole morning on. It has elicited a lot of discussion from the committee, and I'm sure we'll be seeing it again at a future meeting.

So thank you all for your participation.

MR. HOROWITZ: Thank you all very much.

(Applause.)

CHAIRPERSON BOEHLERT: We will reconvene at one o'clock.

(Where upon, at 12:07 p.m., the meeting was recessed for lunch, to reconvene at 1:00 p.m., the same day.)

AFTERNOON SESSION

(1:02 p.m.)

CHAIRPERSON BOEHLERT: Well, we're all here. So I think we can get started.

One issue that I would like to raise with the committee is we have a number of presentations this afternoon, and some of them may also elicit a fair amount of discussion. It's your choice if you want to take a break or not, and just work our way through and perhaps get out 15 minutes early or perhaps, you know, we'll use that time for additional discussion.

Is there any feeling one way or the other on the committee? Raise your hand if you don't want to break.

PARTICIPANT: As long as you can leave at will.

CHAIRPERSON BOEHLERT: You can leave at will. Is that all right if we don't have a break? Skip the break okay?

Skip the break. Okay. We will skip the break, you know, but feel free to get up if the need arises.

Okay. This afternoon we're going to change gears and begin with a presentation by Moheb Nasr on GMPs for the production of Phase 1 IND drugs.

DR. NASR: Good afternoon. I hope you enjoyed your lunch and you are ready for some GMP.

My presentation this afternoon will be very brief. It's intended only -- and I underline "only" -- to provide a very brief background of some of the CMC requirement for Phase 1 IND. I will not discuss the guidance issue. I participated very little in the guidance development. Joe Famulare will address the guidance, and he will take all of the questions and all of the blame and some of the credit later on.

Okay. The primary objective of INDs as most of you know, but maybe many or everyone doesn't know everything, in three phases of drug development, and the focus of IND for Phase 1 is the safety issue. The focus is on safety.

It's basically the first introduction of a new drug into humans. It's intended to conduct some studies and evaluation of pharmacological action of drugs, potential side effects, predict and evaluate early evidences of effectiveness and so forth.

Phase 2, it's limited work control, and then you expand it into Phase 3.

We have some regulations. Some of you are becoming more familiar with these numbers, and we'll throw more numbers at you today, 21 CFR 312, and that's where many of these issues are outlined in our regulation.

As far as CMC requirement, and that's why I'm speaking this afternoon, is to indicate the following or share this important message: that the amount of information needed in the filing depends on the stage of the drug development. For Phase 1 INDs, the amount of information needed depends on where we are with the study, the drug itself, some previous studies, dosage for, route of administration, duration of the study, the patient population, and if we know of some known risks.

All of these things will determine the amount of CMC information that needs to be filed at Phase 1.

Talk about the drug substance, there are several attributes and several quality attributes and information that need to be submitted, and it varies from drug to drug, from study to study, but in general, we need some description and some identification of the drug, how it is being made and prepared, the analytical methods that are used for characterization and/or assay, and a brief description of a stability study, if any, at that stage just to assure that the drug would be stable through that period of clinical trial.

For the drug product, we need to know the components of the drug product, some quantitative description, the formulation, who's making it, where are they, the method of manufacture, schematic description is sufficient at this time. We are not asking for extensive batch records or anything like that, analytical methods, and some information to assure that the product is stable during the planned clinical study. Some information about the placebo as well.

What's important here and if we're talking especially in the new paradigm where the agency work was sponsors as partners in drug development, is the degree and frequent communication between the agency and the sponsors. And as some of you were here yesterday afternoon when I talked about our efforts to reduce the number of supplements and the number of our review cycles to save resources, these resources in my mind should be allocated to facilitate such interaction. That's where we are coming from.

We are not trying to cut the resources from people who are doing the work now. We are trying to better utilize our resources to focus on communicating early and more often with the sponsors to address all of the issues.

This communication and interaction that takes place takes place prior to the IND. There is a pre-IND meeting, and generally the focus of that meeting is twofold. One is safety issues, and one is to look at the potential of any clinical hold issues when I'm sure that the clinical study continue on, and if there is any potential that would raise issues that may end up working the clinical study of hold. We try to identify these issues early on in order to avoid stoppage of the clinical study.

The end of Phase 2 meeting is very important, and that's where more CMC specific issues are raised. Pre-IND meeting generally focuses on filing and format issues, and there are follow-up meetings and teleconferences, fax and so forth.

What I'm saying is here, even though I'm just giving a brief introduction to you, that if you look at this slide, there will be more communication, but the frequency of communication is not as important as the quality and the nature of communication, and that will be coming soon.

Safety concerns. When we say that for Phase 1 IND, the CMC focuses on safety. Our intention is to make sure through the information we have there is an assurance of the identity, the strength of the quality and the purity of the IND drug that's being used as related to safety.

For example, how the product is made, what are the impurities that could be there, that may have been back from safety, the sterility concerns, stability concerns. Profiles need to be sufficiently refined.

We are not talking here at this stage about setting the specification or optimizing the preparation of manufacturing and proper characterization of the drug as well, and that's all I have. thank you very much.

CHAIRPERSON BOEHLERT: Thank you, Mobeb, and then, Joe, you're next.

MR. FAMULARE: Thank you, Moheb.

And now to get into the issue here with discussing these Phase 1 INDs that Moheb well introduced. I want to give you a little background as to why we're looking at the Phase 1 of the INDs.

First of all, the Food and Drug Act, 501(a)(2)(B), requires all products to be manufactured in accordance with current good manufacturing practice, cGMPs, and in '78 of course, we published the current version of for dosage forms the good manufacturing practice regulations, but they are primarily directed towards the commercial manufacturing of approved and even drugs without approval, drugs and biologics.

And the preamble said that the cGMP regulations are applicable to the preparation of any drug product for administration to humans or animals, and that "any" of course is very broad and indicated FDA's intent to public additional regulations specific to investigation of clinical studies.

Well, we never did publish those specific regulations and over the years there was a number of questions as to what is particularly applicable for Phase 1, Phase 2, Phase 3 clinical trials. Methods are invalidated. A lot of things aren't set. You're very much learning about the process, although particularly as Moheb said in Phase 1, what you're particularly learning about is safety is very much the emphasis.

And actually if you look at that quality paradigm that a number of presenters have gone into here, we're really shifting it all on one side in terms of the safety side, in terms of Phase 1.

At any rate, the agency had come out in 1991 with the guideline for preparation of new drug products, but it did not adequately cover all of the various manufacturing situations you might encounter in clinical trials and really did not fully address the expectation that an incremental approach to cGMP compliance is acceptable for investigational products, given where you are in that stage.

And of course, that opened up a lot of questions and concerns. And just to go back to Ajaz had a presentation on FDA's critical path initiative. In looking at what are the number of new molecular entities and treatments that are being approved, and it was disturbing that those numbers were going down.

And, again, if there's one take-away from the initiative, the cGMP initiative or quality initiative, as we like to refer to it as well, is that we want to be at the forefront of innovating and allowing these things to occur.

So, therefore, many of the concerns, particularly with Phase 1 INDs, and what I'll be talking about is microdose and screening INDs, these very early Phase 1 studies, there was inhibition because of the perception of what part or does all of the cGMPs apply.

So what we have done is -- we haven't done it yet because it hasn't been published, but what we are doing is drafting a guidance about Phase 1 INDs and a complementing regulation to articulate FDA's intent to implement an incremental approach to cGMP compliance for clinical investigational products, recognizing that some controls and the extent of controls obviously differ between investigational and commercial manufacturing, as well as the various phases of clinical studies.

And we've had a cross-agency work group with CDER, CBER, and ORA, and I'm just one member of the group. In fact, that group is meeting right now as we're speaking. So I hope they don't change too much of what I'm saying here today.

But when I say "cross-agency," it's not only been the GMP folks that have been meeting. It has been the review folks on both the CDER and CBER side, and one of the purposes of having Moheb explain the IND CMC requirements is that there's a lot of complementary work that goes on here in terms of the folks on the review side see some of these issues as they come in for the IND and so forth.

And the other thing is to realize that we don't have a regular inspection program for investigating or doing inspections of clinical studies. Things are looked at on a for cause basis there.

So we wanted to develop a guidance and an approach which would be, of course, risk based. How could we not be these days? But obviously not to overuse the term, we wanted to have obviously -- use the available knowledge, and we've had a lot of discussion about how knowledge is transferrable. You know from other studies and other trial batches that you've done some knowledge. Take that forward, utilize that, and as I said, in terms of the quality paradigm here the emphasis is very much here on safety. So it's off balance.

And there's a number of examples of that quality paradigm. Just think of all of them except Gary's yesterday which was blank.

And we're talking about, you know, what are some of the general cGMP requirements? The thing that I spoke about earlier is in terms of Phase 1 this guidance will apply to investigation of new drug and biological drug products during Phase 1 clinical studies. So this guidance that we're planning to publish and we hope to publish it for the September rollout of the GMP initiative will address Phase 1 clinical studies.

Along with this guidance we hope to issue a rulemaking pulling out Phase 1 from 210 and 211 so that there will be no lack of clarity, does it apply or not, and what we will do is regulate directly off the statute, 501(a)(2)(B), as I mentioned earlier.

Dan can relate to that because that's what we do with APIs, but this guidance will talk about our expectations, and we will specifically address Phase 1 studies designed to assess tolerability or feasibility for further drug development work.

Excluded are drug metabolism studies, structure activity relationships and food interaction studies. The important thing is that we want to provide direction for special product situations: microdose type studies, and when you factor in other complicating things, such as multi-product, multi-lot situations, and specific product types.

And we ran into a lot of these specific product types. We actually were going to start out doing this draft guidance even less than Phase 1, just sticking to these microdose type issues, but realizing that many trials in the biologic realm really start out more broader in the scope of Phase 1. So, therefore, we took all of these situations into account.

And as I said earlier, this is going to be a companion to other guidance describing CMC information submitted in Phase 1 INDs, and will complement what was said in the ICH 17A document about clinical production of API materials.

We're going to discuss in this guidance when it's released as a draft appropriate quality control standards, well defined procedures, adequately controlled equipment and accurate recording of data appropriate to this level of production. That's the key to remember as I'm talking about this.

So take away your thoughts of general cGMPs, 210, 211. We're trying to scale it according to the scope of these operations.

An application that will lead to implementation of cGMPs which is really consistent with good scientific methods because while some of this takes place in R&D facilities of established firms, some of this is taking place very often in laboratory settings. So we're trying to make a correlation between cGMP here and, again, what would be a good scientific method to do these studies.

It's going to talk about the use of available technology and resources to facilitate product development, cGMP compliance, and lessen cGMP burdens where it's very practical to do so, and it will talk about disposable equipment and process aids, using prepackaged materials, such as WFI, and contract manufacturing and testing facilities where it's appropriate.

There will be discussion of the prevention of contamination and cross-contamination and evaluate potential hazards regarding the production environment and obviously carry over materials from previous operation being removed.

So very, very rudimentary issues we want to talk about, and all of this is very rudimentary material, but again, it focuses on what we see as essential for a good clinical study, factoring off commercial manufacturing.

Personnel would have the education, experience and training to do their assigned functions. In terms of the quality control function, it should be established for every producer of IND products have responsibilities documented in writing, including the examination of components, containers, closures, in-process materials, packaging and labeling materials, review and approval of production and testing procedures, acceptance criteria, review of completed production batch records for release or rejection of each clinical batch.

Talking about the responsibility of staff involved in the production and in operations with limited staff, QC function may be carried out with the same person performing production with possibly periodic review by another qualified person.

Facilities have to have adequate work areas for their tasks, appropriate source water, and air handling and to cover any possible contamination or cross-contamination issues.

Very basic information on equipment being in working condition, calibrated and not additive or absorbative to the test material.

Be able to have control over components, acceptance criteria, use of certificate of analysis, and enough documentation for trace back of what that material was by lot number, et cetera, and supplier.

Enough production information so that the laboratory and production data and equipment used and changes in microbial controls have been covered, and the theme is to remember so that if you need to go back to this information you can. Again, good scientific methods. Nothing earth shaking here.

Laboratory controls such that test are conducted using established written procedures under controlled conditions and using scientifically sound analytical procedures, calibrated equipment, and be able to initiate stability studies to support use of the product during the length of the investigation similar to what Moheb would say.

Again, we're not talking about method validation or anything beyond here; just very rudimentary information and documentation that's needed.

In terms of the container closure and labeling, to make sure that proper packaging is used to protect the product from alteration or contamination throughout storage, handling, and shipping, and of course, the importance of preventing or precluding label mix-ups.

And distribution should describe the transport of the IND product from the point of production to obviously eventual use by the patient.

Record keeping should cover these general areas of equipment maintenance, production, distribution, QC functions, and again, component records. Really the basic rudimentary things you'd need to do to reproduce these issues if this is going to become a viable test article and go further.

And we propose here a two-year retention period for the records after approval of the marketing application or if in the case it doesn't get that far at least after shipment and delivery of the last product.

Given those general GMP requirements, we realized that there are special production situations and actually the screening and microdose INDs where maybe just one person, one dosage is where we really started this, because this is where there is a lot of throughput to develop. Where is there going to be a candidate that will go further?

And, again, with a concern from institutions such as the National Cancer Institute and so forth, and the concerns of liability under the whole rubric of 210 and 211, we wanted to set out these clear but important issues that have to be covered and separate away the issues that need not be of concern and certainly not be an obstacle to going ahead with these studies and find the new discoveries that are needed.

And, again, we go beyond that. Like I say, what our initial charge was with the screening and microdose INDs to cover various situations in Phase 1, such as multi-product facilities and the need of controls there, the special situations that biologic and biotech products pose, and of course, the needs and the importance, the safety aspects associated with sterile and aseptically processed products.

The application of GMP controls to screening IND in microdose studies should be proportional to the scale and scope of the operation, and special provisions for lab scale production are provided in the guidance with respect to the facility, equipment, and laboratory control.

So it's even drilled down a little bit more to more rudimentary elements for these areas.

In multi-product facilities, the emphasis is that of an area a room is used for multiple products, that one product at a time is produced in a given area, and that there be appropriate cleaning and change-over procedures to prevent carryover of materials, of contamination, or actual product mix-ups.

For biotech and biological products, additional safeguards are discussed or planned to be discussed in this draft guidance where some production systems may warrant that, particularly sometimes to protect even the personnel involved, pathogenic microorganisms, spore forming microorganisms, live viral vaccines and gene therapy vectors.

You know, equipment qualification and controls in production should assure the success of unit operations with safety related functions, and again, with these type of products, there's concern for viral clearance, virus toxin attenuation and pasteurization. So all of these issues are touched upon in the guidance for these special situations.

Retain samples, offer an opportunity to go back and look to compare the assurance of the product throughout the clinical development, and in process testing and detailed records where necessary insure for Phase 1 products, you know, that you end up producing multiple lots. So this is where we're starting to scale up now. You're going to need a little bit more detail when you start getting into multiple lots.

Of course, for sterile, aseptically produced products, you know, we thought about actually going to some references, such as USP and so forth as to there's obviously a lot known about that, but on the other hand, you actually listed some rudimentary bullets in the guidance that are planned now in terms of having personnel trained in aseptic techniques, using a proper laminar flow hood and controlling the environment.

And that's pretty much where it ends, and to wrap up on that last slide, the reason we didn't use some of the reference is because many of them, again, are rooted in commercial manufacture, and we were afraid we would put folks right back where they were.

So basically, to sum up, this guidance and this technical change to the regulation to put Phase 1 IND studies under the rubric of 501(a)(2)(B) and taking it away from the general GMPs should facilitate a lot of the initiatives and the critical passion initiative where we're trying to go to not be an obstacle to new discoveries; have clear expectations of FDA of where you need to be at at this type of a study; and provide that pathway.

Once we get through this process, we'll have obviously the draft guidance will be open for comments. The next thing that we'll need to address is clearer guidance, you know, stepping it up again because we emphasize the step-wise approach for Phase 2 and Phase 3 studies. So that will be a later part of our work.

Thank you very much.

(Applause.)

MR. FAMULARE: Questions later?

CHAIRPERSON BOEHLERT: No, we'll take questions now.

MR. FAMULARE: Oh.

CHAIRPERSON BOEHLERT: You know, any questions or comments for Joe and Moheb?

As you heard, the committee is meeting now. So it's our opportunity to have some input.

MR. PHILLIPS: I just have a few comments, observations. I think Moheb and you have framed the situation every well. I'm familiar with the March of '91 guidance that the agency issued, and it did, in fact, give a lot of regulatory relief for the production of clinical supplies, Phase 1, 2, 3.

Now, that's 13 years ago, and over that 13 years, I have personally been involved with many audiences in the States, Europe, Asia and interacted with groups who are involved in manufacturing clinical supplies.

I made two observations. Here we are 13 years down the road and there are still many people in that area who do not understand that that guidance even exists.

Secondly, for those who do understand that it exists, the R&D people always raise the issue that -- and I think Dan alluded to this yesterday -- the R&D people always allude to their interaction with their regulatory affairs counterparts, and the regulatory affairs counterparts always say, "Hey, we're looking at 210, 211, event though that guidance exists, let's be conservative and ratchet it up a little bit.

So with that as background, I think that you are making -- you, the agency -- are making a rational approach to taking the Phase 1 study out from under the 210, 211, and putting it under the legislative piece, and I defer to David to define this, but 501(a)(2)(D).

The other thing that we have to look at in my opinion is patient safety, maintain that safety, and I think in your proposal as you spelled it out, you have dealt with all of those issues. Many of these products are administered by the clinical pharmacologists as injections. If it's going to be an injection, it should be sterile.

You've dealt with that. Cross-contamination has been a traditional problem. When you don't know too much about the manufacturer perhaps, you've dealt with that. So I think you made a rational approach in moving in this direction. I would support it.

That's my comment. Thank you.

MR. FAMULARE: Thanks, Joe.

CHAIRPERSON BOEHLERT: Thank you, Joe.

Dan.

DR. GOLD: Yes, Joe, a couple of questions. Number one, if I recall the guidance that is in effect or has been in effect, it requires written procedures for the manufacture of the drug product, drug substance and the drug product, even at Phase 1. Is that correct, Joe?

MR. FAMULARE: You're talking about the '91 guidance?

DR. GOLD: Yes.

MR. FAMULARE: I'd have to go back and look at that right now.

DR. GOLD: I think it does.

MR. FAMULARE: Basically what we're trying to do now going forward is to have enough documentation to be able to repeat what you did.

DR. GOLD: Okay.

MR. FAMULARE: And that's the general direction.

DR. GOLD: This removes it. As I read it, this removes everything.

MR. FAMULARE: This would remove it out from under the rubric of that guidance.

DR. GOLD: Right.

MR. FAMULARE: That guidance is going.

DR. GOLD: I'm not objecting to that. I'm just -- okay? I just want to verify it.

MR. FAMULARE: The problem was with that guidance it went across Phases 1 through 3, and there's a big difference between Phase 3 and a Phase 1 screening IND.

DR. GOLD: You're absolutely correct, and it does not distinguish properly between the various phases, and that has been one of the problems.

MR. FAMULARE: Right.

DR. GOLD: One of the real problems.

The other issue that I see is missing here and I want to make certain it's deliberate is that there is no QA review or no quality unit review of the documentation of the procedures and so on. Is that a very deliberate approach by your group to remove those restrictions?

MR. FAMULARE: In terms of QA review of documentation and procedures, even in 210, 211, it's under the rubric of QC, and the QC review --

DR. GOLD: But QC -- okay, Joe. I equate QC and QA.

MR. FAMULARE: Right, but QC is discussed here and will be discussed in the guidance as a strong factor that you have to have QC, realizing that that QC could be very limited in a small lab setting. So we do call for that element of review. At least we're calling for that in the draft guidance.

DR. GOLD: Well, I saw some of that in here, but I did not see a QC or QA review of the documentation, and I just wanted to make certain that that's a very deliberate posture on your part.

MR. FAMULARE: No, I believe that is an element in the guidance that we're proposing.

CHAIRPERSON BOEHLERT: On page 7, the top slide in our handout, page 7, the top slide, under the second solid bullet, the second item, review and approval of production and testing procedures and acceptance criteria. Is that what you're looking --

DR. GOLD: Oh, yes, I'm sorry. The third bullet, review of completed production records.

CHAIRPERSON BOEHLERT: Yeah, right.

MR. FAMULARE: Right. Yeah, we did keep ‑- that's what I was saying, that we did. That is a factor there, right. Okay.

CHAIRPERSON BOEHLERT: Other questions or comments?

DR. PECK: Yes.

CHAIRPERSON BOEHLERT: Garnet.

DR. PECK: Under the distribution record or distribution section, it seems rather simple, and there's an element here of since it is Phase 1 that there is a group, a person, a clinician or whatever that's going to do this and not necessarily going directly to the patient.

Is there a need to kind of further define this?

MR. FAMULARE: Well, part of it is that this is corollary over the other 300 regs that go to test article accountability. So there was a good bit of coverage there. Our emphasis here was to make sure, for example if the product needs to be at a certain temperature that it's shipped at that temperature and maintains its quality from production to the actual patient in the clinic.

So, again, because of its complementary nature, we didn't go into certain details where we felt from the IND regs themselves. We also had corollary coverage from some of these issues.

DR. PECK: Thank you.

CHAIRPERSON BOEHLERT: You said you're going to look at Phase 2 and Phase 3 down the road.

MR. FAMULARE: Right.

CHAIRPERSON BOEHLERT: At what point in time are you going to do that because as soon as this issues, the question is going to be, well, then, what about Phase 2-3.

MR. FAMULARE: Well, Phase 2 and 3 will remain under 210, 211.

CHAIRPERSON BOEHLERT: Okay.

MR. FAMULARE: With what we would call appropriate discretion. Those things that don't apply do not apply, and so forth, but our subsequent guidance will clarify those issues, but we really saw this as the bottleneck in an area to start. The time schedule I won't even begin to discuss until after September.

CHAIRPERSON BOEHLERT: It sounds like it's very much later.

MR. FAMULARE: Well, I wouldn't say very much later, but you know, we'll get this draft, comments, get this done, and that will be the next step of the process.

CHAIRPERSON BOEHLERT: Other questions or comments?

Dan.

DR. GOLD: (Speaking from an unmiked location.)

MR. FAMULARE: Thank you, Dan, and when I say "thank you," I mean it's not for me. I'm only just one member of this group. We don't really have a head to this group, but we have a group of us working together on it. So myself, Chris Joneckis, Gurag Poocheekian, and there's a number of folks from CBER and one person out in the audience, Chiang, has been part of the group.

So, yeah, the group has really put their best heads together and experiences to work on that.

CHAIRPERSON BOEHLERT: Last chance. If not, thank you, Joe and Moheb. It looks like you have general support from the committee on this guidance.

Okay. Time to change gears again and look at applying manufacturing science and knowledge in a regulatory horizon when you talk about PAT. Chris Watts or Ajaz?

DR. HUSSAIN: As Chris comes to the podium, I'd just like to sort of give a context and sort of position the discussion we'll have with Chris on comparability protocol and so forth.

One of the aspects I've wanted to sort of point out with these presentations is that we're moving into a new paradigm. We're moving to the desired state, and not only will Chris provide you an update on what is happening in the PAT initiative itself, but also I requested him to emphasize a team approach to review and inspection, and that is the heart of the PAT initiative, is the team approach to doing business, and to emphasize how we are finding new ways of minimizing, say, the supplement process or minimizing the need to have a prior approval supplement as the only means of making decisions.

So I think there are elements of what Chris will talk about which will highlight this, and the second talk after Chris will be on comparability protocol, and it's a summary of all the comments we have received on the drug guidance that was discussed before this committee, and our current thinking. Steve Moore will make that presentation, and Moheb is working very closely with Steve to sort of move that guidance forward.

The struggle in that is I think we took a guidance which was being developed before we defined the desired state. That's the challenge, and I think we're trying to bring the desired state element into that guidance, and it has not been easy.

And I think one way, in my concluding remarks I think I would like to sort of say that, I think. Decisions that I think after this meeting you're making is that we will focus every effort from now on on the desired state and not really worry about the past.

and I think this is a sort of guidance which we are stuck in the middle looking at the old state versus the desired state, and we are struggling to sort of bring that forward, and I think we will come of that approach to say that we are focusing more on the desired state from now on and so forth.

So you'll see that struggle, and Jon Clark, who co-chairs, changes with our private approval supplement group with me under the GMP initiative, will share some thoughts on how we want to proceed.

So that's the context of the discussion this afternoon, and I hope that you'll continue the discussion that we had yesterday and keep giving us ideas and suggestions and so forth on how bet to sort of approach that.

Thanks.

DR. WATTS: Thank you, Ajaz.

I want to thank the committee for allowing me a few minutes of your time today to talk about what we've done and plan to do with PAT and really talk about primary this engine that we have at the agency, the way we refer to it. And I stole that term from Ajaz, "the engine for success," and I'm a firm believer that the team we've established within the agency, the reviewers, compliance officers, and the investigators from ORA, are really going to be the engine that drives the success of the PAT initiative within the agency. And that's really going to be the focus of how we manage review and inspection process for PAT as we move forward.

So just a very brief outline, and a few questions I'd intend to answer with my presentation. I do want to focus on the benefits of PAT and how there may be other approaches aside from supplements into implementing PAT for the industry.

So with that, a slide that many of you have seen on several occasions, probably one too many times for some of you. The definition that we came up with for PAT, and it was discussed at length at the PAT subcommittee of the Advisory Committee for Pharmaceutical Science, a system for designing, analyzing and controlling manufacturing through timely measurements of critical quality and performance attributes of raw and in process materials and processes, and I think the key here is this little three-letter word. Frequently that replaced with a two letter word that creates a lot of confusion. The two letter word is "or," and a lot of people read PAT as just process monitoring, and the control is frequently left out.

But I want to emphasize that we're really talking about a complete system for designing, analyzing, and controlling the manufacturing operation. When we talk about the analytical portion of PAT, process analytical technology, the focus tends to be on the analytical chemistry, and albeit that's an important part of what we're talking about with PAT, that alone is not the focus. When you see the term analytical and PAT, I'd like to have people think more along the lines of analytical thinking rather than just analytical chemistry. You have to consider not only the chemical, but the physical, the microbiological, the mathematical and risk analysis. All of that has to be considered in an integrated system rather than just focusing alone on the analytical chemistry.

So with that background and the definition of PAT, how does that link to what we've been talking about with process understanding? The term is floating, tossed around quite a bit. The focus is process understanding. It's really what we're focusing on with PAT, but what does that mean, you know, process understanding.

What we allied in the guidance was that a process is that a process is considered well understood when all critical courses of variability are identified and explained. That variability is managed by the process and product quality attributes can be accurately and reliably predicted.

I want to walk through a very quick example later on to give you specifically what I'm talking about with those accurate and reliable predictions, and we really feel the ultimate is that the accurate and reliable predictions reflect a high degree of process understanding, and of course, if a process is well understood, we'll assume that that then imposes a lower risk category when it comes to producing a poor quality product.

So with that, I do want to focus much of the discussion on the team, and I do want to emphasize that the initiative is cross several centers within the agency, the field, ORA, CDER and CVM, and you'll see the steering committee. These are the senior managers within the agency who are really pushing the direction that we're going with PAT or setting the course I should say, and you'll see ORA, the Center for Veterinary Medicine, and CDER, but you know, it's not just CDER, Joe. It's obviously from the Office of Compliance, Office of Biotechnology Products, which is whether Keith Webber is from. Frank is from the Office of Generic Drugs, and Moheb is, of course, from the Office of New Drug Chemistry.

So even though there's a lot of CDER representation, it is CDER-wide, biotechnology products, generic products, the new drug products, and of course, the Office of Compliance.

And I really want to highlight this team that we set in place that we're really going to manage the review and inspection process. These team members are from the field, from the center, from the Office of Compliance, from the different review divisions within Generic Drugs and Office of New Drug Chemistry, and they are what we refer to as the engine. This is the -- I think everything is the engine for success here, but these are the people who are going to be managing the review and inspection process, the interaction, if you will, with the industry.

And the training program that we went through, we first began with a team building exercise, and I think that was very important that we could all get together and just begin to open the communication channels with one another because it may not be all that often that people from the field communicate with people in the center, and just to break down those communication barriers and get more of a personal interaction with one another I think was very important.

And just briefly, the training session, we had two didactic sessions, one that began at the agency where we focused on several different technical aspects that we went through, that we felt were important background information for people who were going to be responsible for review and inspecting these facilities and these applications, and of course, we went through three practicus at the University of Washington, Purdue, and the University of Tennessee.

And there we actually focused hands on, if you will, on training to see what the industry may be looking at or what the industry is actually looking at in terms of implementing PAT.

So as far as the training program, we have completed the initial training program. We're currently doing a lessons learned, and I do want to emphasize that we have every intention and, quite frankly, we are moving forward with the continuing education effort because although in many aspects the initial training program was very successful, to think that we have covered all of the bases that we need to cover in terms of being sure this team is well prepared and stage prepared for what may come to us in the future, continuing education is going to continue to play an important role there.

So along those lines, we want to involve this team that we have in place right now in the next training for the people that we have coming around for the next round of training with the PAT team, and they were also heavily involved in the guidance finalization process, finalizing the PAT guidance, the team from ORA, you know, again, the Center for Veterinary Medicine, Center for Drugs, were heavily involved in reviewing the draft guidance, the comments that came in, the public comments that were submitted to the docket, and the process as far as finalizing the draft guidance that we're going to issue.

What I really want to focus on is this team approach to review and inspection, and I can't emphasize enough that it really is a two-way street. A lot of people see it, and they think that the people who are in the center and review the applications are going to have some input into the inspection process.

While that is very true, there's also the other direction of the Street. The people who are responsible for the inspection process will also have some input as to what is said about the review of an application or a supplement, if you will, that may come into the agency.

So we've all heard about the 1,700 some odd supplements that the Office of New Drug Chemistry gets on an annual bassi, and this is, indeed, one route for implementing PAT within your company,b ut I want to highlight two other options or alternatives, if you will, for going forward with PAT implementation, and these are in the draft guidance, and one of these is that you can implement under the facilities or the company's own quality system, and following implementation within the company's own quality system, an inspection by the PAT team or the PAT certified investigator may follow if the team deems it's necessary.

Another option following na inspection, the FDA certified or the PAT train and certify an investigator, can approve this process or the team as a whole can approve this process.

And I really want to highlight that outside of supplements or submissions such as a comparability protocol, there are other avenues for implementing PAT within a specific company or organization, and these are only a couple that we chose to highlight within the guidance. There are many other options that a company may have if they want to come forward and say that this is the approach that we think is appropriate for what we're trying to do here. We want to just stick it in our annual report. You can inspect it when you get here if you feel it's necessary.

There are many other options that a company can consider rather than coming forward with the supplement or comparability protocol, and I really just wanted to get that point across because the team as a unit will manage this when the inspection is taking place or when the review of a supplement or application is taking place. It will be the entire team that's responsible there. So it's not just a submission that has to be made to get approval to implement PAT within your organization.

So a very quick example. I want to walk through a quick example of how regulatory relief may come.

This is an existing title production process, if you will, the typical raw material dispensing, blending. You're going to mail after blending. I'll blend it again. Typically you're going to include your lubricant there and then go straight to compression. This is a direct compression process, and typically of the tests that are done, the dissolution and content uniformity tests are done at the compression stage.

And we've heard many times this tends to be in product focused or the testing to document quality phase, if you will. So if we think in terms of the PAT approach, if you think about that example of the process that I gave you, the PAT approach, if we want to focus again, the emphasis there is focus on the process understanding. What parameters are critical to the quality of this product? How do they affect quality or why do they affect the quality of this product?

That begins to get us down the road of answering those questions. We begin to understand how and why this impacts our process. So we get that understanding. This can be done, just one example, experimental design, and then how do we analyze these parameters. We're talking about on line analysis with PAT. How do we analyze these parameters? Remember the definition for PAT, design, analysis, and control. Once we pick what we feel is the simplest -- and I always emphasis to keep it simple -- the simplest technology, not necessarily the most expensive or newest out there -- the simplest form that's going to allow me to analyze and control the same parameters and design analysis and control. We implement our control strategy.

That's it. If we're focusing on process understanding and we think about the definition of PAT, design analysis and control, how do we control this process?

So the example that I gave you, and again, hypothetical example, if we do an experimental design and we see that the level of disintegrate and the particle size of the active are the critical attribute when it comes to meeting my desired product quality attributes that I'm looking for in the table that I produce.

For example, if it's you know, a pain reliever, you want your relief right away. You don't want to have to wait, you know, an hour or two hours to get relief from your headache. You want the product quality attribute there. Us as consumers would say I want my relief immediately. I don't want to have to wait two hours for my headache to go away, for example.

So the critical attributes here are the disintegrant level and the particle size. So if we move forward to an example of a PAT approach, if particle size is critical, in order to analyze it and control it within the manufacturing process, we first have to begin to understand, well, what's going into the process.

If we understand the particle size distribution of our active is before we go into the process, then we can begin to tailor our process to control that particle size distribution.

So one example of this comes from AstroZeneca is as they're dispensing the material into their blender, for example, they're analyzing this material as they're feeding it into their blender. So they know what the particle size distribution is of this material before we even begin to blend.

So with that in mind, be can begin to control the blending operation. So if we have, for example, an analyzer on our blending operation, that's not only going to tell us when we reach a homogeneous mix because remember the other critical variable that we had was that we needed an even distribution of our disintegrant. It's going to cause our tablet to explode, if you will, when we take it, and we get the active ingredient available for absorption and relief right away.

So not only can we control the disintegrant mix, but we can also be looking at the particle size distribution as we're going through, and this will allow us to begin to build some of those predictive models that will allow us to feed forward into this is the particle size coming in. This is my particle size while I'm blending.

So if you think of the initial process that we had, the raw material operation, blending, milling, and blending, if I know my particle size distribution coming in, I'm blending. I know what my particle size distribution is coming out of my blender. I may not need to blend every single time. I may have the particle size distribution that I'm looking for at this stage.

And we don't want this process to be frozen in time, if you will. If you don't need to mill, you already have the particle size distribution that you're looking for. Skip that milling stage. go directly to blending your lubricant and move forward to compression because you've already met your desired particle size distribution. That milling stage adds no value whatsoever when it comes to meeting the desired product quality attributes of your product quality attributes of your product.

So if you think about the PAT process that we have now versus what we had with the original tablet production, we're beginning to understand what the distribution is, the particle size distribution of our material, the attributes of our raw material coming into the process.

We control as we're moving forward in this operation. We can begin to build predictive models. If we know what the particle size distribution is coming in and we know, for example, if we're right on the edge of the distribution that we need, that's critical for us to meet our desired product quality attributes, we may be able to blend for just a little bit longer and meet that particle size distribution so that we don't have to go forward with the milling step. We can skip that milling step altogether and improve our efficiency, right?

So these predictive models will tell us, all right, if I have this given particle size distribution, I can predict that I'm going to stop my blender at Time X. And while I'm doing my blending operation, my control strategy actually shuts down my blender at the time that I predicted. What is that? That's the process understanding. Remember the accurate and reliable predictions? That reflects a high degree of process understanding.

So if we can convey that in some way to the agency and say, you know, I understand my process. I know what particle size distribution I need, and this is how I control it with my process. If I need to mill it, I'm going to mill it. If I don't need to mill it, I'm not going to mill it, and I'm not going to send the supplement to you to tell you why I'm not milling it because you already know.

We do away with some of those 1,700 supplements that Moheb has to deal with on an annual basis.

So thinking about that example, how is PAT benefitting us here? We no longer have this laboratory determination of blend homogeneity if that is done or the particle size distribution. We're doing it. We're actually controlling it while we're manufacturing our product. We're blending it to an end point rather than to a specific time that we validated when we did our three validation batches.

We're milling only if we need to. If we don't need to mill it, skip it. I'm not going to do it this time. And This begins to open the door for us to real time release because we're assuming we're building in quality as we're manufacturing the product. We don't need to test it at the end whenever we get our tablets out of the compression or out of the tablet press. We don't need to test those every single time.

But when we do, if and when we do, we're actually validating that our process is under control, that the control strategy that we have in place is, indeed, functioning as it should.

Optimization, this allows us to optimize the blend time. If you think back, if we're only going for a specific period of time rather than till an endpoint, there's not really a lot of flexibility in that time point. So you can begin to optimize your blending operation to meet not only homogeneity, but maybe to meet that particle size distribution that you're looking for so you can avoid going through that non-value added milling step.

And, again, this would begin to build in these feet forward models for blend characterization because we have to begin thinking of the blending operation. What we have is not only an output. It's actually an input into the next unit operation that we have.

The material that we get from our blending can go into our milling operation or it may, indeed, be sufficient enough to go straight into our next blending stage and straight to the tablet press.

So how does this reduce the regulatory burden? Questions that we get all the time. The process is no longer, borrowing a phrase from the Wall Street Journal, it's no longer frozen in time. We actually have free rein to avoid that milling step if we have to.

No supplement for a process change. I don't need to mill. I'm not going to send a supplement to you that tells you I'm not going to mill. I need to blend for a little bit longer this time. I'm not going to send a supplement to you that tells you I need to blend for a little bit longer. You already have demonstrated that process understanding.

And a team approach. I really can't emphasize this enough. It's a team approach through review and inspection. So when the inspector shows up, they're on the same page was the reviewer who looked at your supplement, if one came in, or they have a resource that they can use while they're on site. They know people who may be on the team, who may be able to answer a technical question that they have about the process that you have in place.

And during that inspection that's your summary basis for approval. So with that, I hope I gave you really what we're talking about with process understanding and PAT. The inverse relationship between the level of process understanding and the risk of producing a poor quality product, if the process is well understood, there are obviously less restricted approaches to -- less restrictive regulatory approaches to manage change, and if we focus on process understanding, we can facilitate risk managed regulatory decisions and innovations.

And this can really lead to the several options for implementing. We no longer need to go through the submission or supplement process when it comes to making a change to our process. We've demonstrated that it's well understood. We know what the impact are and any changes that we make. So we can go ahead and move forward with those changes.

So I hope that was a good example to really emphasize what we're talking about with process understanding and PAT and how it may be a benefit to the industry.

Very briefly, where we're going with PAT, we are finalizing the guidance. I spoke to you very briefly about how the entire team was involved in that process -- Ajaz mentioned this at the last advisory committee meeting -- expanding the scope of PAT to include the Office of Biotechnology Products, and quite frankly, the reason OBP wasn't included int he draft guidance is OBP didn't exist when we were coming up with the draft guidance.

Continuing education and training of FDA staff, that's going to be, I think, the oil change, if you will, to the engine that's driving the success within the agency.

ASTN technical committee, Del Marlowe, the agency standards coordinator, spoke to you very briefly about that yesterday, and of course, research continues to play an important role with what we're doing in terms of developing the sound scientific basis to the policy that we develop and the training that we conduct within the agency.

So with that, I'm not going to take any more of your time, and I guess I'll turn it over to Steve or Judy.

CHAIRPERSON BOEHLERT: I would just ask if there are any committee members that have specific comments on the PAT presentation. Yes, G.K.

DR. RAJU: So, Chris, you're saying if you ‑-

DR. GOLD: May I ask a question?

CHAIRPERSON BOEHLERT: G.K. is first and then you can.

DR. GOLD: I'm sorry?

CHAIRPERSON BOEHLERT: G.K. got first and you're second.

DR. GOLD: Okay. I'll wait second.

DR. RAJU: So, Chris, you gave a really nice example. So if somebody actually independent of any bioequivalence and despite the SUPACK guidances and their categorization, I mean, exactly that submission to you without any connectivity back to the patient in terms of bioequivalence, that would be within your mandate to say it's okay without any supplements, within the mandate of the PAT group and the guidance?

DR. WATTS: Well, I don't want to say that it's --

DR. HUSSAIN: No, I think the context of the no supplement, the changes for the existing product right now, the changes in the specification, you have no option but to have a supplement process.

DR. RAJU: But if there is no change in specification; only the process.

DR. HUSSAIN: The way it is a quality submission commitment, it is a change. It is a change today. So what we're saying is that the team approach to review and inspection opens up new avenues for allowing some of this to happen, but that is only in the context of process understanding.

When that has been shared, and that goes to the design space that we discussed yesterday. So what it means is the design of experiment mark is actually based on our own lab data. If the design of experiment that Chris showed, the chart, we actually had the questions you're asking. I mean, those were the critical factors that affected resolution and so forth.

That's the knowledge base under which we can start moving in that direction.

DR. RAJU: So you still have to bring that in.

DR. HUSSAIN: Oh, yes, absolutely.

DR. RAJU: But you don't have to bring that in from a patient, inside a patient point of view. You can do that totally from the in vitro information.

DR. HUSSAIN: It will depend on exactly what your process understanding is, what is critical what is not critical. If it is critical enough for the patient, then the biostudies could be part of that.

CHAIRPERSON BOEHLERT: Okay. Dan.

DR. GOLD: How does your work related to the requirement for stratified sampling?

DR. WATTS: I think that's just an example, if you will, of assuring blend uniformity.

DR. GOLD: I'm sorry. I didn't hear you. Say again.

DR. WATTS: That's just an example of how you can assure blend uniformity. That's not the only way. There are many other options for assuring blend uniformity. That just happens to be one that was discussed and came forward with the PQRI.

DR. GOLD: So does this mean that if a firm goes this route they will not have to justify what would happen during interruptions, refilling, or change in hopper, for example, or taking samples during the changing of a hopper? Is that what I'm hearing?

DR. HUSSAIN: No, I think you're missing the point completely.

DR. GOLD: No, I don't think I'm missing the point. I'm trying to clarify the point.

DR. HUSSAIN: No, no, you are because you requested the stratified sampling, which is testing ten tablets in a stratified way. I think the risk of that is much higher than the risk what you're talking here because no in-process controls you. No controls on your incoming raw materials. You're making a decision on ten tablets, although in a stratified way

DR. WATTS: If you look at the definition of PAT, a system for designing, analyzing, and controlling. If you're just looking at tablets, there's no opportunity to control. It's too late. You've already made them. All right?

DR. GOLD: No, I fully appreciate the difference in technology. What I'm asking is from a compliance point of view, if we proceed this way, does this mean that a stratified sampling is not a requirement, a compliance requirement?

MR. FAMULARE: You know, we're talking here about a whole control system in real time release. So any sampling and testing that's done could only, as Chris described, validate the process. You've already done what you have had to do before you even get to stratified sampling.

So they're two completely different things. You know, it's apples --

DR. GOLD: So you mean we still would need to verify stratified; you are introducing a new product?

MR. FAMULARE: No. You could. You could. Let's say you came in with a brand new PAT application or you supplemented an existing one for your product specifications. Your release criteria could be based on the PAT controls, the fact that through these controls you've come out with the product that's meeting its desired quality specifications.

DR. HUSSAIN: The key here is this in the sense I think, for example, if you have a scenario where there is a risk factor of changing a hopper and potential segregation after that, in that case there's a different application. It could be an on-line assessment on every table. So instead of doing ten tablets, let you might be assessing thousands of tables.

I mean, so the sample size goes up dramatically of what you evaluate here. So the decision is not based on ten tables.

DR. MORRIS: Just a comment, and I guess the way I think of it is that you'd be doing the establishing of the criteria during development. so by the time you got to the level of implementing the process of understanding base to monitor and control, you would already know that the release specs based on the PAT approach would have been substantiated.

So if you have segregation in a hopper, you might need another sensor if you have a model that tells you that that is a critical control point to monitor, is the way I think about it. I don't know.

DR. GOLD: So that are you saying that when we introduce this we would still have to do those evaluations initially, for example, on changing hoppers.

DR. HUSSAIN: Well, I mean it's pure and simple product development studies. You have to do what you have to do.

DR. WATTS: You can't do a DOE without defining the extremes.

DR. HUSSAIN: Exactly.

DR. GOLD: All right.

CHAIRPERSON BOEHLERT: Any other questions or comments?

DR. SINGPURWALLA: Yeah. How did stratified sampling get into this picture?

DR. HUSSAIN: Don't bring that up. That's not the topic.

DR. SINGPURWALLA: No, no. Dan asked the question, and you know, I feel obliged to, you know, think about it. So how does stratified sampling get in this? Did you mention the word stratified sampling?

DR. WATTS: No.

(Laughter.)

DR. GOLD: No. I am bringing up stratified sampling because currently it's a requirement in the absence of PAT, is it not?

MR. FAMULARE: It's not a requirement.

DR. HUSSAIN: It's just one way of doing things. It's not a requirement.

MR. FAMULARE: It's a guidance. In fact, that guidance even borrows some of the language from the PAT guidance that this is just one way to go. You don't have to go this way.

DR. GOLD: Well, you can offer an alternative, but you still have to be able to prove that you have uniformity through the various changes that occur through the processing, correct, Joe?

MR. FAMULARE: You don't even have to go as far as that last statement.

DR. GOLD: Okay.

MR. FAMULARE: You want to have uniformity, period.

DR. GOLD: Yes.

MR. FAMULARE: In terms of changes, you know, it's one thing that you identify your critical control or weak points. It's another thing to have a deviation that was unexpected. So, I mean, the whole point of the blend uniformity, the stratified sampling or one of the main points was to take care of sampling bias. I mean, that wasn't focused on if you go back to that guidance, what are your weak points. It was really focused more on sampling bias and the limitations of that.

DR. SINGPURWALLA: Can I articulate on this? I think I see the point that Dan is raising and the presentation that you made. I hope I'm correct in articulating it.

I think what you are talking about is continuous monitoring and control, as done by control theorists.

DR. WATTS: Right.

DR. SINGPURWALLA: What Dan is talking about is when you do not have continuous monitoring and you do not have continuous coupling. You do sampling and to account for the biases, you may want to stratify.

And I think he is monitoring continuously. So from one point of view I would look at his presentation as something in control theory; is that correct?

DR. WATTS: Absolutely.

DR. SINGPURWALLA: It's process controlled, control theory, and somehow you threw in design of experiments because most chemists and chemical engineers and pharmacists like design of experiments. So somehow it's kept in.

(Laugher.)

DR. WATTS: This is the point, but just because you can't control something doesn't mean you have to. Moisture, for example, if it doesn't matter if I have between two and 20 percent, it doesn't affect the performance of this granulation in this process or the stability of the product. Why do I need to control it to 2.5 percent, for example?

DR. SINGPURWALLA: (Speaking from an unmiked location.)

DR. WATTS: To determine what's critical.

DR. SINGPURWALLA: Right to determine the critical points. Yeah, that's fair.

CHAIRPERSON BOEHLERT: Okay. Are we ready to move on?

DR. HUSSAIN: I think so, but I think this is an interesting challenge. You always keep going back to the past. I'm not looking to the past anymore for that. We need to come and talk about the new stuff before we let this --

DR. GOLD: Well, I'm very happy to talk about the new stuff. I'm just afraid that we may also be looking at some of the old stuff during the way, on the way.

CHAIRPERSON BOEHLERT: Next up will be Stephen Moore to talk about comparability of protocols.

DR. MOORE: Thank you. I'd like to give you an update on the comparability of protocols and an update on the progress of the guidances and the revisions of those guidances.

And just to cover today the general topics, definition and general aspects of the probability protocol, regulations that we have published on comparability protocols, the draft guidances that are in the works, and also talk about the public comments and give you some highlights there that we received in the docket, and spend most of the time on our current thinking.

A definition of a comparability protocol, it's a comprehensive detailed plan that describes the specific type of proposed change, the tests and studies that will be performed, analytical procedures that will be used, and the acceptance criteria that will be achieved for the purpose of demonstrating that a change -- that there is a lack of an adverse effect on the product quality for that change as it may relate to the safety and effectiveness of the drug product.

And I'd like to say that this is a basic definition of the comparability protocol that stems from the regulation, and a comparability protocol can be much more, as you'll see later.

A comparability protocol, some of the general aspects that should be well planned in advance. It should be scientifically and technically sound, that is, that is based upon knowledge and understanding, And I will discuss that in more detail in further slides, and it should be adequate and kept current to implement the change and comparability protocols are drug process controls and change specific.

This is the regulations that have been published on comparability protocols. Actually the regulation first came into effect in 1997 for biotechnology and biological products, and most recently in April is now in effect for a chemical entities.

And the regulations state that what must be in comparability protocol and in accordance with that definition that I just gave you, and it also says that a comparability protocol can be submitted in an original marketing application or it can be submitted as a prior approval supplement.

And it says that changes to the protocol have to be submitted as a prior approval supplement, and that FDA will review this protocol and if justified, can designate a reduced reporting category for that change under the protocol.

These are the draft guidances that are up on the Web. There's two of them. They are companion guidances, and the first one applies to the chemical entities, drugs and includes synthetic peptides drug products, and that one was put up in February of 2003.

The other one covers biological and biotechnology products, which went up a few months later.

The public comments are under review now in the comparability protocol working groups and for final publication of these guidances.

And I just wanted to give you some of the highlights of these guidances, and what I've done is excerpt this and paraphrased this for brevity to give you more or less what is the message we're hearing from the public comments.

And these I'll read off: the efficient use of comparability protocols should provide regulatory relief by expediting review and approval of post approval changes. And I think we all agree with that.

And many changes are not anticipated at the time of filing a marketing application. We are seeing mostly changes are comparability protocols filed in prior approval supplements. There have been some submissions in the original marketing applications.

And the commenters in the public documents say that the level of specificity requested, and they're talking about what was in the guidances, may define the protocol so narrowly as to diminish its future usefulness.

And here what we are taking this and what we're hearing is that protocols need to be made more flexible in order to be made more useful, and that the key to the use of a comparability protocol is the availability of sufficient manufacturing science data to demonstrate an adequate understanding of the control process controls and we can't agree more with that.

Continuation on the comments, they wanted us to clarify what we meant by a comparability protocol for changes of a repetitive nature. What we meant was that comparability protocol was for repetitive use or could be used repeatedly, and I think that's very important because this kind of protocol is very valuable. Once we approve it, a company can use it to make changes, and that regulatory relief that's granted initially can apply to changes into the future, and we won't have to go back and review their plans again.

And they asked us to provide examples for reduction in a reporting category from a prior approval supplement down to annual reporting. This we are working on, and I'll show you some more details later.

They also asked for modifications to a comparability protocol. Can we find ways to lower that into categories other than prior approval. As the regulations stated that those modifications would be for prior approval, but 31470 and others, the companion one for biologics also says that we can do this through guidances.

And another point, the cGMP aspects of post approval changes should be addressed and we are doing that.

Also, finally, we applaud the FDA for its efforts, and we do appreciate that feedback from the commenters to the public document.

And now I'm going to turn to the current thinking on comparability protocols. Essentially we see it as two basic kinds of protocols and this is from also built upon our experience of the kinds of protocols that have been submitted.

One kind is a single use comparability protocol, and these are designed to make a specific one time change. Usually these are for rather complicated changes.

And another type of protocols that I was talking about, the repetitive use comparability protocol, that is designed so it can be used to make a specified type of change and changes within that specified type can be made repeatedly and over time.

Some more aspects, details about single use comparability protocols that could cover a single change or multiple related changes, and we have seen examples of both.

And for multiple related changes, what we are finding is that there is not always a distinct discrimination about how they are going to evaluate those individual changes. So we in the guidance are going to make that clear, and that each of the individual changes should be clearly defied how they're going to assess them, and also the combined effect of all the changes if they're making multiple changes should be assessed.

And there are many, many examples of what single use comparability protocols could be used for. I mean, essentially they soul be for any changes in the drug substances, drug product manufacturing process.

And there are some exceptions, and I'll get into that later, about what might not be appropriate in a comparability protocol, and they can be for changes in scale and multiple related changes that are related to changes to scale, and this may also common occur at different facilities.

Aspects of a repetitive use comparability protocol. Generally these are more narrowly defined, and the concept here is these are modular in nature, and we find that boundaries need to be established so that we are certain that the comparability protocol remains valid over the type of change that is defined.

For example, if you had a change for differences in scale, you might want to set a boundary of half X to ten X. Well, inside that range you could be able to freely make those scale changes. Outside the protocol may not be valid, and we need to know that during the review process so that we'll be sure that we're looking at all that needs to be looked at.

And in general these multiple changes are usually comprised only of subcategories of the specified type of change, and I could explain that better by examples.

The classic case of a repetitive use protocol, and these have been used for a long time, are container closer system changes in which we have show equivalency of various container closure components.

And also we want to expand this idea to changes within a unit operation, and you may be able to change the conditions or the parameters of that step, and once that is approved during the protocol, you may have free use, the ability to change that without regulatory oversight.

And just briefly going over what the advantages are and disadvantages are, I think many of these are already apparent, and to industry the main advantage and the original intent of developing regulations and guidances for the comparability protocol is that that would help shorten the time length for distribution of product and reduce the filing burden for commonly made changes.

And so while you're waiting for FDA to approve, and now it's four months for a prior approval supplement, if we can get the plans approved ahead of time, you can make the change under a greatly reduced reporting category and burden.

And the disadvantages, of course, I mean in all cases the risk of an adverse effect is not eliminated, but we intend to say that the comparability protocol should be constructed in such a manner that if during the implementation of a change is found that there is an adverse effect, the protocol would be strong enough, rigorous enough to catch that and would stop the implementation.

The advantages or disadvantages to FDA. We're seeing, hopefully as being responsive, in finding ways to reduces manufacturer's down times is why they're waiting for a prior approval, and we are hoping that this many reduced the overall number of post approval supplements.

One advantage is that unless the protocols were remained in the original application, this is going to increase our work load of supplements because not all cases would we be able to downgrade the change to annual report, and I'll get into that later. It would be related to complexity of change and how much information is provided with the protocol.

So it's possible that I could increase our work unless those things are considered.

And what might be appropriate and what might be not appropriate under comparability protocol. We think it's appropriate under a comparability protocol that the lack of an adverse effect can be demonstrated by analysis of the product quality characteristics. We're talking about CMC here.

And not considered appropriate, nonspecific plan for CMC changes. We have had some protocols that were written apparently too far in advance that they did not know the details of that change or how that change was going to be evaluated.

Also not considered appropriate, if the comparability protocol would require pharm. tox studies, biopharmaceutic studies, other clinical safety or effectiveness studies to be done. And in those cases we would not be able to offer a downgrade, I am afraid.

And continuing with our current thinking on comparability protocols and some of the principles and recommendations we're trying to articulate in the guidance, that comparability protocol should be based on and provide evidence of scientific knowledge and technological knowledge and understanding of the drug. That includes the drug substances, the drug [product] and all of the materials that are used in its manufacturer, the manufacturing process, the controls, the proposed change itself, and what is the potential effect of that change on the product quality; and that this knowledge and understanding could have been gained through pharmaceutical development information pertaining to the drug and its manufacturing process.

And adding to that, commercial scale production experience would contribute, and one may be also able to cite scientific and technical and technical literature.

These are continuing with the principles and recommendations. In developing your comparability protocol, all of the potential effects of the change should be identified and not just the obvious. this is a Q5E concept that was rolled into this guidance.

And the pre and post change drugs should be compared for all changes. I'm speaking of the changes with a drug substance, then the comparison mainly resides there.

And for all the changes this has been a longstanding policy that we normally see in our supplemental applications.

And the combination of routine product quality control testing, supplemented with characterization studies as needed would be utilized, and the analytical procedures that are utilized should be sufficiently discriminatory due to potential differences in the pre and post change products.

And then an integrated analysis of all the available data surrounding the development of change and implementation of the change should be performed prior to concluding a lack of adverse effect of that change and perhaps implementing the change.

And then just a few words, and I won't belabor this. Demonstration of a lack of adverse effect because this is what the protocol was designed to do. This should, of course, be based upon such knowledge and understanding that we have been discussing.

And the product quality characteristics of the pre and post changed products should conform, of course, to their specifications, and the specifications would apply to all the materials, including drug substance, drug product that constitute the drug.

And not only that, but that such conformance of the acceptance criteria should also be made for the characterization studies, and that these data should be comparable with respect to the mean and deviation of previous product made by the current process and also applied to those types of characteristics that are expressed qualitatively.

And also we should consider the effect of the change on the manufacturing process and the process controls. Of course, the process controls will be met. In some cases you may even have to change the process controls, but essentially that would be the bottom line.

And the effect on the process controls as they relate to the product quality would be considered.

And now turning to how do we propose and how does the company propose and how does FDA justify designated a reduced reporting category, given the submission of a comparability protocol, and there are several factors that would be considered, and one factor, the foremost factor, the degree of the demonstrated knowledge and understanding of the product, the process, et cetera, et cetera that is provided with the protocol.

And of course, you need to consider what is the normal reporting category for that change, and that can be found in the regulations and our guidances, and that would be the starting point for the downgrade.

And also we considered the specific aspects of the drug, the process controls, the change would also be considered, for example, complexity of that process, complexity of the product as well. So it would be input into that.

But also I mean this can be tempered with knowledge and understanding in a complex product if it's well understood.

And then also the validity of the comparability protocol and some of the things associated with the validity is is it scientifically and technically sound.

And now getting into the plans on our current thinking, how do we get there, and these are the various categories of changes. Prior approval, CBE, CBE-30, and annual report that are specified in our guidances, specified in our regulations and our guidances, and so those are the starting points.

So you would have to know how your changes fit into this hierarchy originally, and then how can we get from prior approval down to annual report, and we believe that would be capable if a substantial knowledge and understanding is presented, that that is demonstrated with the comparability protocol submission.

And it could be in the submission. It could be referenced or cross-referenced off to the original NDA or other submissions to your marketing application that would allow us to go there and look.

And the use of the comparability protocol would substantially reduce the potential of an adverse effect on the product quality in that case, and this first category is beyond really what the regulations, I think, the original writers had intended. They had talked about a reduced reporting category, not talked about how do we get to prior approval. They leave it to us in guidances to figure this out. And with our current paradigm, this is what we believe.

The current state of affairs is more or less the second bullet, an intermediate or moderate reduction, and where an adequate knowledge and understanding would be provided in the protocol, but that would be differentiated from such substantial knowledge and understanding.

And the third category, we have not seen many of these kinds of protocols submitted where they're downgrading, asking for a downgrade to CBE-30 of CBE down to annual report because the comparability protocol itself takes a prior approval supplement.

I mean, this could be overcome if they were combined in a same submission. We have seen that in some occasions.

And now I want to talk in more detail about how to get from prior approval down to annual report and what is our current ideas where and preliminary comments on how do we get there.

Of course, I just talked about the substantial knowledge and the understanding of the drug, the process controls, the change and the potential effects of that change, and the relevance and the adequacy of the test studies and the analytical procedures to assess the effects of that change and may need to include preliminary data to support a lack of adverse effect.

And of course, the bottom line, FDA will look at this information and then determine whether it was sufficient to downgrade to annual report.

And more specific examples of ways in which we think you can get there, provided with the comparability protocol is data from pharmaceutical development studies, for example in a pharmaceutical development report. That would be included in the protocol. That will help in defining the change, identifying the critical process steps, parameters, variables, controls and interactions of variables, and if needed, data from pilot scale batches, and we know that this is typically done on the road to making a change; that we don't think that companies generally jump directly from the lab to full scale manufacturing. We're not trying this out first on pilot scale and then optimizing the situation.

And data from full scale production batches -- these might be initial batches -- if available, but not necessarily required.

There's other ways to get there. You might have data from a previous change made to a similar product or the same change made -- sorry -- similar changes to the product or the same change to a similar product.

There's other ways to comparability protocol. It might involve a two tiered downgrading, and I won't talk about that much.

There are some exceptions that are perceived that might get in the way, in our ability to down grade to annual report., the change may be too complex. Of course, I talked about very -- complex changes, changes that require pharm. tox input, biopharm, or clinical input.

There may be changes in which the impurity profile is changed, and that will also translate to a change in the need for specifications. These may be possible impediments on the road to annual reports, and we are still discussing that within the OPS.

The commoners in the docket asked us how can we modify comparability protocol in ways that are other than prior approval, and we're thinking about that, and I wanted to give you some specific examples.

We see the need for that, that they may need to modify the acceptance criteria. They may have actually missed the mark in determining what those are in implementing change, and they may need to modify the change itself in order to get it back within the desired target. Changing the change.

And, of course, over time, a comparability protocol could become obsolete. There may be new scientific advances. There may be safety issues that arise, and the comparability protocol needs to be kept current and valid. So we don't want to impede manufacturers in keeping their comparability protocols current.

And we're trying to identify examples, specific examples in which modifications could occur to a comparability protocol in all of the different categories of the FDAMA categories.

And I just want to summarize up. The comparability protocol can be useful to industry to shorten the time line for distribution of drug products, and FDA is exploring ways to make protocols more useful and flexible, and we believe that substantial regulatory relief can be granted through this road or avenue of using a comparability protocol, provided that an applicant demonstrates a substantial understanding of their product and their process.

CHAIRPERSON BOEHLERT: Thank you, Dr. Moore. Any questions or comments? Moheb?

DR. NASR: If you'll allow me, I would like to make some general comments and statements. First, I would like to thank Steve and the working group. You have been working very, very hard, and very diligently, trying to get this document out. Because they understand the need of such a document, and its potential ability to facilitate submissions and so forth. The document is not out yet, and it's not because of Steve. I am the one to blame. So if you have any problem or an issue about the document not being out, please don't put the blame on Steve and his working group, because they are working very hard.

I am holding the document for a variety of reasons, and I would like to share with you, and I would like to seek your input. The main -- the original focus of this document was to create a guidance along the same lines of a guidance for large molecules. And it is very much embedded in the regulations, and regulatory policies, and so forth.

When I came to the Office about a year ago and started stirring things up a little bit. And I started asking many questions. I was troubled by many things, such as the original draft, if you recall, would have meant in many cases of increasing, or to be more quantitative, duplicating the number of supplements. So rather than having a supplement to make a change, now you submit a supplement that we are calling comparability protocol, to be followed by another supplement to make the change. The main advantage could have been that you can implement the change without waiting for the approval for the second supplement. But you cannot get the change going until we approve the first supplement. That's the problem I have. Another problem I have, it would have very much doubled the workload that we have for our staff.

Number three, which is the major issue, the first two we can handle. And Steve has been working very hard to address these two issues. But the main problem I have, the way the draft has been, and the comments we have received, do not really articulate our current thinking. And if you look at what the guidance is, a guidance is not a regulation. It's a way for us to share with you our current thinking, and suggest ways for you to provide the information for us, for proper assessment in order for you to continue to manufacture products. I don't think of a guidance the way it was, before I came to the Office -- so again, don't blame Steve, blame me -- does not really share our current thinking.

What's our current thinking? I think Ajaz has tried for years, for a couple of years at least, to articulate that, and we are still debating and trying to define the desired state.

DR. HUSSAIN: It's define, Moheb.

(Laughter.)

DR. NASR: Right. Explain what it means for different scenarios, and so forth. What we are saying is if you understand your process, if you understand your product, and you have built enough data, generated data, because of the design of experiments and other experimental protocols, and statistical methodology used, and you have defined the space that you have seen in John Berridge yesterday, and Ajaz and others as well, where we are comfortable that within that defined space the quality of the product will not be compromised.

In our current thinking, in the new paradigm if you wish, it is up to you to make and implement these changes. You don't have to come to us and say `I'm going to make that change. Is it okay? Do I need your stamp of approval? How am I going to deal with our inspectors?' What we are telling you is since you have done your work, you understand your process, you understand your product, go ahead and make such a change. And it doesn't have to be a change from prior approval supplement to CBE-30 or CBE-0. And that's where we are struggling with this.

A few other points I would like to make, and after I make my points I will appreciate for you, Judy, and your colleagues to provide us with comments about how can we make this document as useful to you as possible to facilitate the process. Not necessarily to -- not only to reduce the filing categories. I have a problem with my eyes, that's why I have to take my glasses on and off. I'll fix it tomorrow. I mean it.

What we are trying to do with this guidance now is very much to bridge between the existing system, or the existing paradigm, and our future thinking. And that's the reason for struggle. I think in our future, the new paradigm, the idea is not to reduce regulatory requirements, or filing categories. It is to look at ways to possibly eliminate supplements altogether. And that's some new things. And you know, we need to hear from you how we go about that. And I think hopefully the comparability protocol in the final draft after I'm done with it, may provide some ways to facilitate this.

Because we received a lot of comments on this guidance, Steve and his working group have been working very diligently trying to do two things: to expand the guidance to address all the issues raised by the public. That's number one. Number two, to provide more details and examples of when to use it, and when not to use it, and so forth. I think this is very good and noble, but it resulted in increasing the volume of the guidance to become quite a bit. Useful, but more descriptive than I like. So we are working on a compromise, and Steve and I have been working very closely with this, along with people in this immediate office, in making the guidance brief but useful. I think we would like to make it useful, but at the same time there is no reason to make it extremely detailed because I can assure you, no matter how many issues we cover in the examples we illustrate, it will never cover everything. So why not even try. Why should we try.

And I think at last I would like to hear from you, and I hope you focus your comments on what you like to see in the final draft of this guidance. We are working very hard, but we have some internal struggle of how to make the guidance useful, and to bridge between our current regulatory policy and our future paradigm, and facilitate the transition from the existing system into the future regulatory process. Thank you.

CHAIRPERSON BOEHLERT: Okay. Moheb has asked us some questions on how FDA may make this guidance more useful. And I'd be happy to listen to committee comments. Any comments? Gerry?

MR. MIGLIACCIO: First, Moheb, I very much liked what you just said. I guess you expected that.

DR. NASR: I'm surprised, Gerry.

(Laughter.)

MR. MIGLIACCIO: Clearly, a single-use comparability protocol is going to have limited utility. The firm is going to have to prepare two supplements basically, and you're going to have to review two supplements for single-use. Much more utility for repetitive changes. And the concern has always been the specificity may limit repetitive-change use. So, that's certainly one thing that we do see a very good use of comparability protocol for repetitive changes, but how specific does it have to be defined, and how broad can the applicability be. So that's one.

But I think you hit it. You know, John Berridge talked about the design space, the variable space yesterday. We have to figure out a way to continue -- what's the process for first defining it in the original NDA, and then continuing to build it. And as it builds, to continue then to build in the flexibility to make changes without any supplements. That's the process we have to nail down. And it would be ideal if that could come out. But I think you will see firms who choose to do this, and to continue to build that design space, will need some way to get that in to the NDA and reviewed so that they can expand the design space and make those changes. So that is something that we'd be looking to discuss, the mechanism for doing that.

CHAIRPERSON BOEHLERT: Dan?

DR. GOLD: I am very much in favor of the vision that I think you are trying to put forward. And I must say I frankly did not understand why -- if a fully thought out comparability protocol, fully defined, with all the parameters clearly specified, all the data be gathered, fully specified, the acceptance criteria completely defined, if the firm achieves what they say they will achieve if they do the study, I could not understand why I would then have to put in another document such as a CBE-30 or a CBE. I did not understand why I would not be able to go to an AR immediately. Because if I have clearly defined all the requirements that I will meet, and then I do meet those requirements, and your staff has accepted all that in advance, why not be able to go all the way? So I am very much in favor of the vision that you are trying to move toward.

CHAIRPERSON BOEHLERT: G.K.?

DR. RAJU: I agree with the comments that were made before. I just wanted to raise two points. You can choose to make them irrelevant if you don't agree, and don't want to think about it further.

If we allow a rapid transformation of the manufacturing system over the next two years, and we greatly enhance the capability, and in doing so increase the amount of supplements rather than decrease it, is that a bad thing? I move on.

Number two, is the right body of unit the number of supplements, or the quality of the supplements? And isn't that -- once you make it consistent with the vision, shouldn't the focus be on quality per supplemented -- time per supplement, rather than number of supplements. I agree with everything, but those are the two points.

DR. NASR: I think you are raising a very good question, and I want to make that very clear. I'm not saying that time will come where we will eliminate all supplements. I think what we are trying to work on is to justify the need of supplements for considerable changes that cannot be evaluated at the manufacturing site. I mean, if you make some minor changes that will not impact the quality of the product, the process remain under control within that defined space, why do you have to come to NDC? I don't want to see you. Basically go ahead and implement the change, since you have laid out early on your experimental design and how you are going to control the process, and the parameters are well defined within that space. There is no reason for supplement.

However, if you elected to make a major change that may impact for a change in the specification, or may require evaluative study. Where we are getting to potential clinical impact, this may be a time where you can propose the change and bring your experimental design to us for an assessment to make sure, because we have a responsibility to the public that the change you are making, the major change you are making, will not adversely impact the quality of the product as it is related to safety and efficacy. That would be the only time, in my mind, where a supplement is needed. If you are changing a lubricant on a seal on a filling machine, I don't think you need to come to us with a hundred supplements to do that.

DR. RAJU: So we won't get to a place where there's zero supplements, but getting there means first increasing it before it goes down. How are we going to find out?

DR. NASR: I think our role will be to facilitate continuous improvement. And some of this continuous improvement can be done without any regulatory oversight, and some may still need some regulatory oversight in the form of scientific dialogue to have an assurance what you do is scientifically sound.

MR. FAMULARE: A question I might raise to Moheb and Steve. If the change is bringing you closer to the specification, or closer to the design space, as opposed to you're further away from it, then could we -- is that an area of no supplement? Is that how you're looking at it?

DR. NASR: I think, if I hear you correctly Joe, you want to change the space. And you are saying `Are you willing to expand that space?' I think that will be something that we need to look at.

MR. FAMULARE: Well --

DR. NASR: But, but -- just let me finish, please. But, if we agreed on that space, and that's the data, and this is the scientific model you have, you can go ahead and make the changes within that space. If you come and say, `Well, I'm going to expand the space, and instead of having that oval-shaped, I'm going to have some points scattered around and generate another geometry, if you wish,' this will be a time where we need to sit together and see the impact of such a change on the space, on the quality as it relates to safety and efficacy.

MR. FAMULARE: Right, what I was thinking of is if you're going beyond the space, your process is drifting beyond the space and then the change brings it back in, is that something that you want to see?

DR. NASR: No.

MR. FAMULARE: Right. And I think that would make a good corollary to the Q10 and how -- the Quality Systems, and bringing things towards continuous improvements. And I think eventually this will correlate with that.

DR. NASR: Some people, however -- I know you don't -- but some people, however, think of the concept of continuous improvement, that there will be no regulatory oversight whatsoever. I think we need to minimize regulatory oversight to facilitate continuous improvement, but there will be some key elements that must be integrated, must be presented in a coherent manner. And these are elements that may require evaluation assessment, good Quality Systems to manage the process of the plant, a good GMP inspection, and defined space regulatory processes. All these things need to be together.

MR. HOROWITZ: I don't disagree with anything Moheb or Joe said. I agree completely, and I just wanted to echo a couple of the sentiments.

Continuous improvement doesn't require the absence of all regulatory oversight. I think we all agree with that. Our system intentionally has redundancies built into it. And that's a good thing in terms of protecting the public health. Sometimes it can get in the way of continuous improvement to the extent those redundancies become burdensome. And it's partly our job to identify areas where we could do without some of those redundancies. And I think there's often overlap between the safety oversight and the benefits on the review side, and the safety net that we have with Quality Systems and with GMP oversight. And there are certain instances where we could take the chance, if you will, as regulators, to give more flexibility to the regulated industry to make changes, knowing that if something goes wrong, there are other safety nets. There's a Quality System in place. And if we get more assurance that the Quality System is effective not just to prevent errors, through change control and other things, but also to be able to detect them, to detect them in a timely fashion.

And I think that's what Q10 is really about. It's about giving the regulators more confidence in the ability of the Quality System to serve as that safety net, to give us greater confidence and greater ability to remove some of the redundant oversight that may have been in place on the review side.

One last point. It all comes back to specifications, though. We could have all the Quality Systems in the world, but once the specifications, as part of the QA process, become more rational, more clinically based, I think we can ultimately have greater confidence in the ability of enhanced Quality Systems to catch real problems that affect the clinical -- of clinical significance that would affect the patient. And I think that's all part of the desired state. It's going to take awhile to get there because there are a lot of pieces that need to be put in place. And things like Q10 and other aspects of this require a bit of a leap of faith for all of us, to be willing to say `We can't be sure whether this is ultimately going to have the payoff we're expecting, but we've got to build a foundation if that might happen.' It might not be a sufficient condition, but many of these things are necessary conditions to move forward to the desired state. Thank you.

CHAIRPERSON BOEHLERT: Any other questions or comments? Gerry.

MR. MIGLIACCIO: David, the way you've described Q10, obviously we agree with. The question is if we don't get the support in ICH for Q10, it has to happen here. So we need a contingency plan, as we're still not assured that it will move through. It's not approved yet to move forward.

DR. HUSSAIN: It has been accepted. I mean, the timing of that is going to be just --

MR. MIGLIACCIO: The timing.

DR. HUSSAIN: A step of when Q8 and Q9 goes to Step 2. That's the timing. It's a timing issue. I think we supported it throughout the process, and we leave it to our regulatory colleagues from Europe and Japan because of their resource issues. So I think the steering committee has accepted it.

CHAIRPERSON BOEHLERT: Are we ready to move on?

MR. FAMULARE: I just had one short comment, that I mentioned over the course I think of yesterday, that we have this Quality Systems guidance coming forward, and it's more broad than Q10, but certainly comments to that guidance when it issues in September can certainly latch on those things here, and get it moving. And it may spark movement also in ICH.

DR. NASR: I just want to add one thing in response to Gerry's question about Q10 implementation and timing. I think it's a good thing it will have a global agreement of the goals of Q10 and how to get there, but I think we internally here at the Agency have decided to move on. So we are making some drastic changes now, both on the review side and the inspection side to facilitate continuous improvement. And we are very serious about that.

CHAIRPERSON BOEHLERT: Okay, I think we're ready to move on. Thank you, Stephen. And the next speaker is Jon Clark, who's going to talk about changes without prior approval.

MR. CLARK: If I could have someone come up here who knows this computer and get my talk up. I've had experiences, bad experiences, with this before. I don't care to repeat them. Thanks.

One of the things that's striking to me while listening to all this conversation is that it largely steals much of the thunder from what I wanted to say here.

(Laughter.)

MR. CLARK: But I do want to bring -- I will be able to speed up this talk considerably, because I don't think -- much of what I thought might have caused conversation probably won't, now that we've had the conversation.

But one of the things I hear people talk about, and I have a long experience with review work. I've done more reviews than perhaps anybody should. And one of the things that we consistently confuse, and I have confused in the past, is the difference between a specification and a process control. And I want to articulate that by how I got to work today, how I came here today. And I used a car like so many other people do. Mine happens to have the shape of a pickup truck, which gives me a lot of advantages.

But one of the things is the process control is the speedometer, the temperature gauge, tells me everything's working all right. The map that I have on the seat next to me, that's a process control. The specification's about where I have to go. The specification doesn't come out of the process that I've done. It doesn't come out of me looking in the back mirror. The specification has to do with where I want to go. That all comes out of the front window. So, keep in mind that when we talk about specification, we need to clean up a little bit our terminology, because we're being a little sloppy here in places. And if you think about, a specification comes from the next step, not from the one I just completed.

And the way we apply that to pharmaceutical process is that we need to be thinking about the spec for the LOD, or the spec for the moisture in the granulation shouldn't be set by how well my granulation is working. It should be set by what my tabulating machine can tolerate, by what the degradation profile of what the raw material, the API, is. So keep that as a thought. Go into that, and I'll give my formal talk, the one that my supervisors actually approved, and we'll go from there. Thank you.

So, changes without prior approval. How do we get from where we are now to where we want to go. And I hope at the end to talk a little bit about the desired state. But I want to point out that you have to be very careful because I remember a previous great American who once said that the most feared words in the land are, `Hello, I'm from the Government, and I'm here to help you.' So, let's move from that, hopefully get to another quote later on, and see where we go.

An overview of the traditional system. We've gone through it ad nauseum today. But the traditional system of approval and change control does seem burdensome. There should be a way to protect public health without slowing innovation. And the methods and standards for this are already available, and part of this talk will go into some things that weren't brought up. But we'll see if they contributed or not.

We need to train ourselves into new ways of thinking, but we do have shared concerns. One of the concerns is that the pharmaceutical industry is one of the most technologically advanced discovery organizations, but remains more conservative when it comes to using cutting edge technology in manufacturing. Concern over how regulatory agencies will react to using knowledge and technology is a big problem. Agency focus on changes that have inconsequential impact on product quality, and can result in delay, is a very big concern. And that's part of what this talk is all about.

There is, from looking from where I have been standing for so long, looking out, there is a complex interaction between the industry's commitment to high quality products, and their commitment to most rapid introduction to the market. There are some inherent interactions there that concern us as reviewers and approvers.

Optimization before approval has certain good points. One is that it provides the greatest immediate benefit to the patient. That's the last bullet under that subtopic. But the greatest cost is in time and developing all the optimization information. There also is, when you start production in that paradigm, there is no baseline from which to measure improvement. You're kind of thrown into a situation, and you don't really know after that whether or not you're optimized or not. So optimization has a funny definition when you're talking about before approval.

In a continuous improvement environment, the time element is minimized because you can get to the market with an adequate product and with an adequate process. Also, it enables measurement of the improvement because you do have that baseline. And the feed forward data in scope -- protocols, can all be designed around a continuous improvement paradigm, and that helps us from our end.

And I would like to point out, the inclusion of development data helps in the initial review, but it can not equal the knowledge that is obtained during routine production. And yes, even reviewers see this in the applications. We see that in a large way in the number of supplements we get. And we can see that there are improvements being made most often.

I want to steer our way through a few points. Raw materials process. The term "measurement." Steering the process. And last is variability. When it comes to raw materials, it's pretty well demonstrated. The pharmaceutical raw materials are variable. It doesn't mean that there isn't a company out there that hasn't learned how to pressure their suppliers into keeping the raw material variables down to a minimum. That is done very often. The point is that it's very expensive to do. So we cannot also assume that holding inputs constant will always produce a constant product, and that is because you do have variables in the raw materials. So the conclusion: attempting process control through raw material control is really futile. And futile does not mean impossible. It means expensive, and it means inefficient.

Let's talk about the process. Discovery and design suggests a process model, if you will. The model should be designed so that the parameters for that model. This is a sort of a very soft, high-level model. Those parameters that are suggested by the model need to be able to be measured in the real world. So if you say that, well, this outcome is dependent on some nuclear magnetic resonance, it's not going to be measurable. So you have to make sure that you have a measurable parameter. And as the model evolves, the measurement strategy should evolve with it. And the effect of change can be better predicted when you have realistic models.

And I'll also point out, the last point is that there is a dearth of process models in applications. We don't see that. What we see are very specific demonstrations of actually manufacturing the product.

Let's talk about measurement. Measurement is most effective when used to control the process in real time. We heard Chris talk about that. And Chris is gone now. But with PAT, that's all about PAT. But it goes beyond PAT. It's just inherently a fact of nature that measurements are more effective when you're looking at using it to control a process. And yet, in spite of that, the traditional approach, and probably because of the age of the art of chemistry and how long the Agency's been involved, the traditional approach has been to sample a product pretty much after it's been processed or some intermediate product, and then test that for compliance with a criterion via a laboratory determination. And that's the term actually used in the CFR.

And we talk about steering the process. We talk about changing time, speeds, and temperatures, based on measurement to achieve a target value for a product parameter. And we also want to point out that discarding batches, or discarding portions of batches, in a hope to get some recoverable material that's marketable out of them, is a sign of a failure to properly steer a process.

Variability reduction always adds value. It increases the process capability. It also minimizes the risk of out-of-specification results. And it's also a prerequisite for any kind of a successful investigation. Because if you have a lot of variability, you're not going to be able to figure out what's going on. And for the sake of G.K., I'm referring mainly to common variability and not special.

So we have a situation spectrum that I drew up. I presented it before. And basically it's a spectrum to try to demonstrate a world where you have extensive product testing with little process understanding is not as desirable as a world where you have high process understanding, high process understanding to the point of obviating end product testing. Now, I gave this slide at an Arden House conference 10 months ago or so, and it was something of a shattering thing to have an FDA'er say. But today obviously we have everybody saying something very close to this. So it's very good.

And then we have a little "therefore" at the end. The FDA focus on laboratory testing is not ideal for controlling processes. We need to encourage process understanding and engineering. We need to focus on the resources, on manufacturing process instead of lab tests and criteria. And we need to avoid this trap of measure it because you can. There are -- often we've seen, many times, where someone will say, `Well, we know that you can get this value out of your process, so we insist that you get that value every day,' when no one has ever bothered to go back and look and see whether that parameter mattered at all. And if it doesn't matter, then why are we measuring it to begin with.

Also, zero tolerance limits. There is sometimes a need for zero tolerance limits. But I'll make the submission that a zero tolerance limit is mainly a sign of a lack of knowledge. And as you get to a higher level of knowledge, and in this graphic I have up here now increasing process understanding and control, the need for zero tolerance limits goes down. And although in this graph it goes down to a minimum value, I would submit that an edit of this graph would have it go down to zero, because that really is where we want to go.

I also want to point out that post approval regulation, and knowledge, and process understanding are related in this graphic. Of course, the more knowledge you have, the less post approval regulation we would need.

And the current paradigm is described in this graphic. We have raw material going into a manufacturing process. It has locked process variables. And coming out of that we have a product. And any variability in a raw material in this particular schematic, the variabilities pass through the manufacturing process, and because it is so locked, that variability goes right through to the product.

I submit a dynamic system, where you have a raw material going into a manufacturing process. You have measurement-dependent process variables. For whatever purpose that might be, you are actually measuring what's going on, and you might change your process variables according to that measurement in real time. You would have some kind of an input response to that. You would have an endpoint response, and then eventually you would get out the product. You give these terms new names, and you just have PAT. It's raw material manufacturing process. You go feed forward, feed back, critical process parameters, critical quality attributes. The product name still stays the same.

And we are not alone. It's just a series of things that have derived from a military standard that has since become an ANSI standard. It's numbered here for the sake if you want to go look it up. It's not currently used because the military actually references the ANSI standard in this case. It was done in 1996. And their points ring very true today for us. And these are mainly out of the introduction, not the sampling procedures which they also describe, which I'm sure that Dr. Singpurwalla would probably have a problem with. But I don't know.

So leave that where it is, and let's look at the philosophy in their introduction pages. In a process control, the statistical control methods are the preferable means of preventing non-conformances, controlling quality, and generating information for improvement. Sampling inspection by itself is an inefficient industrial practice for demonstrating conformance to the requirements of a contract and its technical data package. That contract in this case is of course CNDA. To the extent that such practices are employed and are effective, risk is controlled, and consequently inspection and testing can be reduced. Now, when I first had this slide, we were talking about prioritizing our inspections in such a way. But as you saw today, we're talking about that with David's efforts earlier today.

The objective is to create an atmosphere where every noncompliance is an opportunity for corrective action and improvement, rather than one where acceptable quality levels are the goals. In other words, throwing away parts of a batch in order to get it within criteria is not a correct methodology. The goal is to support the movement away from an inspection strategy into effective prevention-based strategies, including a comprehensive Quality System, continuous improvement, and a partnership with government. You may have trouble with the word "partnership." It's up for debate, but the point is that we are all after improving the public health, protecting the public health. Use the terms you wish.

And more. Process should be the focus of the Quality System, consistently producing conforming product, controlled as far upstream as possible, robust variation, operated to constantly reduce variation, utilization of equipment in a way that minimizes variability around target values, managed for continuous improvement, designed and controlled using a combination of practices and methods, in order to ensure defect prevention and process improvement. That's the end of the military standard stuff.

And I bring up William Edwards Deming. Can I have an effective presentation without quoting William Edwards Deming? I think not. Not in this area. And this was quoted yesterday in a couple of presentations, at least in part. "Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place." Depending on inspection is like treating a symptom while the disease is killing you. The need for inspection results from excessive variability in the process. The disease is variability.

Ceasing dependence on inspection means that you must understand your processes so well that you can predict the quality of their output from upstream activities. Upstream activities and measurements. Does anybody need a definition of "upstream"? I hope not. That means before the product's made.

Here we have I try to capture some of that in the one single slide. On the left-hand side you'll see a box that says "Range of raw materials in facility attributes." Now, we could have a long list of things I'm talking about. It's a range of things that could be variable. It could be long enough to not fit in that box. What I have there is pretty full anyway. And the ideal situation is that you have a process that's designed to limit the product variability in spite of these other variabilities.

Variation control is also part of Anna Thornton's Variation and Risk Management book, which is something of a how-to book on how to create a Quality System that is designed around controlling not just any variation, but the variation that's important to the parameters of your product that you think are important. And she talks about identification of key characteristics. Those are to assure achieving critical quality attributes. That's what the CQA stands for. And she talks about a variation flowdown, where you look at a variation that you're seeing in one place, and you look upstream until you find out where that variation is really being triggered, and control it there.

It talks about assessment, and which variations put the critical quality attribute at risk. It talks about mitigation. You can either eliminate the source of the variation, or try to reduce its impact, or a little bit of both. And she talks about setting up whole organizational structures on these ideas.

These are examples of evidence that came out of the military standard that I was talking about earlier. I'm going to try to get through them by just flipping through them because it's simply a list of pieces of evidence that one could supply to a third entity to demonstrate that you have control of your process. It's about flow charts, and identifying what essentially are operating procedures and plans for variation. But due to the time on the clock I'm going to run through them.

I submit that the contribution -- the institutionalization of knowledge in your organization is a quality concern. We need to apply solutions wherever they will provide improvement. And a prior regulatory approval for every improvement does in fact defeat this goal.

An application without supplements, what are we talking about? What do we need to see in that application? What are the critical quality attributes and the means of monitoring and controlling them? What are the fundamental scientific mechanisms of the physical changes in the process? Can you describe them? Can you articulate what those are and tell us how you're controlling them?

How do formulation and process factors affect product performance? Control and operation using mechanistic scientific principles directly while you're manufacturing the material. Demonstrate a range of operating ranges, controls, and principles. That creates your space. A history of manufacturing success with similar drugs, or similar operating principles, or similar site operations. All those things contribute to this history. And they should be used to create the space.

Significance of the site location and environment on the quality of the finished product, more of the same. Drug product specification, based on attributes critical to product performance experienced by the patient or the health care provider. Process control relationships to finished product quality. These are all the kinds of things we'd like to see.

Another thing on this list that we do not see now today are models. We don't see models about how to control -- what your control strategies are. And it became a little bit extensive. Didn't find its way on the slide, but I did write it down and would like to take the time to read that to you once I locate it in here. And what I wrote down here was model, model, model. Batch records, batch control cards. There's little value in batch records or batch control cards, or equipment settings or controls, when it comes to process understanding. We're talking about being able to bring the reviewer up to a certain level of confidence that you have. Not bring the reviewer a total amount of process understanding, but bring that person's confidence level up that you have an understanding of the process with a model. And that is what your specification in the application could be.

Operational freedom. Once you've done that, this process understanding knowledge leads to greater freedom from narrow operating procedures, which we often see today because, in place of models, we see batch sheets. Greater freedom from narrow operating procedures and allow focus on drug product quality. We need to provide for use of alternatives to any application requirement. And that includes components, manufacturing, and packaging procedures, in-process controls, analytical procedures. And anyone who thinks this is a surprise needs to read the regulations, because those things are listed, as they are in this bullet point, at 21 CFR 314.50 (d)(1)(ii).

Focus on process science understanding. The FDA wishes to avoid allowing the submission of great operating procedure in the application -- great operating procedure in detail with equipment specifications to create something of a safe harbor. And I have that in quotes because safe harbor is a quick way for me to get you an understanding, but I'm not a 100 percent confident it is a perfect term. But it creates something of a safe harbor for a process. We want to avoid creating that safe harbor for processes that do not consistently result in quality of product that is suitable for use. In other words, the model is more powerful.

Batch records should not be used as manufacturing process control specifications, or change control restrictions. Stability analysis is more valuable than raw data. Understanding degradation mechanisms helps us predict, helps you predict the impact of change.

Agency acknowledges concern about commercial research data. And it has a lot to do with when you do research on production batches, on commercial batches. What is the effect of doing that. And there is some concern about the data coming out of those batches for both commercial production and for research data. And we've had in several guidances some language. And I bring that language to you today for comment. And that language is the FDA acknowledges concern that process research data may indicate a problem when a product still meets its approved release methods. The FDA began the research data exemption concept in several guidance documents. That exemption does not protect a person that knowingly does harm without attempting corrective action. It also is designed to place this information outside the scope of a normal inspection. That's the term used in the guidance paragraphs.

It shouldn't impact on the ability to release products that meet all the aspects of the company's currently registered quality control strategy. And that would include all the terms we've talked about earlier.

And I'd just like to close with the situation spectrum, again. And that is that of course extensive product testing with little process understanding is less desirable than a high process understanding. And even though you have obviated the need for end product testing. And I think that might mean a little bit different thing the second time I say it than it did on the first.

And with that I thank you, and if anybody cares to have any questions or tell us that we're barking up the wrong tree, we'd love to hear it.

CHAIRPERSON BOEHLERT: Thank you, Jon.

(Applause.)

CHAIRPERSON BOEHLERT: Are there any committee questions or comments for Jon? Yes, Paul.

DR. FACKLER: I have one question and one comment. The comment has to do with one of your slides where you said FDA focus on lab testing is not ideal for controlling a process, and asking for data just because it can be obtained is a problem. I fully support that comment, but don't know how you're going to implement it across the Agency. I can't tell you how many times we get asked for information on a product that is, I think, completely meaningless to the quality of the product. But somebody knows that you can make the measurement, and wants to see the measurement, and set a specification on it.

MR. CLARK: I ask you in return have you included in your application the kinds of process, knowledge, and understanding, the kind of models that I've described in this presentation?

DR. FACKLER: Absolutely not. No. So the other thing I was going to say is when you say obviate the need for end product testing, is it possible that we're going to be able to manufacture a product and just ship it? We'll have enough process controls that there won't be any measurements done. We'll just drop it in containers and send it on its way.

MR. CLARK: There is a 21 CFR 165, that requires two tests: strength and appearance in the laboratory determination. Now, I have not been put in a position of playing with the term "laboratory determination." I don't know if that's being planned or not. That's the only roadblock I see.

DR. HUSSAIN: The way we have defined real time release, you're not eliminating any tests. You're using a different test method. It's an online test method. That's about it.

MR. CLARK: Hence the term "obviate."

MR. FAMULARE: The emphasis is on the word "test." You know, there's a lot of things that can meet the criteria for "test."

DR. NASR: I'd like to add one comment. I think you raise a very good question about -- we ask for data, and you go and generate the data just because you can. And how we handle that. And Jon tried to explain what he meant by his slide. But let me ask you a question. What do you do when we ask for data just because you can? Do you generate the data?

DR. FACKLER: Well, there's two scenarios. One is that we need approval for the product as fast as we can, so we give you the data, meaningless as it is. The other scenario is we take the time to communicate back to you and say `Do you really want this? Is it really pertinent to this kind of a product?' But that sets us back, and time is money.

DR. NASR: Well, I see more of the first scenario. I see very little of the second scenario. Where really I think you are pressing for time and we are pressing of time as well. But if we don't deal with this, what we are ending up with is we are in a vicious cycle. We ask for data, generate the data, and the data may require more questions, and so forth.

CHAIRPERSON BOEHLERT: I used to think in those situations, well, we'll give you what you want just to get approval, and then after approval we'll file a supplement. But you never have time to do that then either, so it never does get done. And that does happen.

MR. CLARK: Ken, yes.

DR. MORRIS: Yes, one of the things I think that -- and we've talked about this internally, I know, is the idea of using models to be able to give you enough confidence so that you can, in a relatively short order, be able to make a case. Which is not always based on the specific data that are being requested. But what happens is, and this happens during consulting all the time, is that when somebody comes and says I have a problem, well they do have a problem, but the problem that they have isn't the one that's presented. That's the symptom. The problem came somewhere upstream. And if you have to take the time to find the problem that was manifested as that symptom, then of course you're completely correct, you just can't do it economically. If on the other hand you've already demonstrated understanding the process to the level where you see where it deviates from what you'd expect, or more to the point that you're raising, when it doesn't deviate, irrespective of the test that's being requested, then I think it's a fairly quick process.

There's a lead time, of course, but it's a transferable lead time I think. And I think particularly for generics where you have just tons of data, historical data I mean, for giving tablets. For instance, I think we were talking about yesterday where you have just hundreds and hundreds of examples of tablets where the formulations aren't dramatically different. Those data pooled would seem to me to be a very powerful set of data for making the argument. But that's just my opinion.

MR. CLARK: Thank you. Anyone else?

CHAIRPERSON BOEHLERT: Any other questions or comments? Joe.

MR. FAMULARE: Just to go back to your slide about the ideal application, and then the need for no supplements based on that. A lot of that is built on the new paradigm, having process understanding and so forth. That's all right. Don't touch it, Jon.

(Laughter.)

MR. FAMULARE: I think another scenario, and Moheb and I already kind of discussed it on the side of the table here, is when you don't have that process understanding. The application file is reviewed, it's approved. And you end up learning tings over the processing of many batches. And you realize that over time what you thought would be an optimum process is really going way off to one side of the space. It's going to fall off, and you want to get it back to the middle again. Those are the types of changes that I think can be made by the company as well under that, to get things back on center. You're not changing the specs. You need to do that. And I was saying to Moheb, that's where I see the conflict and conflagration and inspections. You're damned if you do, and you're damned if you don't. You're either cited for not following your application, or you're cited for being way off to the side here.

MR. CLARK: I'd like to build on that a little bit, if you don't mind, Joe.

MR. FAMULARE: Sure.

MR. CLARK: And that is that we've seen -- we talk to companies that come to us. And the bigger disappointment for me now, after doing all that review work I've done, is that I think that a lot of the information we're talking about to build that space is already there. We've talked to companies. They show us what they've done. And then for some reason they feel inclined to reduce this model to a batch sheet, and then they submit that thing. And I'm not sure that we have to worry about them doing a lot of work that they don't already do. You're just asking them to build that model, build the space, give us some confidence in it, and make that your specification.

MR. FAMULARE: Yes, well, that's -- yes, that could bring up another point, whether, you know, I'm talking about you're good with the spec. If it's going to be that you're changing the spec, obviously that's going to come in.

MR. CLARK: The model is the spec.

MR. FAMULARE: Yes. And the spec defines the space. Now, there are other instances where you want to change the space, but that's another story.

MR. CLARK: Well, that's a different story. I'm talking about not necessarily having to change the space. You have a space. You're comfortable with the space, but you need to operate within it instead of worrying about getting permission to operate within it.

MR. FAMULARE: Yes, I guess in my scenario they may have developed that knowledge over time, but they didn't have it when the application was approved.

MR. CLARK: That happens, but --

DR. HUSSAIN: Joe, let me give you a specific example. Let me just create an example. I think we have talked about it.

MR. FAMULARE: Right.

DR. HUSSAIN: An example might illustrate that better to the committee.

MR. FAMULARE: Okay. An example may be a suspension product where the company will realize that they're throwing away the last third of the batch. They can't maintain the suspendability over the filling time. And what they will do is work to change that. In this scenario, they actually got it to where they had a consistent suspension through the filling process. And the observation was on the 483, you did not follow your file process.

MR. CLARK: I would love to answer that now, if you don't mind.

MR. FAMULARE: That's fine.

MR. CLARK: What was the control parameter that caused them to stop filling at the 30 percent level and abandon the batch? What was that control parameter?

MR. FAMULARE: That was testing. It was testing for, you know, the --

MR. CLARK: What they need is a real-time monitor that tells them they've lost suspension. And then that's the model, that's the metric --

MR. FAMULARE: But actually they improve the process so that they can keep it through the whole time consistent, and not, you know, you had the example of not steering when you're throwing out part of the batch all the time. You're throwing out a third of the batch.

MR. CLARK: Well, I'm not sure that the sample -- you couldn't use that same idea in the sampling paradigm. Because if they're pulling the sample to see when they lose suspension, you get away from making 30 percent your mark, or time your mark. You get back into `Did I lose suspension?' as your mark. You still solve some of the problem.

MR. FAMULARE: But I'm saying that change was in the fringe purview, and they resolved it because they got back to closer to their mark. I mean, just as an example. I think it was a good thing that they did. But the confusion, or the need, or whatever, to file all that, and to have that happen -- and this was a product that had to keep producing. It was medically important. It wasn't something that they could just say, all right, we'll stop for a half a year. I mean, it's important to the firm not only medically but financially too. So I mean it's not something they want to stop. A lot of the discussion here was about throughput and efficiency, and keep optimizing that.

MR. CLARK: Right.

MR. FAMULARE: So it's just a matter of the timing of all this as well.

DR. MORRIS: Can I just ask, Joe, are you saying that even given the fact that they were able to improve it and demonstrate their improvement, they still got -- they were still cited for it?

MR. FAMULARE: That's correct. The opposite example is when the firm continues to make something in a non-optimal way because they want to make sure that they have completed all the filing requirements before they make the changes. So that's the flip side of the example.

MR. CLARK: I just caution people, when you make your filing, and you have a parameter that's causing a problem in the batch, it's the parameter that should be the control, not the 30 percent mark. I think you said 30 percent. You were throwing away.

MR. FAMULARE: Throwing away 30 percent of the batch, right.

CHAIRPERSON BOEHLERT: Any other questions or comments? Okay. Thank you, Jon. Ajaz, I think we're ready for summary and wrap-up, if you're ready.

DR. HUSSAIN: Thank you, I'm ready. I think Madam Chairperson, members of the subcommittee, I wrote formally. The invited guests and staff, I really enjoyed this meeting. It was a very productive meeting, and thanks to all for your recommendations, comments, and for challenging our assumptions. I think that that is always good to have.

Just to sort of summarize what I was able to gather, and I think summarize this also for you. We started the discussion with respect to looking at what we have done with -- in a very summary way the pharmaceutical quality, the quality initiative for the 21st century. We received updates on what is happening in ICH Q8, Q9, and the proposed Q10. And we also talked about the ASTME 55.

The key learning from the discussions of the subcommittee at least for me was I think there was a strong agreement among the committee members that these current activities are important and are helping us to move towards the right direction. And by providing more detailed information and what is needed in the desired state. I think these are all helping.

There was a caution that we need to keep these activities as synergistic as possible, especially ASTM and ICH activity. And the committee suggested that I think there needs to be some communication of what we are doing at least in ASTM to our European regulatory counterparts. And I think we will take that advice, and in November seek to update them on this.

I think the scientific principles and principles of risk management that we are embarking on are helping us move in the right direction. But I think this theme came again and again. And this was that there is an urgent need for a concrete example of case studies, both for generic drugs and for innovator drugs, to help us clearly put a strong foundation of what the desired state looks like with that concrete example. And I think that is an important aspect that kept coming back again and again.

After that discussion, I think we also had some specific questions with respect to are Q8, Q9, and the proposed Q10 helping us move in the right direction. And we also asked about quality by design, and how do you sort of consider and link that two failure mode effect analysis and so forth. But the key, I think, answer to that was that I think failure mode effect analysis is a tool, but it has to be used within the broad context of the scientific principles and so forth that cannot be separated. And that was a key message.

And with respect to the second question, I think we really asked for some help in helping to clarify what is minimal requirements, what is optional requirements, and so forth. And I think one of the suggestions, especially from Garnet Peck, was the preamble, at least. How we introduce that question I think has more valuable information, and we probably need to retain that, is how we are providing incentives and so forth.

And in some ways I think that was important, more from -- not from a scientific challenge perspective but from a communication perspective. Because that was the topic for discussion at ICH again and again, and will be so when we go to Japan, especially because I think the European system already has development pharmaceutics, already has some of these elements that we are talking about. The disconnect and the difference I think that we have right now is we did look at the development pharmaceutics, those reports. We didn't find those very useful. So it was not that we wanted to simply adopt that. They're not very useful. They don't give you any process understanding. So what would surprise to all of us is -- not surprise. I think the design state is -- I think we are talking about a different level of sophistication here. And I think that's the challenge to maintain that. And I think that will be a challenge in Yokohama, Japan, as we go towards that. But in many ways I think the committee's discussion was very useful even for that aspect of that.

There was another question that I was hoping to ask, and then hoping to seek committee input directly. But I think I did get that indirectly. It's help in defining the design space that we are talking about. And much of the discussion led to that, and I think Jon actually nicely summarized some of the bullet points that leads to the design space. And I think that was very useful.

We then had an introduction to Bayesian approaches. I really thank Professor Singpurwalla for doing that. Recently, I'm forgetting the date now, FDA and Johns Hopkins University had the joint collaborative workshop on this very topic. In your background packet we included a web link to all the presentations. I think the first two presentations on the introduction are very useful, if you care to look at that site.

But that workshop is a strong signal with all of us inter-directors and our deputy commissioners that are sort of supporting that is that FDA really would like to move in this direction. All of FDA, especially CDRH, is already utilizing some of these principles. And I think we have a strong interest in this aspect, and we will pursuer that. The challenge is, I think many of us, most of us, are not well versed with this. There is a learning curve for all of us. What I like about it, and what I gathered from the presentation of Dr. Singpurwalla was, I think from my perspective, the confidence level of decisions made under Bayesian are better than when we don't make it without the prior. The decision quality improves under Bayesian thinking and approach because you don't just rely on a P value, you bring a prior likelihood measurement.

That's from a strength perspective. But from a personal perspective, you really need a statistician to work with the engineer or a scientist to do that. You just -- to make a statistical decision, the most scientific decision. So, hopefully I was correct in my understanding.

DR. SINGPURWALLA: On the dot.

DR. HUSSAIN: On the dot. Well, I think that is the strength. And I think personally, before coming to FDA my work was in modeling, and was in neuro -- molecular biological intelligence. There's a direct connection to that. So I was always fascinated and excited about that possibility.

I used the time after the Bayesian presentation to just update on the critical part in issue there. I just touched upon the industrialization dimension of that. But that is a significant initiative. And we hope to issue a list of research projects abroad, or just projects that Agency can be working on. You can contribute to that list. I don't have a docket number handy, but I think there is a docket number on that.

In terms of industrialization, I sort of presented some of the challenges I see, especially in research and education. Clearly, I think I suggested to the advisory committee that I think we need to move towards a more support for pharmaceutical engineering program, possibly a national center for pharmaceutical engineering, or multiple centers for pharmaceutical engineering.

The point Dr. Peck made was a good one, that I think we really have to be careful how we define "pharmaceutical engineering" because you have to bring a systems thinking, to bring biology, pharmacy, chemistry, and engineering, all together. It's not just engineering, and I think that's important.

FDA, especially OPS, will be working with a number of schools who have expressed interest in moving in this direction. And we are meeting with some soon. And you will see possibly a collaboration emerging between FDA and these schools, hopefully to support the move in this direction.

Following this I think we had very extensive and very exciting discussion on quality by design, and what it means for specifications. And I think this is important. Specifications, Jon is right. I think you have to be careful how you define "specification." Specifications under the ICH umbrella is defined as an attribute, best method, and acceptance criteria. So three elements go together to define what we mean by "specification."

I shared with you some thoughts on the dissolution test. And the message that I was trying to give, that was I think the challenges we face today is not the dissolution of the drug. That's not important. That was not the message. The message I was trying to give you is the methods that we have might not be the right methods. And even though dissolution is important, when you have a calibrator tablet that keeps shifting, and when you have a calibration standard that is three times the size of what would be accepted under an F2, what are we doing? And we have been using this for years. Isn't it time to put this on the table and start addressing some of this? Industry's very happy with F2 metrics. That's the way I look at it. Right, Gerry? So they haven't complained. So why should FDA complain? So I think it's time to really discuss these issues which have been lingering on for years. And if you really look at the measurement systems that we have, most of our measurement systems where we have problems are physical measurement systems. We still don't have a good means of comparing particle-sized distribution. Hopefully PQRI in one of these years will come up with a solution. But we haven't.

So if we really look at it, the message I was trying to give was when it comes to physics, we do not have to do this. When it comes to chemistry, we are doing extremely well. In chemistry, we actually have done an extremely good job on identification and other things that Moheb described. But when it comes to physics, it's not.

So the future is dominated with physics. If you really look at it, at least with respect to nanotechnology and drug device combinations, say drug-eluding stents, these are all physical problems that are being confronted. And you're not really ready for that. In many ways, when Dr. Singpurwalla asked me to redefine the desired state, it's today we are using all of this to improve our efficiency today, but five years or ten years from now our systems may not be adequate to control the quality of the futuristic product. So we really have to move in that direction anyway. So why not do it in a pro-efficiency now and be ready in a proactive way to address those challenges we'll face of the complex nanotechnology-based drug device combinations. So I think that's the way forward.

Sorry. I learned so much so I have to share this back with the -- but I think the key aspect was, I think you saw already impressive presentation by G.K., as usual, on how we sort of move towards a manufacturing science and knowledge. I tried to cover the specifications and then took it to the next step and said, all right, the root cause investigations when you do it right, and how do you do it right, and how do you sort of communicate that knowledge. And then you had very excellent presentations by Moheb Nasr and Gary Buehler sharing with you some of the activity, some of the programs, how they are planning in a step-by-step fashion to move towards the desired state while managing the current workload and then moving towards that.

And I think clearly the focus today has been on Office of New Drug Chemistry. And because they had a wonderful opportunity with the pharmaceutical development and reinventing themselves quite rapidly. Office of Generic Drug has such a high workload right now, I think they will have some challenges, and the points made are well taken, and I think we'll have to work very closely on that.

And so we wrapped up yesterday with I invited Ken Morris to come back and talk to you, because I think he has been working with our CMC leadership, both to generate and from New Drug division to start brainstorming. And the whole message comes back as unless we come up with very concrete questions, set of examples and so forth, we will have a difficult articulating what the desired state is. I'm not sure Q8 in its full version reached that. I think we need these studies.

And at that point I raised the question and invited John Berridge, and really raised the question. I think we need a working group under this committee. And the committee agreed that that's a good thing to move forward. And as a next step to this activity, I will contact Madam Chairperson, and we will put a working group together, possibly a working group to address all of the challenges we face with respect to pharmaceutical development knowledge, design space, and so forth. So requesting industry reps to consider suggesting names who would be on this working group. At this point I think what I would suggest is people with very broad knowledge base and talent would be the right people, because then we could task out each work to more technical folks. And I think it's important to do that. So we would like to move on that very quickly. Maybe within a week I'll contact -- later this week I'll contact Judy, try to assemble a team.

CHAIRPERSON BOEHLERT: We'll talk later today. I'm leaving on Friday.

DR. HUSSAIN: Okay.

CHAIRPERSON BOEHLERT: Okay.

DR. HUSSAIN: And we will put a group together that will help with our knowing the training programs needed, the workshops needed, and so forth, but also creating some case studies and so forth.

But what I also propose now I think, listening to all the discussion, I think one of the most important, critical project, research project, that we need is creating the case study. And I think we need to sort of put together a program. I know Monsoor is here, and it's a very opportune time that we are trying to meet with one of the major pharmaceutical companies on a research proposal, a creator, and maybe this could be another creator that that company might pick up. So that's one of the things that we can pick up and create that case study with that company.

So we have many opportunities with academia. We can work on creating a case study. But we also have companies coming with a research proposal on very similar grounds, so we might create another case study out of that too. But then we also work with the working group to create case studies from that perspective also. So that discussion was very, very valuable to us, and the importance of case studies is clearly paramount.

I think that the question we had asked is one of the current activities and planned activities in NDC, OGD, that you would suggest, I think. We didn't get many concrete suggestions, but I think what you saw in Moheb's presentation you liked the direction Moheb is moving. And I think you supported that strongly. And I think we will support that strongly. I think some concerns of the workload in generics was raised, and how we will manage moving towards the desired state, and how we will manage the supplement load, which is twice that of ONDC, 3,400 supplements. And the new number of new drug applicants, AND has 566. It's a humongous workload. So we'll have to be very careful how we manage that.

And I think that's not the only two offices. We have Office of Biotechnology Products, which was not discussed today. At some future point we will -- especially I asked Chris to mention to you that we -- they will be part of the PAT, so one of the -- we'll bring Office of Biotechnology for discussion with you next time when we meet.

So that was Day One. If I have missed any important aspects, please committee members, let me know. I think I'll stop for a minute for Day One.

I think before I talk about Day Two here, I had a brief conversation with Helen before she had to run and so forth, because one of the things we wanted to share with you today is that all of our activities in OPS will be focused on moving towards the desired state. I think that's one of the decisions I think we wanted to make after this meeting. This meeting was an opportunity to read, debate, discuss, and so forth. So all the guidances that we have coming out, and which are planned, will have an element. And I think you saw the discussion, the comparability protocol, that illustrates that point. It will be focused on moving towards the desired state.

There are many outstanding guidances, many guidances -- all the guidances like suit pack, we'll have to revisit those. And I think so all of our activities we've planned will be firmly grounded in making sure it is consistent with the desired state that we want to move towards. So that was the message I wanted to tell everybody.

But the challenge is going to be very great because it's not that we -- just tomorrow it will be decided. It's a long process. There's a lot of work to be done, a lot of education, a lot of interim training and so forth. But the opportunity is for companies that understand the processes, that do their good research and good science, and that share information. The desired state is not that great for companies that want to do the bare minimum. So the advantages are -- and the good part is most companies do that today. And it's a communication and sharing of all that information is what it is. Because the quality of drugs today is good. And I think it's an efficiency question, but tomorrow we'll be ready for the challenges.

I mean, today was an important discussion. We started with I think the study done in the collaboration with us -- not in direct collaboration -- by the two management professors I think will be very useful. And you got an update on that. There were a number of questions that will be useful to them to improve that model.

And then our colleagues from Compliance presented their pilot model for site selection. I think that was a wonderful discussion. At last, after my -- David can share any comments if he has any. And I think the discussion was very, very useful. The three questions that were asked we did get some input, and they did comment on that.

Well, let me wrap up my parts. The discussion that followed on Phase I investigation of new drugs, I was just sort of observing and listening. It is actually quite a big deal. It is a wonderful step in the right direction. So I hope you understand the magnitude of that impact. And I think Joe, others, have been working on it for quite some time. And that's a significant step in the right direction, I hope.

The afternoon session, we wanted to sort of give you more of update, rather than pose questions to you. But we wanted to show you with the PAT process that the guidance will be final. We have been innovative in ways of finding ways that do not require prior approval supplement. Again, clearly I think the regulations require when you have a change in specification, you have no option but to have that. But when you bring alternate methodologies where you don't need a change in specification, you have ways of getting the supplement. And through communication and team approach, especially product reviewers and inspectors working together creates more opportunities.

And Steve talked to you about his challenges, his group's challenges, on moving the comparability protocol guidance to be more useful. And I think the feedback that was received was very valuable again. And I think Moheb and others are working with that group now to make sure that it remains focused on the desired state also. And I thank Steve for all of his efforts.

With that I think Jon, I think, summarized some of the thoughts quite well. Very well done. And I think you can see the level of understanding Jon shared with you. And in many ways I think the bullets that he has, especially of what's to be the submission that gets you literally no supplement from a change perspective, I think is a good start, and will be very useful for Q8 and so forth.

With that I'll stop and thank you, and invite David and Helen to say a few words.

MS. WINKLE: Well, I just want to echo what Ajaz has said. I think that this was actually an excellent discussion. In fact, it was probably some of the best discussion I've heard at any of the advisory committees since I've been here. Your all's contributions were very, very helpful to us, I think, in moving ahead.

I think I may need to be really clear. It's going to take us all a while to get where we need to go. As far as I'm concerned, I guess we've crossed the Rubicon, and we're on the other side, but finding our way now that we're on the other side is going to take time.

And I really feel that there's a lot of contributions that this subcommittee can make to helping us. And I think the idea of having a working group to look at some of the specifics of the framework of where we're going, helping us design that, and helping us address things that are important to industry as we do design that framework is going to be really the crucial part of us finding the direction and moving ahead.

So again, I think it's really been good. I think the people in our review area, as you can see from what Moheb and Gary both had to talk about yesterday, we do understand the need to change. We do understand that we need regulatory flexibility, not only for ourselves but for industry as well. And we've got to find the appropriate ways to do that so that the quality of the product remains at the high level it's at today. So we don't want to just make change for change's sake, but I think that there's a lot to be gained from that.

So again, I want to thank you. I want to again thank Ajaz for putting this together. I think it was a very good agenda. I think it helped stimulate the conversation, and I want to thank David and the people in compliance too for coming and talking about some of the issues on that part of the whole product quality. I think this is a big continuum, from review through the compliance, through the whole life cycle of the product, and working with Compliance has been very valuable to us as we move forward. Thank you.

MR. HOROWITZ: I don't have much to add other than to echo in expressing my gratitude to the committee for the comments that we got. And I hope that you'll consider submitting written comments, or even calling me up informally to give me your views that you weren't able to express during this forum. And in particular, in the September announcement, there will be a brief white paper that expresses some of these same ideas. And that will be another opportunity to solicit comments. So I hope you'll take advantage of that. Thank you very much.

DR. HUSSAIN: I have to thank and recognize Bob King. I mean, he -- this was the first meeting he took on fully himself, and I was three weeks on vacation. So I think without Bob King's help, we really could not have put it together.

(Applause.)

CHAIRPERSON BOEHLERT: Okay. Thank you for that excellent summary, Ajaz, and for your kind comments on the committee's deliberations. I'd also like to thank all the committee members for very active participation. I also think it was a good meeting, and look forward to further discussion on many of these same topics as we go down the road. So just in closing, I'd like to wish you all good travel to wherever your destination may be, and we'll see you all next time. Enjoy your summers.

(Whereupon, the foregoing matter went off the record at 4:01 p.m.)