Medical Device Manufacturer's Survey Evaluation of FDA Quality System/Good Manufacturing Practice Inspections

Posted: 2001-04-26

MEDICAL DEVICE
INSPECTION EVALUATION

Medical Device Industry Initiatives Task Force

Part 1

Anita L. Iannucci, Ph.D.
The UC Irvine Center for Statistical Consulting
University of California
Irvine, California 92697-5105
(949) 924-1682 voice
(949) 824-1683 fax
iannucci@uci.edu

Contents

Part I

INTRODUCTION
     Survey Goals
     Survey Procedure
     Response Rate
     About the Respondents

I.  ALL RESPONDENTS COMBINED
     Reason for the Inspection
     Before the Inspection Began
     During the Inspection
     Outcome of the Inspection
     After the Inspection
     Overall Evaluation of Inspection

Tables
     Table 1
     Table 2
     Table 3
     Table 4
     Table 5
     Table 6
     Table 7
     Table 8
     Table 9

Part II

II.  BREAKDOWN BY REGION

     Before the Inspection Began
     During the Inspection
     Outcome of the Inspection
     After the Inspection
     Overall Evaluation of Inspection

III.  LENGTH OF VS. REASON FOR THE INSPECTION

IV.  INSPECTION OUTCOME COMPARISONS

V.  FROM THE MEDICAL DEVICE INDUSTRY INTITIATIVES GRASSROOTS TASK FORCE

APPENDIX A: THE QUESTIONNAIRE COVER LETTER

APPENDIX B: THE QUESTIONNAIRE

Tables
     Table 10
     Table 11
     Table 12
     Table 13
     Table 14
     Table 15
     Table 16
     Table 17
     Table 18
     Table 19
     Table 20
     Table 21

INTRODUCTION

        This report presents an analysis of the 559 surveys returned from the beginning of March 1999 through the end of April 2000. The report consists of (1) this introduction, (2) the tabulated responses to each survey question and a few comparisons between questions, (3) a comparison of responses across FDA Regions, (4) a comparison of responses between firms with shorter versus longer inspections, (5) a comparison of responses between larger and smaller firms, and (6) a comparison of responses according to inspection outcome – NAI, VAI, or OAI. The questionnaire and cover letter are in the appendices.

Survey Goals

        The Medical Device Industry Initiatives Grassroots Taskforce, composed of representatives from the FDA and medical device industry organizations, sponsored this survey. The survey’s purpose was to determine how satisfied medical device firms are with the current FDA inspection process, discover if and where there are any problems with the process, and to foster communication between industry and the FDA.

Survey Procedure

        At the close of all pre-market and QS/GMP inspections that began between March 1, 1999 and February 29, 2000, the FDA investigator gave a survey packet to a company representative. The survey packet included a questionnaire, a cover letter to the company, signed by FDA officials, industry representatives, and myself, that explained the questionnaire’s purpose, and a postage-paid return envelope (foreign companies received reply envelopes without postage). The boxed portion on the first page of the questionnaire was to have been completed by the investigator before giving the packet to the firm. This area asked the name, address, and phone number of the firm, the product(s) inspected, the inspection’s start and end dates, the investigator(s) name(s), which district performed the inspection, whether or not an FDA 483 was issued, and the reason for the inspection.

        The firms mailed their completed questionnaires directly to me at the UC Irvine Center for Statistical Consulting. I oversaw the data entry and analyzed the results. I also fielded questions from companies concerned about confidentiality. The cover letters to the companies and the outer and reply envelopes were all on UCI stationary.

1

Response Rate

        I estimate the response rate for both domestic and foreign inspections to be at about 40%. Complete FDA inspection records were not available so this figure cannot be calculated more precisely.

About the Respondents

        This section shows the percentage of respondents from each district and the firm sizes of the respondents.

        Table 1 shows the number and percentage of responses by district. Foreign inspections were always classified as foreign rather than as from the district that performed the inspection. Investigators did not always complete the top portion of the questionnaire, the area inside the blue box. Some questionnaires arrived with this area blank, and sometimes the companies completed it themselves. When the FDA district was blank, the address of the company was used to determine which district most likely performed the inspection. In cases where there was also no address, the envelope’s postmark was used to determine the location.

2

        Table 1 shows that foreigh inspections comprise about 23% of the returned surveys and domestic inspections comprise the other 77%.

        The questionnaire asked the number of people the firm employs in the medical device division (question 20). The median firm size was 100 medical device employees meaning that half the responding firms had 100 or fewer such employees and half had 100 or more. Table 2 shows the distribution of firm size for all responding companies.

3

4

I.  ALL RESPONDENTS COMBINED

Reason for the Inspection

        Inspections were classified as preapproval, QS/GMP, or other. The reason for inspection was not known for the nine companies with blank FDA boxes and no identifying information on the back of the questionnaire where a telephone number was requested. For the remaining 550 companies, 12% were inspected for preapproval, 83% for QS/GMP, and 16% for other. The reason the total for these figures exceeds 100% is because some inspections were for multiple purposes.

Table 3 breaks down the reason for inspection, whether a single reason or multiple.

        Table 3 shows that the overwhelming majority of the respondents (73%) were inspected for QS/GMP only and just 10% were inspected for multiple reasons.

Before The Inspection Began

        This section covers responses to questions 1 through 3 and relates to events that occurred before the inspection began.

5

        Responses to question 1 show that 85% of the domestic firms and over 99% of the foreign firms (all but one foreign firm) reported receiving at least some advance notification of the inspection.

        Of those who said they did receive advance notice, a follow-up question asked the number of days’ notice they received. Table 4 shows the descriptive statistics for the number of days of preannouncement for both domestic and foreign inspections. It shows that the preannounced domestic inspections received an average of a week’s notice and the preannounced foreign inspections averaged about eight weeks’ notice.

Table 4.  Number of Days Advance Notice:
Descriptive Statistics (Q-1a)
(for Firms that Received Advance Notice)

        Table 5 presents the frequency distribution of the number of days’ advance notice received for all who received advance notice.

6

Table 5.  Number of Days’ Advance Notice:
Frequency Distribution (Q-1a)
(for Firms that Received Advance Notice)

        The table shows that about 10% of the domestic inspection respondents who received advance notice received only one or two days’ notice, but 69% received five or more days’ notice. No foreign respondents received fewer than six days’ notice, and over half received 50 or more days.

        Additionally, the percentage of all respondents who received five or more days’ advance notice was calculated. Of the domestic inspection respondents, 58% reported advance notice of five or more days; the remainder reported either no advance notice or advance notice of fewer than five days. All but one of the foreign inspection respondents, thus 99%, reported having received advance notice of five or more days.

        For those 494 firms that reported a single reason for inspection (38 preapproval, 399 QS/GMP, and 57 other), the responses to question 1 were also examined according to type of inspection. All firms receiving a preapproval inspection reported receiving at least one day’s advance notice of the inspection, 84% of the domestic QS/GMP inspections were with advance notice, and about 82% of the domestic “other” inspections reported receiving advance notice.

7

        Of those 491 firms that reported having advance notice of the inspection, responses to question 2, regarding the clarity of inspection requirements, were also tabulated. Of those who received preannouncement, approximately 78% of the respondents felt clear about the products needed for the inspection, 74% felt clear about the records needed, and 66% felt clear about the personnel needed.

        Firms were also asked whether it was necessary to reschedule the start date of their inspections and, if so, how this impacted their business. Twenty-seven percent reported that their start date had been rescheduled (question 3). Of this 27%, 58% said the change was helpful to their firm and only 8% said it was disruptive.

During the Inspection

        This section shows summary results about the length of the inspections and various things that may have happened during the inspection process, including interruptions, personnel and records availability, and behaviors of the investigator.

        Since the questionnaire does not ask the length of any interruptions during the inspection, just start date, end date, and whether or not any interruptions exceeded two days, the length of the inspections cannot be accurately calculated. For the estimate below, firms that reported a greater than two day interruption are excluded. Thus, inspection length in the table and figures below is (1) overestimated by a maximum of two days, and (2) biased a bit toward the shorter inspections because longer inspections more often have long interruptions and thus are more often excluded from all length of inspection calculations and tables. The mean of the 488 inspections for which length can be estimated is 4.1 days, the median is 4 days, and the standard deviation is 3.7 days. The distribution of inspection lengths is shown in Table 6.

8

Table 6.  Length of Inspection
(for Those Not Interrupted by More than Two Days)

        Table 6 shows that 16% of the inspections lasted only a day and only 4% lasted more than ten days.

        Responses to question 4 showed that 10% of all inspections were interrupted for more than two working days. For the 130 inspections with start and end dates separated by more than five working days, 35% reported interruptions of more than two days. Of the interrupted inspections, firms reported that 69% of the interruptions were requested by the FDA, 22% by the firms, and 9% by both. When the FDA requested the interruption, firms generally found the interruption either neutral (40%) or disruptive (37%). Not surprisingly, when the firm requested the inspection interruption, they found the interruption either helpful (58%) or neutral (42%).

        As for meeting the needs of the investigators, 92% of respondents said they were able to have all the right personnel available (question 5). When the personnel were not available, generally they were out of town (54%), or home for medical reasons (10%). Ninety-one percent of the respondents said they had all the right records available (question 6). Records were most often unavailable because they were stored off the premises (35%) or lost (30%).

        The next two questions asked about the communication between the investigator and the firm. Question 7 asked if the firm was always notified daily of the investigator’s observations. Since this question only applies to inspections of more than one day, the 79 one-day inspections and 13 inspections without start or end dates were excluded. Of the remaining 465 respondents, 88% said they were notified daily of the investigator’s observations. The most frequent comments from those who said they were not notified daily included being notified less often than daily (22%), being notified the last day only (16%), and being notified only upon request (20%).

        Ninety-three percent of the respondents said the investigator gave helpful information or suggestions (question 8).

9

Outcome of the Inspection

        The next set of questions asks about the outcome of the inspection, specifically, details about the FDA 483s received.

        Fifty-four percent of the firms reported receiving an FDA 483 at the close of the inspection (question 9). Only responses from this 54% (n=303) are included in figures given in the remainder of this section and in all but one of the items in the next section.

        With regards to the FDA 483 observations, 95% of the respondents thought all their 483 observations were understandable (question 15). Table 7 summarizes the errors respondents saw in their FDA 483s.

Table 7.  Inappropriate or Inaccurate
FDA 483 Observations (Q-14, 16)?

        As shown above, 80.5% of the firms said there were no inappropriate (as opposed to inaccurate) observations on their 483s (question 16). Those who felt that there were inappropriate observations on their 483s were asked a follow-up question about why they believed the observations were inappropriate. Seventy-nine percent cited difference of interpretation, 39% cited “insignificant observation,” and 16% cited “other.” These figures sum to more than 100% because respondents were allowed to select more than one response to this question. As for other problems with the 483s, 11% of respondents reported that they had at least one non-annotation related inaccuracy (question 14). The table also shows that 26% of the respondents felt that there was at least one error on their FDA 483: either an inappropriate observation (question 16), or a non-annotation related inaccuracy (question 14), or both. Only 5% reported both non-annotation related inaccuracies and inappropriate observations.

10

        Regarding corrective actions in response to the FDA 483 observations, 11% of the firms took, 34% promised, and 48% both took and promised corrective actions (question 10). Of the firms that took corrective actions, 90% said that all corrective actions which could have been verified by the investigator had been. Of the firms that promised corrective actions, over 99% either had already or planned to fulfill these promises (question 12). Of the companies that took and/or promised corrective actions in response to an FDA 483, 87% reported that all of their actions were annotated on the 483, 5% believed that some of their actions were omitted, and 8% reported that none of their actions were annotated (question 13). All but one firm either had already or planned to fulfill their promised actions (question 12).

After the Inspection

        The information in this section pertains to the closeout meeting and the firm’s planned responses to their FDA 483 observations.

At the final discussion between the investigator and the firm’s management, the firm’s highest level executive was present 76% of the time. (This question, number 19, pertained to all respondents.)

Ninety-three percent of the firms that received 483s planned to respond to them in writing (question 17). Most of those who did not plan a written response said it was not necessary (question 17a).

Overall Evaluation of Inspection

When asked how this inspection compared with previous inspections, slightly more than half (52%) of those who had experienced inspections previously thought this inspection was better, 40% thought it was about the same, and only 8% thought this inspection was worse than previous inspections (question 18).

Respondents who said this inspection was better or worse were asked to explain why. Table 8 shows the tabulated responses from those who said this inspection was better. The most often cited reason was the investigator’s attitude, approach, or personality. The QSIT method of inspection and good communication between the investigator and the firm were also frequently mentioned.

11

Table 8.  Why This Inspection Was Better (Q-18)

*Some respondents gave multiple reasons                                                  

        Those who said this inspection was worse than previous inspections gave a variety of reasons as shown in Table 9. Two explanations were given slightly more often than others:  the inspection was too long and the investigator was not knowledgeable.

Table 9.  Why This Inspection Was Worse (Q-18)

*Some respondents gave multiple reasons                                                  

12

        Additionally, about 33% of respondents wrote comments in the lined section at the end of the questionnaire. The vast majority of these comments pertained to the evaluated inspection as well. About 29% of the end comments were positive about the evaluated inspection; most of these mentioned aspects of the investigator’s knowledge and attitude (23%), preannouncement (3%), and QSIT (2%). About 22% of the end comments were negative about the evaluated inspection. Here most respondents mentioned the length of the inspection, its efficiency, focus, or scope (8%), the investigator’s attitude or lack of knowledge (4%), that advance notice would have allowed them to adequately prepare for the inspection (2%), or confusion over QSIT (2%). Other negative comments were widely varied and could not be classified. Another 14% of the comments were specific suggestions for the FDA:  help the firms prepare by providing an agenda and checklist; try to consolidate inspections or at least understand the requirements of other inspections; suggestions on how to shorten the inspection time and minimize disruption to firms; and other ways the FDA can be more helpful to the firms. Most of the remaining end comments were elaborations of the firm’s responses to specific questionnaire items. All of the comments from all sections of the questionnaire have been stripped of any specifics that might possibly allow company identification, and have been forwarded to the FDA for their consideration. See section V of this report for more details.

13

Link to Part 2