Medical Device Manufacturer's Survey Evaluation of FDA Quality System/Good Manufacturing Practice Inspections

Posted: 2001-04-26

MEDICAL DEVICE
INSPECTION EVALUATION

Medical Device Industry Initiatives Task Force

Part 2

Anita L. Iannucci, Ph.D.
The UC Irvine Center for Statistical Consulting
University of California
Irvine, California 92697-5105
(949) 924-1682 voice
(949) 824-1683 fax
iannucci@uci.edu

Contents

PART II.  BREAKDOWN BY REGION
     Before the Inspection Began
     During the Inspection
     Outcome of the Inspection
     After the Inspection
     Overall Evaluation of Inspection

III.  LENGTH OF VS. REASON FOR THE INSPECTION

IV.  INSPECTION OUTCOME COMPARISONS

V.  FROM THE MEDICAL DEVICE INDUSTRY INTITIATIVES GRASSROOTS TASK FORCE

APPENDIX A: THE QUESTIONNAIRE COVER LETTER

APPENDIX B: THE QUESTIONNAIRE

Tables
     Table 10
     Table 11
     Table 12
     Table 13
     Table 14
     Table 15
     Table 16
     Table 17
     Table 18
     Table 19
     Table 20
     Table 21

[ Link to: Part 1 ]

INTRODUCTION
     Survey Goals
     Survey Procedure
     Response Rate
     About the Respondents

I.  ALL RESPONDENTS COMBINED
     Reason for the Inspection
     Before the Inspection Began
     During the Inspection
     Outcome of the Inspection
     After the Inspection
     Overall Evaluation of Inspection

Tables
     Table 1
     Table 2
     Table 3
     Table 4
     Table 5
     Table 6
     Table 7
     Table 8
     Table 9

II.  BREAKDOWN BY REGION

        In this section many of the results from Section I are compared across the five FDA regions. The original plan was to examine responses by district, but many of the districts had few respondents and low response rates, both of which make results very unstable. One more respondent with very different answers could have considerably changed that district’s breakdown.

Note that all foreign inspections are listed in this section as foreign rather than as from the region that inspected them.

Before the Inspection Began

        All but one of the foreign companies reported receiving advance notice (preannouncement; question 1). As for domestic inspection preannouncement, those firms that reported at least one day of advance notice were:  90% of both the Southwest and Central Regions’ inspections; 85% of the Pacific and Southeast Regions’ inspections; and 72% of the Northeast Region’s inspections.

        Additionally, the percentage of respondents who received five or more days’ advance notice was calculated. Sixty-six percent of the Southeast Region’s respondents; 65% of the Pacific Region’s respondents; 60% for the Central Region; 47% for the Southwest Region; 45% for the Northeast Region; and 99% of foreign inspection respondents reported preannouncement of at least five days. The remaining respondents reported either no advance notice or advance notice of fewer than five days. Again, the reason for inspection may be a factor in these regional differences.

        Those respondents who received advance notice of the inspection were asked whether they felt clear about the products, records, and personnel inspection requirements (question 2). The following figures include only firms that reported receiving at least one day’s advance notice. The Pacific and Southeast’s firms most often expressed clarity about product inspection requirements, 87% and 83% of the time, respectively. For both the Southwest and Central Regions’ firms, 78% reported clarity. Clarity was least often found with the Northeast Region’s firms, 70%, and the foreign firms, 74%.

For record inspection requirements, question 2b, the Southeast Region’s inspections again reported a high percentage of clarity; 83% of their firms reported that they understood what the record inspection requirements were. Most of the other regions had nearly as high a percentage of their respondents report clarity:  in the Central Region, 82% were clear; in the Pacific Region, 80% were clear; and in the Southwest Region, 78% were clear. Again the Northeast (57%) and foreign (64%) inspections showed the smallest percentages of clarity from the preannouncement phone call.

14

        As for the clarity of personnel inspection requirements, again the Pacific and Southeast Regions’ respondents very often reported clear requirements, 75% and 74% respectively. Sixty-seven percent of the firms in the Central Region reported clarity of personnel inspection requirements, as did 65% of the Southwest Region’s respondents. Again, the Northeast and foreign respondents reported the least clarity, with 55% and 62% respectively.

Respondents were asked in question 3 if it was necessary to reschedule the start of the inspection. Foreign inspection start dates were rescheduled the most often, at 33% of the time. For domestic inspections, the Pacific Region rescheduled start dates nearly as often, 29% of the time, followed by the Central Region, 27%, the Northeast, 25%, the Southeast, 21%, and, finally, the Southwest with only 15% of their inspection start dates rescheduled.

During the Inspection

        The next table shows the number of working days the inspections lasted, broken down by region. As explained in Section I, the length of the inspection for those firms that had their inspections interrupted by more than two working days cannot be estimated. Those firms are therefore excluded from the table below.

Table 10.  Length of Inspection

                 Length of Inspection
                  (If Not Interrupted by More than Two Days)

15

        Table 10 shows that, of the uninterrupted inspections, the Northeast had the highest percentage of inspections that ran fewer than four days (72%). Although more of the longer inspections were omitted from the table because they were more often interrupted, the Southeast Region’s inspections seem to have run longer than average:  36% lasted more than a week and only a third lasted fewer than four days. Note that none of the foreign inspections lasted longer than a week.

        The percentage of inspections with interruptions of longer than two working days ranged from none with the foreign inspections to 23% with the Pacific Region’s inspections. Fifteen percent of the Southwest Region’s respondents reported that their inspections were interrupted; 12% of both the Northeast and Central Regions’ inspections were interrupted for longer than two days. The domestic region with the smallest percentage of respondents reporting two or more day interruptions was the Southeast Region, with only 6% of its inspections interrupted.

        The above mentioned interruptions resulted from FDA request (as opposed to firm request or both) roughly two-thirds of the time (60% to 77%) in each district (question 4a). More specific percentages are not provided because the numbers are small and thus the percentages are unstable.

        Respondents were asked in question 5 whether they were able to have all the right personnel available during inspection. Responses were virtually constant across districts with the Northeast at 94%, Central and Southwest at 93%, foreign at 91%, Southeast at 90%, and Pacific at 89%.

Whether or not all necessary records were available to the investigators (question 6) varied a bit across regions. The Southeast Region’s respondents reported that they were able to provide all the records needed for the inspection in 97% of the cases. Other regions’ respondents were able to provide all the records a bit less often:  93% for the Northeast; 91% for Central and foreign; 86% for the Pacific; and 85% for the Southwest. Clearly, the vast majority of all regions’ respondents were in agreement that they were able to meet the investigators’ needs.

Daily notification of the investigator’s observations (question 7) was also tabulated by region and is shown in Table 11. Since this question only applies to inspections of more than one day, only such inspections are included in the table below.

16

Table 11.  Was the Firm Always Notified Daily
of the Investigator's Observations (Q-7)?
(for Inspections Longer than One Day)

        The foreign firms reported most often that they were always notified daily of the inspector’s observations. Three regions, the Northeast, Pacific, and Central, each had about 90% of their responding firms with inspections of more than a day report that they were notified daily of observations. The smallest percentage of reported daily notifications was for the Southeast where 76% reported having been notified daily. Note that since the numbers in Table 11 are a bit smaller than in other tables, the percentages are a bit less stable.

        The final question examined in this section compares opinions of whether the investigator gave any helpful information or suggestions (question 8) across regions. The foreign firms most often found their investigator to be helpful with 98% reporting that the investigator gave helpful information or suggestions. For the five domestic regions, 96% of the Southeast respondents, 93% of the Central, 90% of the Southwest, and 89% of the Pacific and Northeast respondents reported that the investigator gave helpful information or suggestions.

Outcome of the Inspection

        This section contains comparisons across regions of corrective actions taken and promised by the firms and possible problems with their 483s. First, Table 12 shows the firms’ responses about whether they received an FDA 483 (question 9).

17

Table 12.  Issuance of an FDA 483

        As shown, roughly 50 to 70% of each region’s inspections received 483s. Proportionately more 483s were given in the Southwest and fewer in the Northeast and Central regions, but the differences were not striking.

        Only the 303 respondents who were issued 483s are included in the remainder of this section.

        Question 15 asked the firms if all the observations on their FDA 483s were clear. Table 13 shows their responses broken down by region.

18

Table 13.  Clarity of the
FDA 483 Observations (Q-15)

        The table shows that firms overwhelmingly thought the observations on their 483s were understandable, though the Southwest region’s respondents were clear about the observations a bit less often.

        Respondents were also asked if they found any FDA 483 observations to be inappropriate, as opposed to inaccurate (question 16). Mostly firms felt the 483 observations were appropriate. Differences between the regions were relatively small – all were in approximately the 70 to 80 percent range for accuracy (a “no” response). Foreign respondents most often felt all their 483 observations were appropriate, as 91% of foreign firms gave a “no” response to the question. Eighty-four percent of the Northeast Region’s respondents said that all their FDA 483 observations were appropriate, as did 81% of both the Southwest and Central Regions’ respondents, 72% of the Southeast’s respondents, and 69% of the Pacific Region’s respondents.

        Table 14 shows the percentage of respondents with any errors on their FDA 483s. As in Table 7, Section I, it summarizes responses to questions 14 and 16 so that any respondent who felt that they had a problem on their 483 with a non-annotation related inaccuracy or an inappropriate observation is classified as a “yes.” Corrective action annotation problems (question 13) and lack of clarity (question 15) were again not considered errors.

19

Table 14.  Were There One or More FDA 483
Inaccuracies or Inappropriate Observations (Q-14, 16)?

        The most satisfied respondents were from the Foreign and Northeast Region’s inspections. The Pacific and Southeast Regions’ respondents were the least satisfied, with about a third of their respondents reporting at least one non-annotation inaccuracy or inappropriate observation on their 483s.

        Next we turn to problems with the firms’ promised and/or taken corrective actions. The vast majority of firms (85% to 98%) in all regions either promised or took corrective actions. Question 11 asked the firms if any of the corrective actions they took could have been verified by the inspector but were not. Again, the vast majority from all regions reported that their investigator had acted appropriately. Ninety-five percent of the foreign respondents whose firms took corrective actions said that the investigator had verified all their corrective actions. For the domestic respondents, 94% of the Northeast Region’s respondents who took corrective actions said that all that could have been verified were verified. This figure was 92% for the Southwest Region, 90% for the Southeast Region, 87% for the Pacific Region, and 86% for the Central Region. In other words, there are slightly fewer complaints about investigators not verifying corrective actions in the Northeast, and slightly more complaints in the Central and Pacific regions. However, since this question only pertained to those who responded “yes” to having taken corrective actions in question 10, the numbers are small and therefore less stable than most of the other percentages given in this report.

20

        Question 13 asked if the promised and/or taken corrective actions were properly annotated on the FDA 483. Again, the overwhelming majority of firms reported that they were. For the Northeast Region’s respondents, 91% of the firms that took or promised corrective actions reported that all their actions were properly annotated on the 483 and an additional 3% reported that some were properly annotated. For the Southwest Region’s respondents, 91% of the firms that took or promised corrective actions reported that all were properly annotated on the 483. No one from the Southwest Region reported partial annotation. Eighty-eight percent of the Central Region’s respondents reported full annotation, and an additional 5% reported partial annotation. Similarly, eighty-seven percent of the Pacific Region’s respondents reported full annotation, and an additional 6% reported partial annotation. For the Southeast, 79% reported full annotation and another 14% reported partial. Foreign respondents again were among the most content:  88% of those who took or promised corrective actions reported full annotation and 3% reported partial. Although these figures show that respondents criticized the Southeast investigators’ use of the annotation process a bit more often than those from other regions regarding failure to fully annotate corrective actions on the FDA 483, this difference was small when the instability of the small numbers is taken into account.

After the Inspection

        This section compares responses about things that happened at the close of the inspection and afterwards.

        Seventy-six percent of all 559 firms reported that their highest level executive was present at the final discussion between the investigator and the firm’s management, the closeout meeting (question 19). The breakdown according to region was:  86% Pacific; 83% foreign; 81% Southeast; 70% Northeast; 69% Central; and 65% Southwest.

        Of the 303 firms that received FDA 483s, most regions had over 90% of their responding firms report that they planned to respond to the FDA in writing (question 17):  96% of the Southeast firms; 94% of the Central and Pacific firms; 93% of the foreign and Southwest firms; and 84% of the Northeast firms.

Overall Evaluation of Inspection

        Table 15 shows the respondents’ opinions about how this inspection compared with previous inspections, for those who had experienced previous inspections.

21

Table 15.  How Did This Inspection Process
Compare with Previous Inspections (Q-18)?

          How Did This Inspection Process
          Compare with Previous Inspections?

        As shown, the foreign and Central Region respondents were the most positive about this inspection with nearly 60% of each reporting that this inspection was better than previous. The Southwest Region’s respondents most often reported that this inspection was worse than previous – about 16% said this inspection was worse as compared with about 8% to 10% of respondents in the other four regions. This may be related to the fact that the Southwest Region firms were issued a slightly higher percentage of 483s (Table 12).

22

III.  LENGTH OF VS. REASON FOR
THE INSPECTION

        In this section, the number of working days the inspection lasted is compared with the type of inspection.

        Recall that the length of inspection is difficult to estimate. The questionnaire does not ask the length of any interruptions during the inspection, just whether there was an interruption that exceeded two days. Thus, firms that reported an interruption of more than two working days were excluded from all “length of inspection” calculations and thus from this section of the report. Note that firms with interruptions of only one or two working days are included in this section and that their length of inspection is not adjusted downward for the interruption. Thus, (1) the longest inspections are underrepresented in this section and (2) the length of inspection is slightly overestimated.

        Length of inspection was broken into five categories based on the number of working days:  1 day, 2 or 3 days, 4 or 5 days, 6 to 10 days, and more than 10 days. Table 16 shows the comparison between purpose of the inspection and the number of days the inspection lasted.

Table 16.  Reason for the Inspection
by Length of Inspection

          Length of Inspection
          (If Not Interrupted by More than Two Days)

23

        The table shows that inspections performed solely for preapproval generally lasted 4 to 5 days (the median was 4 days), QS/GMP inspections generally lasted 2 to 5 days (the median was also 4 days), and inspections performed primarily for other reasons were often shorter, generally 2 or 3 days (the median was 3 days).

24

IV.  INSPECTION OUTCOME
COMPARISONS

        The inspection outcome was provided by the FDA for 306 domestic inspections, about 71% of the domestic inspections covered in this report. Outcomes were classified as NAI, either no FDA 483 or no substantive FDA 483; VAI, substantive FDA 483 but no Warning Letter; or OAI, Warning Letter or worse was received by the firm. Table 17 shows the breakdown of these outcomes for the 306 domestic firms.

Table 17.  Current Inspection Outcome

        In this section, inspection outcome is compared with the number of medical device employees in the firm, preannouncement, length of the inspection, whether the firm’s highest level executive attended the closeout meeting, and how this inspection compared with previous.

        Table 18 shows that there is little relationship between the firms’ number of medical device employees and the three inspection outcomes except that the largest firms were a bit less likely to be OAI.

25

Table 18.  Current Inspection Outcome
by Firm’s Number of Employees (Q-20)

        The majority of the firms inspected, whether they received advance notification or not, were NAI or VAI, as shown in Table 19.

Table 19.  Current Inspection Outcome
by Preannouncement (Q-1)

        The inspection outcome and length of the inspection were related. Table 20 below shows results consistent with those in Section III of this report:  the more serious the problems found the longer the inspection. This table shows the 267 firms for which both outcome results were available and length of inspection could be estimated.

26

Table 20.  Current Inspection Outcome
by Length of Inspection
(if Not Interrupted by More than 2 Days)

        Finally, the inspection outcome was compared with whether the respondent felt this inspection was better or worse than previous inspections. Table 21 shows that there is a weak relationship between how much the respondent liked this inspection and how favorable its outcome.

Table 21.  Current Inspection Outcome by
How Did This Inspection Process Compare
with Previous (Q-18)?

        It is interesting to note that 12 of the OAI respondents felt this inspection was better than previous and two of the NAI respondents felt this inspection was worse than previous. Neither of those two reported having received an FDA 483. One said that this inspection was worse because it took more time, the other said it was worse because employees had to cancel vacations to accommodate the inspection schedule.

27

        For the 12 respondents with very unfavorable inspection results who were still very positive about the inspection, four cited the positive attitude of their investigator (“understanding,” “helpful,” “courteous,” “flexible,” “knowledgeable”), two liked the comprehensive nature of the inspection, one liked that it was shorter and more focused, one mentioned QSIT’s predictability, two mentioned good communication and daily wrapups, and one of those two also mentioned preannouncement.

28

V.  FROM THE MEDICAL DEVICE
INDUSTRY INITIATIVES
GRASSROOTS TASK FORCE

Task Force Subcommittee Members:
Nancy Singer, AdvaMed and Denise Dion, FDA
Lauren Andersen, AdvaMed & Andersen Caledonia Ltd.
Elaine Messa, Quintiles Consulting & Former Director of Los Angeles District Office, FDA
Leif Olsen, AMDM & BioWhittaker
Susan Reilly, ASQ Biomedical Division & Reilly and Associates

        The Medical Device Industry Initiatives Grassroots Task Force wishes to thank the FDA officials who coordinated the distribution of the Medical Device Inspection Evaluation, and the industry officials who took the time to fill out the questionnaire and return it to the University of California, Irvine, Center for Statistical Consulting.

        We feel that this survey has been valuable in that it (1) provided firms an opportunity to give anonymous feedback to the FDA and to industry about their inspection experience; (2) allowed comparisons across regions of companies' reactions to inspections; and (3) helped determine if the medical device industry initiatives (pre-announced inspections, annotated 483s, etc.) were being followed.

29

        Feedback to the FDA consisted of (1) a quantitative analysis of the survey results and (2) the many comments which respondents wrote on the questionnaires. The FDA Office of Regulatory Affairs and Center for Devices and Radiological Health management have received a thorough report of the analyses of the questionnaire data. They have also been provided with all the many comments respondents wrote, both short comments written in response to specific questions and longer comments written at the end of the questionnaire. Before forwarding them, comments were stripped of any specifics which might possibly have allowed identification of the company, including FDA district and region, dates of inspection, product being inspected, inspection outcome, and anything unique about the inspection, etc. Comments were typed and categorized according to content by the UC Irvine Center for Statistical Consulting before being forwarded to the FDA.

        Regional differences appear to be minimal, but the Office of Regulatory Affairs is continuing its ongoing efforts to assure uniformity and consistency in inspections and enforcement.

        In light of the fact that 52% of firms believed the inspection was better than previous inspections, the Committee believes that the medical device industry initiatives of pre-announcing inspections and annotating Form FDA 483s are causing the medical device industry to view the inspection process in a more positive light than it has in the past. The Committee was pleased that most inspections are pre-announced, and 58% of the domestic and 99% of the foreign companies were given five or more days of advance notice before the start of an inspection.

        The actual questionnaires have now been shredded by the UC Irvine Center for Statistical Consulting. Only the electronic data file remains, and it has been stripped of all fields that might allow identification of respondents, including region, district, dates of inspection, and all comments.

30

APPENDIX A:
THE QUESTIONNAIRE COVER LETTER

        The FDA investigator gave the questionnaire (Appendix B), a reply envelope addressed to Dr. Anita Iannucci at UC Irvine, and the following cover letter to the firm’s representative at the close of inspection. The cover letter was printed on UC Irvine stationary – white paper with blue and black ink. All three items were together in a UC Irvine stationary envelope without any addressee, but with a UC Irvine return address.

31

Questionnaire Cover Letter  (PDF 79KB)

32

APPENDIX B:
THE QUESTIONNAIRE

The questionnaire was printed in the form of a booklet, two double-sided sheets of blue ink on white paper. The FDA investigator was to have filled out the box at the top of page 1 before giving the questionnaire packet to the firm, but in actual practice this was not always done. When the investigator left the box empty, often the firm filled in the information. When it was returned blank, my assistant telephoned the firm to obtain the information. We did not contact the FDA for the data because the FDA was not to know which firms returned their questionnaires and which did not.

33

Medical Device Inspection Evaluation Questionnaire (PDF 148KB)

34

Link to Part 1