HHS Log--links to Department of Health and Human Services website

Independent Evaluation of FDA's First Cycle Review Performance – Final Report

Exhibit 2: Overview of Study Cohorts and Data Sources

The study consisted of 185 applications submitted from FY2002 – FY2007 and included both PDUFA II (FY2002) and PDUFA III (FY2003 – FY2007). The GRMPs were rolled out in April 2005. The Analysis Cohorts consisted of the Retrospective Analysis Cohort (products submitted FY02 – FY04 with first action by 12/1/04) and the Prospective Analysis Cohort (products submitted between FY05 – FY07 with first action by 9/30/07). The Retrospective Analysis consisted of 77 products. An additional 23 products from FY04 reached action after 12/1/04, which combined with the Retrospective Analysis Cohort, yield a total of 100 products during FY02-04. The Prospective Analysis consisted of 85 products. Among these 85 products, sponsors were interviewed for 20 products, and meeting and Regulatory Project Manager (RPM) input were gathered for 41 products. Regarding data sources, action packages were used for the entire FY02 – FY07 period. FDA systems that support application review, sponsor interviews, review meeting and RPM input, industry focus groups, and FDA focus groups were utilized for applications submitted during FY05 – FY07.

Return to Exhibit 2: Overview of Study Cohorts and Data Sources

Exhibit 4: Drivers and Hypotheses of Multi-Cycle Reviews

Factors	Key Analysis Areas	Sample Hypotheses
Product/Disease Characteristics	Review Designation Mechanism of Action Life Threatening Disease Licensing Status	Implementing GRMPs will contribute to an increase in the first-cycle approval rate and an increase in the transparency of the review process Regulatory Project Manager turnover will result in lower first-cycle approval Products developed by experienced companies are less likely to require multiple review cycles Submissions designated Priority are less likely to require more than one review cycle as compared to those designated as Standard
GRMPs Compliance	Impact of GRMPs
Issues and Communication	Application Quality Issues/Deficiencies and Resolution Postmarketing Commitments Communication Effectiveness/Timing
Sponsor Characteristics	Company Size/Type Country of Origin Experience with FDA Experience in Therapeutic Area
FDA Characteristics	Reviewer/Division Workload Reviewer Experience Staff Turnover Inspections

Notes: Application Quality, Reviewer Division Workload, and Reviewer Experience were hypothesis categories that were not tested because appropriate test indicators or metrics could not be identified or insufficient data existed.

Return to Exhibit 4: Drivers and Hypotheses of Multi-Cycle Reviews

Exhibit 6: Product Application Cohorts, Data Sources, and Analyses

Cohort	Number of Applications	Analyses	Data Collection Method
Cohort	Number of Applications	Analyses	Action Packages	Public Data Sources	FDA Application Review Systems
Study Cohort	185	Review Designation Sponsor Characteristics Product Characteristics Pre-Submission Meetings Filing Review Notification Advisory Committee Meetings Major Review Deficiencies Postmarketing Commitments Review Office Characteristics Manufacturing Inspection	Yes	Yes
Prospective Analysis Cohort	85	Total Review Communications GRMPs Impact FDA Staffing Changes	Yes	Yes	Yes

Exhibit 6: Product Application Cohorts, Data Sources, and Analyses

Exhibit 7: Cohort Product Status and Approval Rates

One hundred eighty-five products were studied (49 BLAs and 136 NDAs). After the first cycle, 93 products (50%) were approved (35 BLAs and 58 NDAs). Fourteen BLA applications (8% of the total products) received a Complete Response Letter. For the rest of the products, 61 NDAs were Approvable (33%) and 17 NDAs were Not Approvable (9%). After an additional cycle, two NDAs were withdrawn after receiving an Approvable Letter and one NDA was withdrawn after receiving a Not Approvable Letter. For those products that received either a Complete Response Letter or an Approvable Letter, 34 (8 BLAs and 26 NDAs) were approved (18%), 6 BLAs (3%) received a Complete Response Letter, 32 NDAs, (17%) received an Approvable Letter, and one NDA (1%) received a Not Approvable Letter. For the 17 NDAs that received Not Approvable Letters in the first cycle, three (2%) were approved and 13 (7%) received Not Approvable Letters in later cycles.

Return to Exhibit 7: Cohort Product Status and Approval Rates

Exhibit 9: Approval Rates vs. Novelty and Indication Life-Threatening

	Indication is not for life-threatening condition	Indication is for life-threatening condition
Does not have novel mechanism of action	39% single-cycle (n=27); 61% multi-cycle (n=43) Total: 100% (n=70)	53% single-cycle (n=27); 47% multi-cycle (n=24) Total: 100% (n=51)
Does have novel mechanism of action	59% single-cycle (n=19); 41% multi-cycle (n=13) Total: 100% (n=32)	62% single-cycle (n=20); 38% multi-cycle (n=12) Total: 100% (n=32)

Source: Public Sources, Action Packages, FY02-07 Cohort

Return to Exhibit 9: Approval Rates vs. Novelty and Indication Life-Threatening

Exhibit 11: First-Cycle Approval Rate by Application Type

	Priority Review	Fast Track	Orphan Status	PDUFA Fees Waived
Yes	67% single-cycle (n=56); 33% multi-cycle (n=27) Total: 100% (n=83)	65% single-cycle (n=33); 35% multi-cycle (n=18) Total: 100% (n= 51)	60% single-cycle (n=25); 40% multi-cycle (n=17) Total: 100% (n=42)	48% single-cycle (n=28); 52% multi-cycle (n=30) Total: 100% (n=58)
No	36% single-cycle (n=37); 64% multi-cycle (n=65) Total: 100% (n=102)	45% single-cycle (n=60); 55% multi-cycle (n=74) Total: 100% (n=134)	48% single-cycle (n=68); 52% multi-cycle (n=75) Total: 100% (n=143)	51% single-cycle (n=65); 49% multi-cycle (n=62) Total: 100% (n=127)

Notes: 43 of the 51 (84%) products designated as fast track were priority applications; Of the 58 that had PDUFA fees waived, 38 had orphan status and 29 were small companies

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Return to Exhibit 11: First-Cycle Approval Rate by Application Type

Exhibit 12: GRMPs Phases and Activities

After application receipt, the Filing and Planning Phase (I) begins which includes the following activities:

1. Review Application
2. Assign RPM
3. Begin Regulatory Filing Review
4. Acknowledge Application Receipt in Writing
5. Assign Review Team
6. Schedule Filing Meeting
7. Request Consults (most frequently used)
8. Identify Inspection Actions
9. Applicant Orientation Presentation (optional)
10. Designate Priority Review
11. Conduct Filing Review
12. Convey Potential RTF Issues to Applicant
13. Identify Signatory Authority
14. Hold Filing Meeting (assessed to measure compliance)
15. Make Filing Decision
16. Conduct Planning Meeting
17. Inform Applicant of a Priority Designation in Writing
18. Communicate Filing Determination to Application, if RTF
19. Communicate Filing Review Issues to Applicant (assessed to measure compliance)

After Day 74, the Review Phase (II)/ AC Phase (III) begins, which includes the following activities:

20. Conduct Review
21. Mid-cycle Meeting (assessed to measure compliance)
22. Complete Primary Review (assessed to measure compliance)
23. Secondary Sign-Off
24. Issue DR Letters, as appropriate
25. Plan Advisory Committee (AC) Meeting
26. Disseminate and disclose applicant and FDA background materials
27. Conduct AC Meeting
28. Internal meeting to integrate AC input
29. Confidential memo to AC to announce action

By the end of Month Eight (Month Five for Priority Applications), the Action Phase (IV) begins, which includes the following activities:

30. Wrap up Meeting
31. Internal Briefings for Signatory Authority (as needed)
32. Preapproval Safety Conference (for NMEs in CDER)
33. Initiate Compliance Check Request (BLAs)
34. Labeling Discussions (for Approval and Approvable Actions) (assessed to measure compliance)
35. Compile Action Packages
36. Draft Action Letter with Conditions for Approval
37. Draft Action Letter with List of Deficiencies
38. Circulate and Review Action Package and Letter
39. Letter to Signatory Authority
40. Action

By the end of Month Ten (Month Six for Priority Applications), the Post-Action Phase (V) begins, which includes the following activities:

41. Conduct Lessons Learned
42. Clarify Deficiencies and Expected Outcomes

Finally, the PDUFA Goal Date is reached.

Notes: The following activities have a red box around them in the exhibit to indicate that they were assessed to measure compliance: 14, 19, 21, 22, 34

Return to Exhibit 12: GRMPs Phases and Activities

Exhibit 13: First Cycle Approval Rate by GRMPs Compliance Rates

	20% or less	40%	60%	80% or more
Single-Cycle	50% (n=7)	41% (n=14)	55% (n=11)	71% (n=12)
Multi-Cycle	50% (n=7)	59% (n=20)	45% (n=9)	29% (n=5)

Notes: One product with 0% GRMPs compliance was not approved in the first cycle; Five products with 100% GRMPs compliance had 60% first-cycle approval rate.

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Return to Exhibit 13: First Cycle Approval Rate by GRMPs Compliance Rates

Exhibit 14: First Cycle Approval Rate by GRMPs Compliance Rates and Review Designation

		20% or less	40%	60%	80% or more
Priority	Single-Cycle	60% (n=6)	67% (n=10)	90% (n=9)	80% (n=4)
Priority	Multi-Cycle	40% (n=4)	33% (n=5)	10% (n=1)	20% (n=1)
Standard	Single-Cycle	25% (n=1)	21% (n=4)	20% (n=2)	67% (n=8)
Standard	Multi-Cycle	75% (n=3)	79% (n=15)	80% (n=8)	33% (n=4)

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Return to Exhibit 14: First Cycle Approval Rate by GRMPs Compliance Rates and Review Designation

Exhibit 15: GRMPs Compliance and FDA-Sponsor Communication Frequency and Early Communication Timing

GRMPs Compliance vs. Sponsor Communication Frequency

Review Designation	Compliance	Average Interactions per Product
Overall	80% or more	51
	60%	56
	40%	48
	20% or less	50
Priority	80% or more	43
	60%	55
	40%	44
	20% or less	53
Standard	80% or more	55
	60%	58
	40%	51
	20% or less	41

GRMPs Compliance vs. Sponsor Communication Timing

Weeks Since Submission	Compliance	Proportion of Products Having Early Communication
First week	80% or more	24%
	60%	20%
	40%	3%
	20% or less	14%
First 2 weeks	80% or more	35%
	60%	45%
	40%	24%
	20% or less	21%
First 4 weeks	80% or more	65%
	60%	70%
	40%	38%
	20% or less	57%

Note: Interactions with sponsor counted all observed communications between review team and the sponsor (e.g., meeting, phone call, email, fax, and etc.)

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Return to Exhibit 15: GRMPs Compliance and FDA-Sponsor Communication Frequency and Early Communication Timing

Exhibit 16: Overview of FDA-Sponsor Communications

This exhibit has a time line that runs from the Pre-IND stage through post-approval, or Phase 4, activities.

Return to Exhibit 16: Overview of FDA-Sponsor Communications

Exhibit 17: Effect on End-of-Phase 2 Meetings on Approval Rate for FY2005 – FY2007 Cohort

EOP2 Meeting

	Yes (n=66)	No (n=19)
Single-Cycle	29 (44%)	15 (79%)
Multi-Cycle	37 (56%)	4 (21%)

Timing of Major Issue Identification for Multi-Cycle Products with EOP2 Meeting

EOP2	16 (43%)
Pre-NDA/BLA	6 (16%)
During Review	15 (41%)

Source: Booz Allen Analysis, Action Packages, FDA Systems, FY05-07 Cohort

Return to Exhibit 17: Effect on End-of-Phase 2 Meetings on Approval Rate for FY2005 – FY2007 Cohort

Exhibit 18: Effect of Pre-NDA/BLA Meetings and Timing on Approval Rate for FY2005-FY2007 Cohort

Pre-NDA/BLA Meeting Held?

	Yes (n=77)	(n=8)
Single-Cycle	39 (51%)	5 (62%)
Multi-Cycle	38 (49%)	3 (38%)

Timing of Pre-NDA/BLA Meeting

	Less than 6 months before submission (n=28)	Greater than 6 months before submission (n=49)
Single-Cycle	71%	39%
Multi-Cycle	29%	61%

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Return to Exhibit 18: Effect on End-of-Phase 2 Meetings on Approval Rate for FY2005 – FY2007 Cohort

Exhibit 19: Incidence of Pre-NDA/BLA and EOP2 Meetings

	FY2002-2004 (n=100)		FY2005-2007 (n=85)
	Pre-NDA/BLA Meeting		Pre-NDA/BLA Meeting
	No	Yes	No	Yes
No EOP2 Meeting	50% single-cycle (n=10); 50% multi-cycle (n=10) Total: 100% (n=20)	45% single-cycle (n=10); 55% multi-cycle (n=12) Total 100% (n=22)	75% single-cycle (n=3); 25% multi-cycle (n=1) Total 100% (n=4)	80% single-cycle (n=12); 20% multi-cycle (n=3) Total: 100% (n=15)
EOP2 Meeting	100% single-cycle (n=1) Total: 100% (n=1)	49% single-cycle (n=28); 51% multi-cycle (n=29) (57% had both meetings) Total: 100% (n=57)	50% single-cycle (n=2): 50% multi-cycle (n=2) Total: 100% (n=4)	44% single-cycle (n=27); 56% multi-cycle (n=35) (73% had both meetings) Total: 100% (n=62)

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Return to Exhibit 19: Incidence of Pre-NDA/BLA and EOP2 Meetings

Exhibit 20: Filing Review Issues Identified by Year

	Issues Identified	No Issues Identified	No Filing Review Notification Sent	Total Number of Applications
2002	0	1	28	29
2003	21	12	2	35
2004	21	13	2	36
2005	21	17	0	38
2006	19	12	1	32
2007	7	8	0	15

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Return to Exhibit 20: Filing Review Issues Identified by Year

Exhibit 21: Impact of Issue Identification in Filing Review Notification for FY2002-FY2007 Cohort

Issues Identified in the Filing Review Notification?
Yes (n=89)	No (n=63)
43% single-cycle (n=38); 57% multi-cycle (n=51)	62% single-cycle (n=39); 38% multi-cycle(n=24)

Note: Of the 185 applications, 33 did not have a Filing Review Notification. The total products included in this analysis is 152.

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Return to Exhibit 21: Impact of Issue Identification in Filing Review Notification for FY2002-FY2007 Cohort

Exhibit 22: Applications with All Filing Review Issues Resolved by Action Date for FY2005-FY2007 Cohort

Yes (n=29)	No (n=18)
62% single-cycle (n=18); 38% multi-cycle (n=11)	22% single-cycle (n=4) (unresolved issues in single-cycle approvals were addressed with PMCs); 78% multi-cycle (n=14)

Note: Of 85 prospective products, one did not have a 74 day letter and 37 had no potential review issue identified, hence this analysis is based on 47 applications that had issues identified

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Exhibit 22: Applications with All Filing Review Issues Resolved by Action Date for FY2005-FY2007 Cohort

Exhibit 23: Approvability Impact of 74 Day Letter Issues

Out of 29 multi-cycle products, ten had no 74-Day issues reported and 19 had 74-Day issues reported. For those that had 74-Day issues reported, six products had all issues resolved and 13 products had issues remain unresolved during review. Of those 13 products that had issues remain unresolved: three products had unresolved that were not the approvability issues, six products had unresolved issues that were part of the approvability issues, and 4 products that had unresolved issues that were the approvability issues. For the six products that had unresolved issues there were part of the approvability issues, two products had approvability issues that seemed to be resolvable by the sponsor and four products had approvability issues that could not be resolved during the review period. For those products that had unresolved issues that were the approvability issues, 2 products had approvability issues that could not be resolved during the review period and two products had approvability issues that seemed to be resolvable by the sponsor. Examples of issues that were approvability issues that seemed to be resolvable by the sponsor include a stability test (CMC), the need for a short-term animal toxicity study (Safety), and stability/acceptance test, facility inspection (CMC). Examples of issues that were approvability issues that could not be resolved during the review period include a lack of evidence, failure to meet pre-specified p-value or non-inferiority margins (Efficacy) and a serious dose-related adverse event (Safety).

Note: Approvability analysis was based exclusively on Booz Allen high level analysis of summary documents; this analysis was generated solely for the purposes of illustration regarding overall process, rather than any specific application

Source: Booz Allen Analysis, Action Packages, FDA Systems; Subset of FY05-06 Cohort

Exhibit 23: Approvability Impact of 74 Day Letter Issues

Exhibit 24: FDA-Sponsor Communications for FY2005-FY2007 Cohort

Cumulative Average First-Cycle Communications During Review Period (Written and Meetings)
Review Cycle Quarter	Cumulative Average Number of Communications (Single-Cycle)	Cumulative Average Number of Communications (Multi-Cycle)
First	5.6	6.6
Second	17.5	16.2
Third	33	31.2
Fourth	56.7	44.9

	Single-Cycle	Multi-Cycle
# of products	44	41
Range	17-158	5-115
Median	49	40
Mean	57	45

73% more communications for single-cycle than for multi-cycle products in the fourth quarter of the review cycle

Notes: Written include letters, faxes, and emails; Meetings include face-to-face meetings and telecons

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Exhibit 24: FDA-Sponsor Communications for FY2005-FY2007 Cohort

Exhibit 25: Communication Type for FY2005-FY2007 Cohort

Communication Types

	Single-Cycle (n=44)	Multi-Cycle (n=41)
Written	1950, 78%	1651, 90%
Telecon	519, 21%	169, 9%
Face-to-Face	25, 1%	21, 1%

Cumulative Average First-Cycle Communications During Review Period
Written
Review Cycle Quarter	Cumulative Average Number of Communications (Single-Cycle)	Cumulative Average Number of Communications (Multi-Cycle)
First	4.5	5.1
Second	14.3	13.2
Third	26.3	25.9
Fourth	44.3	37.5
Telecon
Review Cycle Quarter	Cumulative Average Number of Communications (Single-Cycle)	Cumulative Average Number of Communications (Multi-Cycle)
First	1.1	1
Second	3	1.8
Third	6.2	2.9
Fourth	11.8	3.8
Face-to-Face
Review Cycle Quarter	Cumulative Average Number of Communications (Single-Cycle)	Cumulative Average Number of Communications (Multi-Cycle)
First	0.05	0.07
Second	0.16	0.16
Third	0.5	0.3
Fourth	0.6	0.5

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Exhibit 25: Communication Type for FY2005-FY2007 Cohort

Exhibit 26: Amendments Submitted for Applications for FY2005 Products

Single-Cycle
Individual Applications	Number of Amendments	Standard or Priority
1	5	Priority
2	6	Priority
3	8	Standard
4	12	Priority
5	12	Priority
6	15	Priority
7	16	Priority
8	16	Priority
9	17	Priority
10	18	Priority
11	19	Priority
12	22	Priority
13	28	Priority
14	35	Standard
15	36	Priority
16	37	Standard
17	38	Priority
18	41	Priority
Mean	21.2

Multi-Cycle
Individual Applications	Number of Amendments	Standard or Priority
1	7	Standard
2	8	Standard
3	10	Priority
4	10	Priority
5	10	Standard
6	10	Standard
7	12	Priority
8	13	Standard
9	15	Standard
10	17	Priority
11	18	Standard
12	18	Standard
13	20	Priority
14	20	Standard
15	22	Standard
16	28	Priority
17	28	Standard
18	40	Standard
Mean	17

Source: Booz Allen Analysis, FDA Systems; FY05 Cohort

Exhibit 26: Amendments Submitted for Applications for FY2005 Products

Exhibit 27: Types of Amendments Submitted for FY 2005 Applications

Total Amendments (n=464)

	Number of Amendments
FDA Requested	351 (76%)
Unsolicited	83 (18%)
Unknown	30 (6%)

Types of Amendments

	Number of Amendments
	FDA Requested	Unsolicited	Total
Clinical	127	28	155
CMC	80	14	94
Labeling	31	20	51
Clinical Pharmacology	23	0	23
Pharmacology/Toxicology	16	4	20
Statistics	14	0	14
Microbiology	4	0	4
Multiple	56	17	73

Source: Booz Allen Analysis, FDA Systems; FY05 Cohort

Exhibit 27: Types of Amendments Submitted for FY 2005 Applications

Exhibit 28: Advisory Committee Meetings and Review Designation

	Advisory Committee Meeting
	Yes	No
Priority	63% single-cycle (n=12); 37% multi-cycle (n=7) Total: 100% (n=19)	69% single-cycle (n=44); 31% multi-cycle (n=20) Total: 100% (n=64)
Standard	36% single-cycle (n=5); 64% multi-cycle (n=9) Total: 100% (n=14)	36% single-cycle (n=32); 64% multi-cycle (n=56) Total: 100% (n=88)

Source: Booz Allen Analysis; Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 28: Advisory Committee Meetings and Review Designation

Exhibit 29: Advisory Committee Meeting Timing and Priority/Standard Designation

Timing of Advisory Committee Meetings for Priority Applications

Review Cycle Quarter	Single-Cycle	Multi-Cycle	Total Number of Applications
First	1	0	1
Second	1	0	1
Third	3	2	5
Fourth	7	5	12

Timing of Advisory Committee Meetings for Standard Applications

Review Cycle Quarter	Single-Cycle	Multi-Cycle	Total Number of Applications
First	0	0	0
Second	0	0	0
Third	3	2	5
Fourth	2	7	9

Note: The timeframe of review cycle quarter for a priority review is 0-1.5, 1.5-3, 3-4.5, and 4.5-6 months individually and that for a standard review is 0-2.5, 2.5-5, 5-7.5, and 7.5-10 months separately

Source: Booz Allen Hamilton Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 29: Advisory Committee Meeting Timing and Priority/Standard Designation

Exhibit 30: Impact of Major Issues Identified on Approval Rate in FY2005-FY2007 Cohort

Were Deficiencies Identified Pre-Submission?

	Single-Cycle	Multi-Cycle
Yes (n=43)	17 (40%)	26 (60%)
No (n=42)	27 (64%)	15 (36%)

For Applications with No Deficiencies Documented Pre-Submission, Were Major Issues Identified During Review?

	Single-Cycle	Multi-Cycle
Yes (n=16)	3 (19%)	13 (81%)
No (n=26)	24 (92%)	2 (8%)

Note: Regarding the two applications with no documented deficiencies prior to the Action Letter, a serious adverse event was discovered for one, while sufficient efficacy for approval was not demonstrated for the other. Both deficiencies were first documented in the Action Letter.

Source: Booz Allen Analysis; FY05-07 Cohort

Exhibit 30: Impact of Major Issues Identified on Approval Rate in FY2005-FY2007 Cohort

Exhibit 31: Major Deficiencies Cited in First Action Letter of Multi-Cycle Applications by Category for FY2002-FY2007 Cohort

Issue Category	Number of Applications
One Category (55% of applications) (n=51)
Efficacy	19
Safety	19
CMC	13
Two Categories (35% of applications) (n=32)
Efficacy and Safety	15
Efficacy and CMC	9
Safety and CMC	6
Safety and Format	1
CMC and Format	1
More than Two Categories (10% of applications)	9
Total Applications	92

Note: A major deficiency is defined as an issue that prevented approval

Source: Booz Allen Analysis; FY02-07 Cohort

Exhibit 31: Major Deficiencies Cited in First Action Letter of Multi-Cycle Applications by Category for FY2002-FY2007 Cohort

Exhibit 32: Major Deficiencies Cited in First-Cycle Action Letter of Multi-cycle Applications by Area for FY2002-FY2007 Cohort

	Trial Design	Execution	Endpoints	Application (Format)	Total
Efficacy	18	23	14
Safety	17	16	11
CMC		14
Application				6
Other	4	1
Total					124
Percent	31%	44%	20%	5%	100%

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 32: Major Deficiencies Cited in First-Cycle Action Letter of Multi-cycle Applications by Area for FY2002-FY2007 Cohort

Exhibit 34: Products with Postmarketing Study Commitments in FY2002-FY2007 Cohort

	First-Cycle (n=93)	Multi-Cycle (n=37)
PMC	86% of products (n=80)	84% of products (n=31)
No PMC	14% of products (n=13)	16% of products (n=6)

Note: Analysis cohort was 130, reflecting 55 multi-cycle products had not been approved

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 34: Products with Postmarketing Study Commitments in FY2002-FY2007 Cohort

Exhibit 35: Distribution of PMCs for Single- and Multi-Cycle Approvals

Single Cycle
Individual Application (n=80)	Type of PMC
Individual Application (n=80)	Agreed-Upon	Required	CMC
1	1	0	0
2	0	1	0
3	1	0	0
4	1	0	0
5	1	0	0
6	0	1	0
7	1	0	0
8	0	0	1
9	1	1	0
10	2	0	0
11	1	1	0
12	2	0	0
13	1	0	1
14	2	0	0
15	0	2	0
16	2	0	0
17	2	1	0
18	2	1	0
19	3	0	0
20	2	1	0
21	3	0	0
22	1	2	0
23	2	1	0
24	2	0	2
25	4	0	0
26	4	0	0
27	4	0	0
28	4	0	0
29	4	0	0
30	4	0	0
31	5	0	0
32	5	0	0
33	1	1	3
34	5	0	0
35	1	0	4
36	5	0	0
37	6	0	0
38	5	1	0
39	4	2	0
40	3	0	3
41	2	1	3
42	2	1	3
43	4	0	2
44	4	0	2
45	5	0	1
46	6	0	0
47	5	1	0
48	6	0	0
49	6	0	0
50	7	0	0
51	3	1	3
52	7	0	0
53	7	0	0
54	7	0	0
55	8	0	0
56	3	6	0
57	1	1	8
58	8	2	0
59	8	0	2
60	11	0	0
61	2	0	9
62	11	0	0
63	10	2	0
64	10	2	0
65	4	0	8
66	12	0	1
67	10	3	0
68	6	0	7
69	13	0	1
70	7	0	7
71	14	2	0
72	13	0	3
73	7	0	9
74	10	1	6
75	16	2	0
76	7	0	12
77	11	0	10
78	3	1	20
79	15	0	9
80	26	0	0

Mean = 7.5

Multi-Cycle
Individual Application (n=31)	Type of PMC
Individual Application (n=31)	Agreed-Upon	Required	CMC
1	0	1	0
2	1	0	0
3	0	1	0
4	0	1	0
5	0	1	0
6	0	1	0
7	1	0	0
8	0	1	0
9	0	1	0
10	1	0	0
11	2	0	0
12	0	2	0
13	0	2	0
14	2	0	0
15	3	0	0
16	2	1	0
17	3	0	0
18	3	1	0
19	2	1	1
20	4	0	0
21	4	1	1
22	6	0	0
23	0	0	7
24	1	0	6
25	0	0	8
26	6	0	2
27	5	0	3
28	9	0	0
29	9	0	0
30	12	2	0
31	16	0	0

Mean = 4.4

Notes: Required PMCs include Accelerated Approval, PREA, and Animal Efficacy Rule studies; Includes only those products that have PMCs

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 35: Distribution of PMCs for Single- and Multi-Cycle Approvals

Exhibit 36: Focus Area of Postmarketing Study Commitments for FY2002-FY2007 Cohort

Commitments by Area of Focus (111 products:733 PMCs)

	Number of Postmarketing Commitments
Efficacy	114 (16%)
Safety	155 (21%)
Clin/Pharm	108 (15%)
Non-clinical	41 (6%)
CMC	183 (24%)
Required	59 (8%)
Other	73 (10%)

Type of Postmarketing Commitment (111 products; 733 PMCs)

	Number of Postmarketing Commitments
New Study	372 (51%)
Continue/Extend Study	84 (11%)
Submit Data/Report	61 (8%)
CMC	183 (25%)
Other	33 (5%)

Notes: Studies were classified by their primary focus area from the PMC description; Required PMCs include PREA and Accelerated Approval studies; “Other” PMC includes assay development, surveillance plans, literature reports, and RiskMAPs; New studies are those that were not initiated prior to approval; Continue/extend study refers to those that were ongoing but not complete; Submit data/report is refers to completed studies for which a final report or data had not been submitted to FDA prior to approval.

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 36: Focus Area of Postmarketing Study Commitments for FY2002-FY2007 Cohort

Exhibit 37: Single-Cycle Approval Rate by Sponsor Experience

Did the sponsor have prior FDA approvals?

	Yes (n=130)	No (n=55)
Single-cycle	55%	38%
Multi-cycle	45%	62%

Of those with prior FDA approvals, were any in the same therapeutic area as the current application?

	Yes (n=101)	No (n=29)
Single-cycle	58%	45%
Multi-cycle	42%	55%

Source: Booz Allen analysis, Public Sources, FY02-07 Cohort

Exhibit 37: Single-Cycle Approval Rate by Sponsor Experience

Exhibit 38: Sponsor Experience and Impact of Pre-Submission Meetings

	EOP2 Meeting		Pre-NDA/BLA Meeting
	Yes	No	Yes	No
Prior FDA Approval	51% single-cycle (n=48); 49% multi-cycle (n=46) Total: 100% (n=94)	67% single-cycle (n=24); 33% multi-cycle (n=12) Total: 100% (n=36)	54% single-cycle (n=61); 46% multi-cycle (n=51) Total: 100% (n=112)	61% single-cycle (n=11): 39% multi-cycle (n=7) Total: 100% (n=18)
No Prior FDA Approval	33% single-cycle (n=10); 67% multi-cycle (n=20) Total: 100% (n=30)	44% single-cycle (n=11); 56% multi-cycle (n=14) Total: 100% (n=25)	36% single-cycle (n=16); 64% multi-cycle (n=28) Total: 100% (n=44)	45% single-cycle (n=5); 55% multi-cycle (n=6) Total: 100% (n=11)

Source: Booz Allen Analysis, Action Packages, FDA systems; FY02-07 Cohort

Exhibit 38: Sponsor Experience and Impact of Pre-Submission Meetings

Exhibit 39: Cumulative Average First Cycle Communications During Review Period by Sponsor Experience (Written and Meetings)

Review Cycle Quarter	Prior FDA Approval	No Prior Approvals
First	5.9	6.8
Second	17.2	15.9
Third	33.2	28.5
Fourth	53.2	43.8

Summary Statistics of Communications Data

	Prior FDA Approval	No Prior Approvals
Number of Products	65	20
Range	5-158	16-115
Median	44	39
Mean	53.2	43.8

Notes: Written communication includes letters, faxes, and emails. Meetings include face-to-face meetings and telecoms.

Source: Booz Allen Analysis, FDA Systems; FY 05-07 Cohort

Exhibit 39: Cumulative Average First Cycle Communications During Review Period by Sponsor Experience (Written and Meetings)

Exhibit 40: Approval Rate vs. Sponsor Type and Origin

First-cycle Approval Rates by Type and Size of Sponsor:

	Biotech	Pharma
Small	56% (n=25)	21% (n=19)
Medium	44% (n=9)	38% (n=26)
Large	79% (n=19)	53% (n=83)

First-cycle Approval Rate by Sponsor Location:
US-based: 54% (n=112)
Foreign-based: 45% (n=73)

The overall average first-cycle approval rate was 50%.

Notes: Sponsor Size was based on market capitalization: Large: > $5B; Medium: < $5B and > $1B and Small: < $1B; Does not include institutions and non-profit organizations.

Source: Booz Allen Analysis; FY02-07 Cohort

Exhibit 40: Approval Rate vs. Sponsor Type and Origin

Exhibit 41: First Cycle Approval Rate by Review Designation and Company Size

	Priority Designation	Standard Designation
Large Companies	78% single-cycle (n=39); 22% multi-cycle (n=11) Total: 100% (n=50)	38% single-cycle (n=20); 62% multi-cycle (n=32) Total: 100% (n=52)
Small Companies	48% single-cycle (n=10); 52% multi-cycle (n=11) Total: 100% (n=21)	35% single-cycle (n=8); 65% multi-cycle (n=15) Total: 100% (n=23)

Note: Does not include institutions and non-profit organizations

Source: Booz Allen Analysis, Action Packages, FDA Systems, Public Sources; FY02-07 Cohort

Exhibit 41: First Cycle Approval Rate by Review Designation and Company Size

Exhibit 43: Cumulative Average First-Cycle Communications During Review Period by Company Size (Written and Meetings)

Review Cycle Quarter	Small Companies	Medium Companies	Large Companies
First	6.2	5.9	6.1
Second	14.2	14.3	18.6
Third	25.9	24.2	36.8
Fourth	40.2	48.2	56.1

Summary Statistics of Communications Data

	Small Companies	Medium Companies	Large Companies
Number of Products	18	14	52
Range	5-115	26-118	18-158
Median	36	39.5	49.5
Mean	40.2	48.5	56.1

Notes: Written communication includes letters, faxes, and emails. Meetings include face-to-face meetings and telecoms.

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY 05-07 Cohort

Exhibit 43: Cumulative Average First-Cycle Communications During Review Period by Company Size (Written and Meetings)

Exhibit 44: Cohort Applications by Office and Review Designation

Applications by Review Designation and Office:

	Review Office
	ODE I	ODE II	ODE III	OAP	OODP	OBRR	OVRR	OCTGT
Priority	5	8	5	26	26	6	6	1
Standard	22	20	14	14	12	13	7	0
Total	27	28	19	40	38	19	13	1

First-Cycle Approval Rate by Review Office

	Review Office
	ODE I	ODE II	ODE III	OAP	OODP	OBRR	OVRR	OCTGT
Single-Cycle	5	10	10	19	27	11	11	0
Multi-cycle	22	18	9	21	11	8	2	1
Total	27	28	19	40	38	19	13	1
First-Cycle Approval Rate	19%	36%	53%	48%	71%	58%	85%	0%

First-Cycle Approval Rate for Standard Applications by Review Office

	Review Office
	ODE I	ODE II	ODE III	OAP	OODP	OBRR	OVRR	OCTGT
Single-Cycle	3	6	6	5	4	7	6	0
Multi-cycle	19	14	8	9	8	6	1	0
Total	22	20	14	14	12	13	7	0
First-Cycle Approval Rate	14%	30%	43%	36%	33%	54%	86%	NA

First-Cycle Approval Rate for Priority Applications by Review Office

	Review Office
	ODE I	ODE II	ODE III	OAP	OODP	OBRR	OVRR	OCTGT
Single-Cycle	2	4	4	14	23	4	5	0
Multi-cycle	3	4	1	12	3	2	1	1
Total	5	8	5	26	26	6	6	1
First-Cycle Approval Rate	40%	50%	80%	54%	88%	67%	83%	0%

Notes: ODE: Office of Drug Evaluation; OAP: Office of Antimicrobial Products; OODP: Office of Oncology Drug Products; OBRR: Office of Blood Research and Review; OVRR: Office of Vaccines Research and Review; OCTGT: Office of Cellular, Tissue and Gene Therapies.

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 44: Cohort Applications by Office and Review Designation

Exhibit 45: Submission Timing and First Cycle Approval Rates

Number of Submissions by Quarter

	First Quarter	Second Quarter	Third Quarter	Fourth Quarter
Priority	16	16	20	31
Standard	11	24	16	51
Total	27	40	36	82

First-Cycle Approval Rate by Quarter Submitted

	First Quarter	Second Quarter	Third Quarter	Fourth Quarter
Priority Applications Approved	11	13	15	17
Standard Applications Approved	4	8	11	14
Total	15	21	26	31
First-Cycle Approval Rate (Overall cohort first-cycle approval rate = 51%)	56%	53%	73%	38%

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 45: Submission Timing and First Cycle Approval Rates

Exhibit 46: Staffing Changes Between Pre-Submission and Review

Applications with Staff Changes from Pre-Submission to Review

Change from Pre-Submission to Review?	Office Leadership	Division Leadership	Medical Team Lead	Regulatory Project Manager
Yes	17	17	22	30
No	68	68	51	47
Data not available	0	0	12	8

First-Cycle Approvals Associated with Staffing Changes

	Office Leadership		Division Leadership		Medical Team Lead		Regulatory Project Manager
	Change	No Change	Change	No Change	Change	No Change	Change	No Change
Single-Cycle	9	33	9	33	8	28	17	20
Multi-Cycle	8	35	8	35	14	23	13	27
Total	17	68	17	68	22	51	30	47

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY05-07 Cohort

Exhibit 46: Staffing Changes Between Pre-Submission and Review

Exhibit 47: Impact of Foreign or Domestic Manufacturing Site

Manufacturing Site Inspections

	Foreign Inspections	Domestic-only Inspections
Single-Cycle	53	38
Multi-Cycle	56	34
Total	109	72
First-Cycle Approval Rate	49%	53%

Manufacturing Site Location and Review Designation

	Priority Designation	Standard Designation
Domestic-only Inspection	76% single-cycle (n=28); 24% multi-cycle (n=9) Total: 100% (n=37)	29% single-cycle (n=10); 71% multi-cycle (n=25) Total: 100% (n=35)
Foreign Inspection	60% single-cycle (n=27); 40% multi-cycle (n=18) Total: 100% (n=45)	41% single-cycle (n=26); 59% multi-cycle (n=38) Total: 100% (n=64)

Note: Four products (3 NDAs and 1 BLA) did not have manufacturing site inspections

Source: Booz Allen Analysis, Action Packages, FDA Systems; FY02-07 Cohort

Exhibit 47: Impact of Foreign or Domestic Manufacturing Site

Exhibit 48: Summary Overview of Recommendations

This exhibit has a time line that runs from the Pre-IND stage through post-approval, or Phase 4, activities. Recommendations 1 and 2 are opportunities for improving submission quality, which impact processes prior to NDA or BLA submission. Recommendations 3, 4, and 5 are opportunities for review process improvement, and impact processes after NDA or BLA submission. Recommendations 6 through 12 impact processes across the entire lifecycle.

Exhibit 48: Summary Overview of Recommendations

Exhibit 49: Baseline of GRMPs Practices Adopted by Divisions – January 2006 Snapshot

This exhibit illustrates the implementation characteristics of all 42 GRMPs practices (specified in Exhibit 12) for ten review divisions. The review divisions are identified by letters A through J, rather than the actual division name. For each division, unless otherwise noted all activities were implemented prior to GRMPs.

Division A: 88% implemented prior to GRMPs, 90% implemented post-GRMPs.
Not implemented: Activities 9 and 38
New activity specified by GRMPs and implemented: Activity 19
Not done for all applications; not all sub-activities performed: Activities 22 and 36

Division B: 89% implemented prior to GRMPs, 93% implemented post-GRMPs.
Not implemented: Activities 9 and 24
New activity specified by GRMPs and implemented: Activity 19
Data not available: Activities 8 and 30-42

Division C: 79% implemented prior to GRMPs, 88% implemented post-GRMPs.
Not implemented: Activities 8, 22, 24 and 34
New activity specified by GRMPs and implemented: Activities 7, 16, 19 and 21
Not done for all applications; not all sub-activities performed: Activity 30

Division D: 84% implemented prior to GRMPs, 92% implemented post-GRMPs.
Not implemented: Activity 9
New activity specified by GRMPs and implemented: Activities 21 and 22
Not done for all applications; not all sub-activities performed: Activity 34
Data not available: Activities 8, 24-33, 35-39 and 41

Division E: 81% implemented prior to GRMPs, 86% implemented post-GRMPs.
Not implemented: Activities 9 and 30
New activity specified by GRMPs and implemented: Activities 19 and 21
Not done for all applications; not all sub-activities performed: Activities 4, 8, 27 and 34

Division F: 67% implemented prior to GRMPs, 71% implemented post-GRMPs.
Not implemented: Activities 7-9, 12, 14, 16, 22, 34, 36 and 37
New activity specified by GRMPs and implemented: Activities 19 and 21
Not done for all applications; not all sub-activities performed: Activities 3 and 4

Division G: 86% implemented prior to GRMPs, 86% implemented post-GRMPs.
Not implemented: Activities 4, 6, 9, 14, 22 and 30

Division H: 74% implemented prior to GRMPs, 76% implemented post-GRMPs.
Not implemented: Activities 7-9, 14, 22, 27, 36 and 37
New activity specified by GRMPs and implemented: Activity 19
Not done for all applications; not all sub-activities performed: Activities 4 and 28

Division I: 71% implemented prior to GRMPs, 71% implemented post-GRMPs.
Not implemented: Activities 9, 16, 27, 30 and 34-41

Division J: 76% implemented prior to GRMPs, 76% implemented post-GRMPs.
Not implemented: Activities 9, 22, 30, 35-39 and 41
Not done for all applications; not all sub-activities performed: Activity 34

Notes: Only a sample of the divisions (10) had in-depth interviews on the GRMPs practices; The GRMPs activities of the associated numbers were referred in Exhibit 12; Percent computed was based on available data

Source: Booz Allen analysis; FDA interviews

Exhibit 49: Baseline of GRMPs Practices Adopted by Divisions – January 2006 Snapshot

Exhibit 50: GRMPs Compliance by Product

GRMPs Compliance	Number of Products	Percent of Products
80% or more (highly compliant)	17	20%
60%	20	24%
40%	34	40%
20% or less	14	16%

Source: Booz Allen Analysis, Action Packages, FDA systems, FY05-07 Cohort

Exhibit 50: GRMPs Compliance by Product

Exhibit 51: CDER and CBER GRMPs Compliance by Office

GRMPs Compliance	Review Office
GRMPs Compliance	OBRR	ODE I	OVRR	OODP	ODE III	ODE II	OAP	OCTGT
80% or more (highly compliant)	6	4	2	3	1	1	0	0
60%	2	3	2	8	6	2	3	0
40%	1	4	3	6	0	7	6	1
20% or less	0	3	1	2	1	1	6	0
Total	9	14	8	19	8	11	15	1

Source: Booz Allen Analysis, Action Packages, FDA systems, FY05-07 Cohort

Exhibit 51: CDER and CBER GRMPs Compliance by Office

Exhibit 52: GRMPs Compliance in Review Phase

	Held mid-cycle meeting	Completed primary clinical review
2005 (n=38)	8%	16%
2006 (n=31)	29%	16%
2007 (n=16)	63%	38%

Source: Booz Allen Analysis, Action Packages, FDA systems; FY05-07 Cohort

Exhibit 52: GRMPs Compliance in Review Phase

Exhibit 53: GRMPs Compliance in Filing and Action Phases

	Filing Phase		Action Phase
	Held filing meeting	Sent 74-day letter	Discussed labeling
2005	58% (n=38)	92% (n=38)	45% (n=33)
2006	61% (n=31)	97% (n=31)	45% (n=29)
2007	63% (n=16)	94% (n=16)	46% (n=13)

Source: Booz Allen Analysis, Action Packages, FDA systems; FY05-07 Cohort

Exhibit 53: GRMPs Compliance in Filing and Action Phases

Exhibit 54: Review Communication Practices Preferred by Industry and FDA Focus Group Participants

	Preferred	Not Preferred	Potential Benefits of Preferred Method
Planning	Defined Sponsor/FDA communication strategy at review start	Communication preferences not defined at review start	Managed expectations Improved FDA Division consistency
Interface	Managed communication through the RPM with direct reviewer access, as appropriate	Ad hoc communications with different members of the review team	Clear definition of when direct reviewer access is acceptable and appropriate
Frequency/Status	Ongoing and frequent communication throughout Proactive and early issue identification Review status updates provided routinely	Communication limited to key milestones or issues	Defined status updates without adversely impacting FDA’s review workload, and provide sponsors helpful updates

Source: Booz Allen Analysis, Sponsor and FDA Focus Groups

Exhibit 54: Review Communication Practices Preferred by Industry and FDA Focus Group Participants

Exhibit 55: Industry and FDA Focus Group Feedback on GRMPs Filing and Planning and Review Activities

Activity	Industry	FDA
Filing	Internal business priorities and goals affect application quality and timing (e.g., push for Q4)	Filing incomplete and/or poor quality submissions affected the timeliness of FDA’s review later in the process
Application Orientation	Application Orientation presentation viewed as a valuable opportunity to provide an application overview, review submission format and data location, and understand FDA concerns	Value of Application Orientation presentations varied and not all Divisions held these meetings or approved meeting requests
74-day Letter	Inconsistent receipt of the 74-day letter Mixed views on the value –useful only if major review issues are included	Perception that the 74-day letter was helpful to sponsors
Issues Identification/Response	Response time was based on the severity of issues and access to FDA review staff to clarify issues	Timing of issue identification varied by specific application, due to complexity of review
Mid-cycle Meeting	Mid-cycle meeting was not always conducted on time Interim schedule increases review transparency and allows sponsors to discuss review status	Conducting the Mid-Cycle meeting on-time for priority reviews was challenging, and Division-level flexibility in content/format was preferred

Source: Booz Allen Analysis, Sponsor and FDA Focus Groups

Exhibit 55: Industry and FDA Focus Group Feedback on GRMPs Filing and Planning and Review Activities

Exhibit 56: Industry and FDA Focus Group Feedback on GRMPs Advisory Committee, Action and Post-Action Phases

Activity	Industry	FDA
Plan AC Meeting	Extremely time consuming and resource intensive (especially for small companies) FDA did not always follow the AC guidelines	Significant challenge for review team especially if priority review Appointing committee members is difficult
Conduct AC Meeting	AC meeting was favored when FDA/Sponsor disagreed on review issues	Annual AC meeting schedule is fixed within CBER AC meeting can inhibit ability to meet 1st-cycle performance
Wrap-up Meeting/Pre-Approval Safety Conference	OSE should be involved earlier in the process	OSE should be involved earlier and existing guidances and MaPP documentations should be reviewed for consistency against GRMPs timelines
Labeling Discussions	Labeling discussions should start sooner, or sections could be approved in advance; but negotiations could be prolonged if discussions commenced too early in process	Labeling discussions conducted before the AC meeting or primary review are not effective Three weeks for discussion is too short
Sign-off & Action		Challenges in drafting action letter – late review input, many versions circulating
Lessons Learned	Lessons learned meeting with FDA was helpful even for approved products because it provided insights into the FDA’s perspective	Typically these meetings are conducted only with sponsors whose applications are not approved

Source: Booz Allen Analysis, Sponsor and FDA Focus Groups

Exhibit 56: Industry and FDA Focus Group Feedback on GRMPs Advisory Committee, Action and Post-Action Phases

horizontal rule