National Conference on Interstate Milk Shipments (NCIMS)

Evaluation of the NCIMS HACCP Pilot Program Phase II Expansion

Appendix I.
Industry, State Auditor, State Rating Officer, State Program Administrator and FDA Regional Milk Specialist Questionnaires- Responses, and Summaries

(Table of Contents)

• Executive Summary
• Attachment 1. Summary of HACCP pilot questionnaires for all participants
• Attachment 2. Industry questionnaire
• Attachment 3. State regulatory Program Director's questionnaire
• Attachment 4. State regulatory Milk Plant Regulatory Auditor's questionnaire
• Attachment 5. State regulatory State Listing Officer's questionnaire
• Attachment 6. Regional Milk Specialist questionnaire

---

Executive Summary

HACCP Pilot Phase I and II Participant Survey

Recommendations supported by the responses to these questionnaires were reviewed by the NCIMS HACCP Evaluation team and presented to the NCIMS HACCP Committee. All three groups of participants, industry, state regulators, and regional milk specialists stated that under the HACCP pilot, product safety was not adversely affected because they agreed there was equivalency between the traditional PMO system requirements and industry's implementation of HACCP.

The maximum score possible on the scored portion of the questionnaire was 5.0. Industry and the RMSs responded with an average score of (4.4) and the state regulators with an average score of (4.1) when asked if HACCP was equivalent to traditional controls. Industry comments reflected the positive impact of HACCP noted by improvements in plant sanitation (4.0) and corrective action programs (4.0), and reductions in withheld product (see attached Excel spreadsheets).

All participants agreed that monitoring of other NCIMS requirements they were accountable for was equivalent under the pilot, industry (4.6), state (4.3), and RMS (4.4). When asked if there was equivalency between the state HACCP listing and the traditional state rating or the FDA audit of the HACCP listing versus the traditional check rating the participants again agreed there was equivalency as noted in the summary table.

Industry responses were 4.6 and 4.1 respectively, state responses were 4.5 and 4.2, and the RMS rated the equivalency of the state HACCP listing to the traditional state rating with a score of 3.6 but rated the FDA audit of the HACCP listing to traditional check rating with a higher score of 4.2. These statements are made based on the fact that the responses were either agreed, strongly agreed or in a few instances no opinion was checked but there were no responses in either the disagree or strongly disagree category regarding the equivalency of the HACCP and traditional control programs.

Summary of responses to common questions asked of the industry, state, and RMS pilot plant participants in Phase I and II.

Question	Industry*	State*	RMS*
Equivalency of HACCP to traditional food safety plant controls	4.4	4.1	4.4
Plant monitoring of other NCIMS requirements under the pilot was equivalent	4.6	4.0	4.4
State monitoring of other NCIMS requirements under the pilot was equivalent	4.1	4.3	4.4
Equivalency of state HACCP listing/traditional state rating	4.6	4.5	3.6
Equivalency of FDA audit of HACCP listing to traditional check rating	4.1	4.2	4.2
*These statements are made based on the fact that the responses were either agreed to, strongly agreed to, or in a few instances no opinion was checked but there were no responses in either the disagree or strongly disagree category regarding the equivalency of the HACCP and traditional control programs. The response to each question was recorded using the following scale: (1-1.4) Strongly disagree, (1.5-2.4) Disagree, (2.5-3.4) No opinion, (3.5-4.4) Agree, (4.5-5) Strongly agree and in some cases () NA. The NA response has limited use.

Industry

The responses by the industry participants are supportive of the pilot. As stated in the executive summary benefits included decreases in plant sanitation deficiencies and improved corrective action programs. Several plants also noted their relationship with regulatory groups improved as a result of their participation in the pilot. Several of the industry participants felt implementation of the pilot had significant impact on their individual job responsibilities and sites required additional resources for training under the pilot.

One major concern with the response by the industry pilot participants, though, is their understanding of the significance of hazards. Many responded to the question "were additional hazards identified" by listing dead end piping, piping changes, O rings, rusted seals, and washer coolers. While the improvements are important changes to a manufacturer's sanitation and maintenance programs, they are not hazards from the HACCP context of biological, physical, and chemical hazards. The HACCP Pilot Team will need to address this misunderstanding either through training materials or possibly posting a question to the technical resource team. One site did note the manufacture of eggnog for inclusion in its hazard analysis.

Examples of industry comments include:

• "Operators more cautious and thorough"
• "Because of HACCP, a number of opportunities have opened up for us including air blows, improved shelf-life for non-Grade "A" products, etc"
• "Unify government then implement HACCP w/industry input and justify what is required by scientific data for both the regulatory and industry"
• "Involve more upper management in initial training to help fully understand what is involved"
• "State has more confidence we can't hide anything"

We also asked the participants what costs they incurred implementing HACCP. Costs reported ranged from $150,000 to $2,000 but on average without the high and low responses it was indicated that HACCP implementation could be accomplished with an investment of $20,000 dollars for the average plant. Implementation hours again had the same extreme range but training averaged 100 hours for most plants. Significant time commitments were made to corrective action and verification activities.

State Regulators

Scores on the questionnaires support the executive summary finding that respondents believe public health under HACCP was not compromised. For example the following comments reflect the lower scored or no opinion responses to questions on equivalency:

• "I believe that HACCP gives the perception that a plant is better than the Grade A traditional system".
• "I believe the best way to make HACCP the very best Grade A system - is to make the Prerequisite Program mandatory in the traditional Grade A system. Allow regulatory to monitor the PP's, make mandatory employee training and give the inspector the obligation to quiz employees on their job knowledge. This would marry the best of HACCP with the PMO (greatest public health document in existence for dairy)".
• "A plant under HACCP could evaluate that no public health hazard existed using all Grade A traditional laboratory analysis and accept ungraded (R.O.) milk for Grade A products. This is a possibility under the HACCP guidelines". Editors Note: The traditional system and the HACCP pilots required all milk to originate from a Grade A listed source.
• "My personal experience with the HACCP Audit is that a score should be given on the audit form. This would allow for a plant that has a lot of verbiage on an audit to know its place value. I believe this would keep a plant from ignoring PP's and only worrying about CCPs, by giving PP a point value it would give more significance to them as a deficiency. I do not believe that a plant who corrects problems as the audit is ongoing should not be subject to a failing score and or delisting. If we allow plants to make corrections as the audit is ongoing we have done nothing to protect public health the day after we leave the plant and then for the next four months (some States 6 months)".

Positive statements regarding HACCP were also made:

• "Total understanding of the plant process and flow, open communication between plant and regulatory. HACCP puts the responsibility on the plant and regulatory as an overseer".
• "Due to the extensive hazard analysis I was able to better understand the complex processing of milk and milk products".
• "It focuses on each step of a product process. It better documents the steps of the process".
• "HACCPs strongest edge over the traditional Grade A system is that HACCP requires that the rules (PP's) be written down. This allows the plant to remove interpretation and to train their employees consistently every time. If the HACCP plan is written properly with all pertinent manufacturing operations included this can be used as a training manual for all new and present employees".
• "The program needs some good press. It needs to be understood that HACCP is a total commitment for both regulatory and industry. HACCP take time. More time is spent on record review then actual plant inspection".

Regional Milk Specialists

The average scores for equivalency and adequate public health protection (under the pilot) by the RMSs were actually higher than those submitted by industry. This position is reflected in the executive summary. The industry participants actually scored some of the same questions slightly lower but there is no significant difference between the two groups of pilot participants. The MSS also responded positively with a rating of 4.2 for the following question, "Under the NCIMS HACCP Pilot, the Regulatory Agency is able to verify and assure the safety of Grade A milk products from a milk plant at least as well as they could under the traditional system"?

However, there were wide differences of opinion between the RMSs that submitted written responses to the question, "What is the greatest benefit of the HACCP pilot"? This ranged from, "I have as yet not seen any benefit" to "The greatest benefit is that you can witness what the plant is doing during the times that there is no outside overview of how the plant operates. We can now understand how the plant is actually operating". In that regard the RMS that had not seen any benefit from HACCP submitted the lowest scores in reference to the pilot.

Background: Summary of HACCP Pilot Questionnaire Evaluations

The attached charts and summaries detail the responses of industry participants (all twelve plants participating in Phase I or II responded), state regulators (all states participating in the pilot had regulatory representatives respond to the questionnaire but the number of State Directors, listing officers or rating officers varied), and regional milk specialists (all five involved in the pilot) working with the pilot plants. The three groups were asked common questions as well as questions unique to their role as a manufacturer, state regulator, or RMS. Examples of each questionnaire are attached for review. The summary of the surveys was been blinded by removing any reference to the manufacturing site or brand names, FDA region, or state that had regulatory oversight for the specific pilot plant. The response to each question was recorded using the following scale:

(1) Strongly disagree
(2) Disagree
(3) No opinion
(4) Agree
(5) Strongly agree
and in some cases () NA. The NA response has limited use.

The responses to each question were averaged. A high or low average score must be interpreted in context of the question. Whenever the three groups were asked a common question those results were also compared to determine the degree of agreement among the HACCP pilot participants. Comments by the three groups that responded to the questionnaire are addressed separately.

Attachments

1. Summary of HACCP pilot questionnaires for all participants
2. Industry questionnaire
3. State regulatory Program Director's questionnaire
4. State regulatory Milk Plant Regulatory Auditor's questionnaire
5. State regulatory State Listing Officer's questionnaire
6. Regional Milk Specialist questionnaire

---

Attachment 1. Summary of HACCP Pilot Questionnaires for all participants

Milk Plant Responses to Scored Questions Participated in both the Phase I & II Pilots*

Question	Plant 1	Plant 2	Plant 3	Plant 4	Plant 5	Average
1. Equivalency of HACCP to traditional plant controls	4	4	4	5	4	4.2
2. Was product safety monitored more completely under HACCP	4	4	4	4	4	4.0
3. Regulatory verification is equivalent under HACCP	4	5		5	4	4.5
4. State HACCP listing was equivalent to traditional listing	4	4	4	5	5	4.4
5. Product shelf life increased	2	3	3	4	3	3.0
6. Consumer complaints decreased	4	3	3	3	3	3.2
7. Withheld product decreased	4	3	4	3	3	3.4
8. Sanitation monitoring results improved	4	4	5	5	4	4.4
9. FDA audit of HACCP listing was equivalent to traditional check rating	5	4	4	5	4	4.4
10. Plant monitoring of other NCIMS requirements are equivalent	4	4	5	5	4	4.4
11. State and FDA monitoring of other NCIMS requirements is equivalent	4	4		5	4	4.3
12. Corrective action programs improve under the HACCP pilot	4	4	4	5	5	4.4
13. HACCP pilot had no impact on my job responsibilities	2	1	1	1	4	1.8
14. HACCP listing reduced requests for third party audits		5	4	3	4	4.0
15. Relationship with state regulator improved under the pilot	2	4	3	5	3	3.4
16. Relationship with FDA regulator improved under the pilot	4	4	5	4	4	4.2
17. Volunteering for the HACCP pilot was the right thing to do	4	3	5		4	4.0
18. Use of a HACCP consultant was beneficial		3		4	2	3.0
19. NCIMS Technical Assistance Team was valuable	4	4	5	4	4	4.2
20. Volunteering for the pilot was a sound decision	5	4	5		4	4.5
*Note: All plant designations have been randomized between tables so Plant #1 in this table is not the same as Plant #1 in other tables.

Milk Plant Responses to Scored Questions Participated in Phase II Pilot only*

Question	Plant 1	Plant 2	Plant 3	Plant 4	Plant 5	Plant 6	Plant 7	Average
1. Equivalency of HACCP to traditional plant controls	5	4	4	4	5	5	5	4.6
2. Was product safety monitored more completely under HACCP	5	2	4	4	5	5	5	4.3
3. Regulatory verification is equivalent under HACCP	5	4	4	4	5	5	5	4.6
4. State HACCP listing was equivalent to traditional listing	5	4	4	4	5	5	5	4.6
5. Product shelf life increased	2	2	3	2	4	4	3	2.9
6. Consumer complaints decreased	2	3	3	2	4	3	3	2.9
7. Withheld product decreased	2	2	3	4	4	3	3	3.0
8. Sanitation monitoring results improved	4	2	3	4	5	4	4	3.7
9. FDA audit of HACCP listing was equivalent to traditional check rating		4	4			3		3.7
10. Plant monitoring of other NCIMS requirements are equivalent	5	5	5	4	5	5	5	4.9
11. State and FDA monitoring of other NCIMS requirements is equivalent	5	2	5	4	5	5	3	4.1
12. Corrective action programs improve under the HACCP pilot	2	2	4	4	5	5	4	3.7
13. HACCP pilot had no impact on my job responsibilities	1	4	2	1	2	2	1	1.9
14. HACCP listing reduced requests for third party audits	3	3	5	4	5	3	4	3.9
15. Relationship with state regulator improved under the pilot	1	3	3	1	4	2	3	2.4
16. Relationship with FDA regulator improved under the pilot	2	2	5	4	4	4	4	3.6
17. Volunteering for the HACCP pilot was the right thing to do	2	4	3	4	4	3	3	3.3
18. Use of a HACCP consultant was beneficial		4	5			3		4.0
19. NCIMS Technical Assistance Team was valuable		4		5	5	3		4.3
20. Volunteering for the pilot was a sound decision	4	5	5	4	5	5	4	4.6
*Note: All plant designations have been randomized between tables so Plant #1 in this table is not the same as Plant #1 in other tables.

Plant Personnel that participated in both Phase I and Phase II Pilots

How would you improve the program?

• I'll think more about this before your visit. I personally think the pilot has gone very well. We have been fortunate to have support from regulatory officials who are willing to be ahead of the curve and try something new. Although there have been challenges along the way, in hindsight I think our plant has succeeded with HACCP and any challenges were worth it when we see the end result which has been increased employee buy in and product that is at least as safe as it was under the PMO.
• Unify government then implement HACCP w/industry input and justify what is required by scientific data for both the regulatory and the industry
• Plant HACCP team believes that the NCIMS HACCP pilot program has been conducted very well
• Because of HACCP, a number of opportunities have opened up for us including air blows, improved shelf life for non-Grade "A" products, etc.
• I would like to comment on question 17. I do not know if we would have volunteered for the Pilot Program if we knew our plant would be under going so many changes. The positive thing about XXXX participating in the Pilot would be that we have develop a strong HACCP Program and our associates are better trained and understand how their job affects food safety and we accomplished this while our Plant was going through personnel and construction changes. I believe this says a lot for our Plant and associates plus sets the standard that HACCP can be done and done well through adverse conditions. Looking back to the beginning of the Pilot to now, I am very pleased with our decision to participate in the Pilot and would encourage others to choose HACCP over the traditional system.
• I would like to address the hours in E. The hours are stated to reflect the amount of time invested per incident. The HACCP team and I perform verification on a quarterly basis. The entire team does not devote 20 hours - most of that time to me but they spend about 8 hours. The yearly validation is a more extensive review of the HACCP document with the team. Also, the hours for training are not accurate. HACCP is part of our associates' job description so they are trained initially on the concept of HACCP then it's continuous. Our new associates do not know what it was like before HACCP or how their job description has changed. Much of the time was for associate training.
• The only suggestion I would make concerning how to improve the program would be to have clearer guidelines and better communication. I know a lot of issues have been dealt with in Phase II but I think more structure needs to be given as far the program.
• Make guidelines clearer and more specific in key areas so less confusion and more direction in beginning. Give people ideas and suggestions and acceptable control measures for consistency across the industry. Examples of forms and documentation would also be helpful.
• Because of HACCP, plant experienced increased flexibility that resulted in operational efficiencies that contributed to plant operation, product safety and product quality.
• Provide more specific and detailed information on verification and validation.
• No comment
• More direct help from NCIMS or IDFA at each plant or better examples.
• Regulatory agencies from different states should work together and share experience and knowledge.
• One five-day HACCP course is not enough. Why not do two five-day workshops approx. 4 - 6 month apart. The second workshop could be done in small groups by plant and the plant team should present their HACCP work to the experts.
• Un-experienced regulators should not do listing audits; NCIMS should get one expert involved.
• The Baltimore training was good but as an initial training for someone with no background in HACCP I think it went a bit too fast. It would have been helpful to have reviewed the training material before the session. Also I feel that in order to assist plant to get up and running. An onsite visit by someone with HACCP training would be most helpful. We had Steve Pierson visit in May. He gave us important direction and guidance that got us much more quickly along than if we plodded along on our own.
• Involve more upper management in initial training to help fully understand what is involved

Milk Plant Responses Estimated Resources, Phase I & II Plants*

HACCP training	Plant 1	Plant 2	Plant 3	Plant 4	Plant #	Average
HACCP team	84 hrs.	600	2 hrs./mo.	8	100	198.0
Production Personnel	6	400	1 hrs./mo.	2		136.0
45 day audit prep	20	500	32			184.0
4 month audit prep	28	500	32	30		147.5
Verification activities	520	5,200	5 hrs./wk.	20		1913.3
State Program Evaluation by FDA		200				200.0
Other (audits since initial 4
month audit)	58					58.0
Other (Correcting deficiencies identified				1,800		1800.0
HACCP Document Preparation	60	600	1 day/wk.			330.0
Baseline Survey	8	300	1 day/wk.	20		109.3
Advisory Visit's) by State & FDA	12	300	16	20		87.0
State Listing Audit's)	32	400	32	30		123.5
FDA Check Audit (Prep. & Audit	28	300	32	50		102.5
Validation Activities	12	200	1 day/wk			106.0
HACCP Written Program
Review & Updating	30	2,000	120	40		547.5
Did the Hazard analysis change since the initial listing?	Yes	Yes	No	Yes	Yes
Costs for HACCP Implementation	$5,200	$150,000	$10,000	$24,000	$10,000	$39,840
*Note: All plant designations have been randomized between tables so Plant #1 in this table is not the same as Plant #1 in other tables.

Milk Plants Estimated Resources, Phase II *

HACCP training	Plant 1	Plant 2	Plant 3	Plant 4	Plant 5	Plant 6	Plant 7	Average
HACCP team	3 hours	100	120	100	120	400	65	129.7
Production Personnel	3	80	3	10	80	70	40	40.9
45 day audit prep			6	12	36	15		17.3
4 month audit prep				12		5		8.5
Verification activities	10	50	120	48	180	200	2	87.1
State Program Evaluation
by FDA				48				48.0
Other (audits since initial
4 month audit)
Other (Correcting deficiencies identified	10	20	50	24		10		22.8
HACCP Document Preparation	100	100	150	200	180		300	171.7
Baseline Survey	16	8		3	12	20	16	12.5
Advisory Visit's) by State & FDA	20	20		50	36		36	32.4
State Listing Audit's)	16	14	12	144	30	20		39.3
FDA Check Audit (Prep. &
Audit				0				0.0
Validation Activities		20		16		4		13.3
HACCP Written Program Review & Updating		50		8	72	6	100	47.2
Did the Hazard analysis change since the initial listing?	No	No	No	No	No	Yes	NA
Costs for HACCP Implementation (Dollars)	Unknown	$12,500	$2,000	$50,000			$20,000	$21,125
*Note: All plant designations have been randomized between tables so Plant #1 in this table is not the same as Plant #1 in other tables.

 

FDA Milk Specialists Response to Scored Questions - Phase I & II Pilots*

Question	RMS 1	RMS 2	RMS 3	RMS 4	RMS 5	Average
1. Equivalency of HACCP Listing/FDA check rating	4	3	4	5	5	4.2
2. Equivalency of plant control/traditional system	4	4	4	5	5	4.4
3. Ability to verify safety under HACCP/traditional	4	3	4	5	5	4.2
4. Equivalency of state HACCP listing/traditional state rating	4	3	4	5	2	3.6
5. State monitoring of other NCIMS requirements under pilot	4	4	4	5	5	4.4
6. Plant monitoring of other NCIMS requirements under pilot	4	4	4	5	5	4.4
7. Equivalency of state HACCP listing/traditional listing	4	4	4	5	5	4.4
8. Adequacy of Baltimore HACCP training	4	4	4	5	4	4.2
*Note: All RMS have been randomized between tables so RMS #1 in this table is not the same as RMS #1 in other tables.

FDA Milk Specialists Estimated Hours, Phase I & II Pilots*

	RMS 1	RMS 2	RMS 3	RMS 4	RMS 5	Average
Baseline survey	16	16				16
Advisory visit(s) to milk plant	24	8			64	32
4 month audits				108		108
FDA Audits	24			33		28.5
State listing audit		12		48		30
State program evaluation		40				40
*Note: All RMS have been randomized between tables so RMS #1 in this table is not the same as RMS #1 in other tables.

FDA Milk Specialists that participated in both Phase I and Phase II Pilots

Greatest benefit

• The greatest benefit that I have seen is the reaffirmation of food safety by the management of the pilot plant.
• Benefit-I have as yet not seen any benefit.
• Working together with the plant personnel and state regulatory personnel from the very beginning. We all learned together and it appeared to work successfully the pilot site and was thus, a positive experience.
• The greatest benefit is that you can witness what the plant is doing during the times that there is no outside overview of how the plant operates. We can now understand how the plant is actually operating.
• HACCP is a more relevant state enforcement model. The HACCP process is more consistent with the traditional inspectional and training activities of SRO and RMS.

Greatest concern

• My greatest concern is that the HACCP pilot program will continue to be expanded & extended at the 2003 NCIMS. Please let me explain.
• My concern actually stems from what I see as a trend in the NCIMS and was actually brought to light while attending a meeting on Automated Milking Systems. First off, I believe that we (FDA) dropped the ball in allowing milk produced in this pilot program (AMS pilot) to be marketed as Grade A. The speaker stated that there might be a proposal to expand & extend the AMS pilot. I see this as an attempt to set precedents. Extending & expanding this pilot will continue to allow milk of questionable quality to be marketed as Grade A, thereby setting a precedent to argue against any objection by FDA. (The milk was Grade A for the pilot, therefore it is Grade A now)
• Please don't misunderstand me. I have a tremendous appreciation of the vast amount of work that has been done on the HACCP pilot and I am not accusing the HACCP pilot of any wrongdoing. Allowing any pilot to expand and or extend conference-to-conference is similar to having a toe then foot then leg in the door. I hope that this is something that we consider in the future prior to agreeing to pilot programs. A firm deadline should be set by the NCIMS for the submission of data and pilot evaluations.
• Personnel changes at the facility, or a change in plant motivation could put the program at risk.
• The concern is that the plant is not keeping accurate records of the problems and corrections that they are making. For the most part they are making the corrections properly and timely but there is no record to show what was done to make the correction and how to prevent it from happening again. (This has been extensively better in the last six months.) The other concern is that it takes the plant too long to get into the dairy HACCP mode after they have started up. I think most of the plants are already doing the correction items, but have never followed it through to look at what they are doing.
• No comment
• No comment

Suggestion for improving the HACCP pilot

• The HACCP committee has spent a great deal of time trying to formulate a rating/check rating system for HACCP. Farms that are regulated by the Performance Based Inspection System are rated & check rated under the traditional system. I believe that HACCP plants could be regulated by States under HACCP but rated & check rated under the traditional system. I believe that the states would accept a proposal such as this
• I believe that HACCP is a very beneficial tool that industry should use to enhance product protection.
• Training is the single most important ingredient for FDA, State and Industry alike in order for HACCP to be a viable alternative to the traditional system. Many states at this juncture have had minimal exposure to HACCP. They will be getting some training through juice HACCP. I have had some recent conversations with some states in our region that want some exposure. To conducting a HACCP audit. During the 510 courses in XXXXX last week some preliminary plans were made to have some members of the XXXXX Dairy Division receive some Training in XXXX.
• Making a basic model of the way for the plants to track deviations, showing corrections, and prevention of the event happening. That seems to be one of the difficult parts of the HACCP system for plants to grip. The other is that the prerequisites need to be emphasized a lot more as the plant is starting up.
• As a regulatory model - nothing. As a rating and listing model an enforcement rating is needed. States are responsible for permitting, sampling, interpretation, inspection, pasteurization testing (salt), enforcement, and records. These items are not captured in the HACCP listing.

State Regulatory Auditors Responses to Scored Questions, Phase I & II Pilots*

Question	State 1	State 2	State 3	State 4	State 5	Average
1. Equivalency of listing audits/traditional inspection		4	4	5	5	4.5
2. Equivalency of plant control/traditional system	5	4	4	4	5	4.4
3. Equivalency of state HACCP listing/traditional state rating	5	3	4	5	5	4.4
4. Equivalency of HACCP Listing/FDA check rating	5	3	4	3	5	4.0
5. State monitoring of other NCIMS requirements under pilot	5	4	4	4	5	4.4
6. Plant monitoring of other NCIMS requirements under pilot	4	3	4	4	5	4.0
7. FDA/listing officer monitoring of other NCIMS requirements under pilot	5	4	4	3	5	4.2
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Regulatory Auditors Responses to Scored Questions, Phase II Pilot*

Question	State 1	State 2	Average
1. Equivalency of listing audits/traditional inspection	4	3	3.5
2. Equivalency of plant control/traditional system	4	1	2.5
3. Equivalency of state HACCP listing/traditional state rating	4	1	2.5
4. Equivalency of HACCP Listing/FDA check rating
5. State monitoring of other NCIMS requirements under pilot	4	1	2.5
6. Plant monitoring of other NCIMS requirements under pilot	4	1	2.5
7. FDA/listing officer monitoring of other NCIMS requirements under pilot	4	1	2.5
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Regulatory Auditors Time Required, Phase I & II Pilots*

Question	State 1	State 2	State 3	State 4	Average
Baseline survey			8		8.00
Advisory visits to plant			12		12.00
State listing audits	24		8	24	18.67
4 Month Audits	24	43.5	38	8	28.38
"Follow-up" Audits	2		4		3.00
FDA Audit	24		16		20.00
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Regulatory Auditors Time Required, Phase II Pilots*

Question	State 1	State 2	State 3	State 4	Average
Baseline survey	64		12	8	28.00
Advisory visits to plant	9		170	12	63.67
State listing audits			54	8	31.00
4 Month Audits			205	38	121.50
"Follow-up" Audits			24	4	14.00
FDA Audit		4	30	16	25.00
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Regulatory Auditors that participated in both Phase I and Phase II Pilots

Greatest benefits

• I see the greatest benefit as more responsibility for the plant-it get them more involved in the safety of the products they produce
• My greatest benefit and I believe HACCP's best asset is that the RULES (PP's) are all written down this allows regulatory and all plant persons to know and understand the "maintenance" of the manufacturing plant. For example: the life expectancy of air filters, instead of waiting for the filters to clog up we now change them on a timely interval. CIP and COP chemical strengths are now listed and monitored, this keeps us from reacting to dirty equipment because the operator did not know or understand their Chemical strengths.
• I had to take a closer look at how the plant operates and functions which leads to better audits.
• Educational component ensures company-wide dialogue/practices

Greatest concern

• My greatest concern is the addition of yet another HACCP style, we've already got USDA HACCP, FDA and National Marine Fisheries Seafood HACCP, juice HACCP, health department HACCP, corporate audit HACCP
• I believe that HACCP gives the perception that a plant is better than the Grade A traditional system.
• My greatest fear is that we will begin to monitor records more than we physically monitor the plant and equipment. We have already found issues with the plant having trouble with record keeping and that the record keeping is requiring an additional person to keep up with these records. I believe that a small plant might remove a person from the manufacturing operation and make this person responsible for record keeping.
• Integrity and cooperation of plant employees is a must.
• Concerned about manpower issues when key personnel are called away for HACCP
• Suggestions for improving HACCP as voluntary alternative to the traditional system
• Provide a hazard guide and regularly scheduled HACCP training for regulatory and industry folks, since they are frequently using HACCP already as a corporate tool and for juice HACCPI expect they're just waiting for the outcome of the pilot project to participate
• My personal experience with the HACCP Audit is that a score should be given on the audit form. This would allow for a plant that has a lot of verbiage on an audit to know its place value. I believe this would keep a plant from ignoring PP's and only worrying about CCPs, by giving PP a point value it would give more significance to them as a deficiency. I do not believe that a plant who corrects problems as the audit is ongoing should not be subject to a failing score and or delisting. If we allow plants to make corrections as the audit is ongoing we have done nothing to protect public health the day after we leave the plant and then for the next four months (some States 6 months).
• I believe the best way to make HACCP the very best Grade A system - is to make the Prerequisite Program mandatory in the traditional Grade A system. Allow regulatory to monitor the PP's, make mandatory employee training and give the inspector the obligation to quiz employees on their job knowledge. This would marry the best of HACCP with the PMO (greatest public health document in existence for dairy).
• For HACCP to be successful it must be a bottom up approach. Plants must be able to write their HACCP plan with as little input as possible from regulatory. There must be some items which regulatory must require but the plants need some leniency as to how to accomplish these items.
• I believe the program is going well as is, at least until I have a chance to work with more plants.

State Regulatory Auditors that participated in both the Phase II Pilots

Greatest benefits

• It focuses on each step of a product process. It better documents the steps of the process.

Greatest concern

• This program may become mandatory.
• Having to run a dual program
• Initial record keeping. Plants may tend to "over demand" records of themselves. (i.e. in the prerequisite programs, etc.)

Suggestions for improving HACCP as voluntary alternative to the traditional system

• Simplify
• The plant I inspect is interested in doing some or all of its own HTST testing. More guidance/information should be provided (i.e. standardized training? Annual training for plant personnel? A new license? Etc?

State Milk Program Director Responses to Scored Question - Phase I & II Pilot*

Question	State 1	State 2	Average
1. Equivalency of listing audits/traditional inspection	5	4	4.5
2. Equivalency of plant control/traditional system	5	3	4.0
3. Equivalency of state HACCP listing/traditional state rating	5	4	4.5
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Milk Program Director Responses to Scored Questions - Phase II Pilot*

Question	State 1	State 2	State 3	Average
1. Equivalency of state, FDA audits under pilot/traditional inspections/ratings	5	4	4	4.3
2. Equivalency of state NCIMS HACCP listing/traditional state rating	4	4	4	4.0
3. Equivalency of state monitoring of other NCIMS requirements under pilot	4	5	4	4.3
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Milk Program Directors Time Estimate in Hours - Phase I & II Pilot*

Time Required in hours	State 1	Average
Baseline survey
Advisory visits to plant
State listing audits
4-month audits	24	24
"Follow-up" audits
FDA Audit
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Milk Program Directors Time Estimate in Hours - Phase II Pilot*

Time Required in hours	State 1	State 2	Average
Baseline survey	64		64.0
Advisory visits to plant	9		9.0
State listing audits
4 Month Audits
"Follow-up" Audits
FDA Audit		4	4.0
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Program Director Comments Phase I & II Pilots

Suggestions for improving HACCP as voluntary alternative to the traditional system

• Implement a scoring system for audits. Regulators are comfortable with a scoring system and it will improve the credibility of and usefulness of the audit.
• Use 4-month audit intervals (rather than 6-month) to keep auditors from being overwhelmed

Most important suggestion to improve HACCP as a voluntary alternative to the traditional system

• Don't dismantle the traditional inspection system. Maintain it as a safety net. Allow state regulators the option of a traditional inspection whenever they feel that one is necessary or warranted.

State Rating (Listing) Officers Response to Scored Questions - Phase I & II Pilots*

Question	State 1	State 2	State 3	State 4	State 5	Average
1. Equivalency of listing audits/traditional inspection	5	4	5	4	5	4.6
2. Equivalency of plant control/traditional system	5	4	5	4	5	4.6
3. Equivalency of state HACCP listing/traditional state rating	5	2	5	4	5	4.2
4. Equivalency of HACCP Listing/FDA check rating	4	3	5	4	5	4.2
5. State monitoring of other NCIMS requirements under pilot	5	4	4	4	5	4.4
6. Plant monitoring of other NCIMS requirements under pilot	5	4	4	4	5	4.4
7. FDA/listing officer monitoring of other NCIMS requirements under pilot	4	3	4		5	4.0
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Rating (Listing) Officers Response to Scored Questions - Phase II Pilots*

Question	State 1	State 2	Average
1. Equivalency of listing audits/traditional inspection	5	5	5
2. Equivalency of plant control/traditional system	5	4	4.5
3. Equivalency of state HACCP listing/traditional state rating	5	5	5
4. Equivalency of HACCP Listing/FDA check rating	5	5	5
5. State monitoring of other NCIMS requirements under pilot	5	5	5
6. Plant monitoring of other NCIMS requirements under pilot	5	5	5
7. FDA/listing officer monitoring of other NCIMS requirements under pilot	5	5	5
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Rating (Listing) Officers
Time Required - Phase I and Phase II Pilots*

Time required for Phase II ( hours)	State 1	State 2	State 3	State 4	State 5	Average
Baseline survey
Advisory visits to plant
State listing audits	24					24
FDA Audit	24			24		24
Time required for Phase I (hours)
Baseline survey	16					16
Advisory visits to plant	30					30
State listing audits	32					32
FDA Audit	24		93			58.5
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Rating (Listing) Officers Time Required - Phase II Pilot*

Time required for Phase II Hours	State 1	State 2	Average
Baseline survey	30	20	25
Advisory visits to plant	88	30	59
State listing audits	48	20	34
FDA Audit
*Note: All states have been randomized between tables so State #1 in this table is not the same as State #1 in other tables.

State Rating (Listing) Officers that participated in both Phase I and Phase II Pilots

Greatest benefit

• Benefits-total understanding of the plant process and flow, open communication between plant and regulatory. HACCP puts the responsibility on the plant and regulatory as an overseer.
• I suppose the greatest benefit derived, from a regulatory standpoint, is the opportunity to actually participate in the formation and implementation of an authentic HACCP plan in a processing plant.
• Greatest benefit is the plants (employees) increased understanding of process and procedures
• Due to the extensive hazard analysis I was able to better understand the complex processing of products made at this facility.

Greatest concern

• Time to complete audits
• My primary concern centers around the important decisions the HACCP Committee will make in the next few months. Dairy HACCP will only be as successful as the regulatory criteria established to monitor its success or failure in each facility.
• Greatest concern is amount of time involved
• I was, and remain concerned this program will as cumbersome and "command and control" as the USDA FSIS counterpart.

Suggestions for improving HACCP as voluntary alternative to the traditional system

• The program needs some good press. It needs to be understood that HACCP is a total commitment for both regulatory and industry. HACCP take time. More time is spent on record review then actual plant inspection.
• Mandate the regulatory structure currently used with the traditional system and a state audit frequency of once each four months. Our experience in the pilot program has demonstrated that the traditional regulator structure blends very well with HACCP and the four month frequency allow our agency to stay in touch with changes in operation and address problems in a more timely manner.
• In addition, I would suggest a scoring system be developed that would encourage a more uniform application in auditing from state to state. A scoring system would also place a value on prerequisite programs that does not currently exist.
• Regulatory side-audit form is hard to work with and more training in doing HACCP audits
• Submit the program for approval and implementation n the 2003 NCIMS conference. Set a deadline for XXX to start or withdraw their interest in phase II of the pilot.
• Benefit-the auditor has the latitude to return and observe all aspects of a plant including training, etc.
• By integrating a HACCP system, I see plant management and employees thoroughly involve in collecting data that serves to mentor the safety of their products. I see employees receiving training s in HTST pasteurization so that they recognize a problem that they probably would not have notice under the traditional system. Through their HACCP involvement, I see commitment form the plant management to all issues that we consider of vital importance as regulators.

Greatest concern

• My greatest concern is that the initial momentum that it has taken to implement HACCP in the milk plant must be sustained. In other words, this HACCP approach must remain a way of life for the milk plant management and employees and be prioritized above all other responsibilities.
• Suggestions for improving HACCP as voluntary alternative to the traditional system
• It must be stated that some plants are ready for HACCP, some never will be. Those that are ready and can run better and more safely under a HACCP plan should be allowed to operate under HACCP.
• I believe that we need continual training and evaluation: we need to implement a certification program for those of us who are performing HACCP listing audits; a completed Hazards Guide reference would represent an immediate improvement in the current HACCP alternative.
• I believe that we need continual training and evaluation: we need to implement a certification program for those of us who are performing HACCP listing audits; a completed Hazards Guide reference would represent an immediate improvement in the current HACCP alternative.

---

Attachment 2. Industry Questionnaire

NCIMS HACCP EVALUATION TEAM

PILOT PLANT QUESTIONNAIRE

The NCIMS HACCP Evaluation Team will be visiting each dairy plant involved in the NCIMS HACCP Pilot Program between mid-May 2002 to the end of October 2002. For the Evaluation team to maximize its efficiency, we request the following survey be filled out and returned to Allen Sayler, International Dairy Foods Association at asayler@idfa.org or faxed to 202-331-7820. All survey information will be compiled into one report with confidentiality maintained so no one individual or plant can be identified. Thank you for your cooperation.

A. Plant Name _________________Person Completing Survey & Title ______________________
Phone Number _________________ FAX ________________ Email _________________________
Plant Manager _______________________ Plant HACCP Team Leader ______________________

B. Products Covered by the HACCP System (check all that apply):

☐ Fluid Milk (white or flavored)	☐ Cottage cheese	☐ Dried products
☐ Cultured (yogurt, sour cream, buttermilk)	☐ Aseptic Products	☐ Ice Cream Products
☐ Other (e.g. Butter, Ice Cream, Condensed Products, etc. - please list)

C. Attach list of HACCP Team Members Name, Title, & HACCP experience.

D. Listed below are statements and beneath each statement are five possible responses. Based on your experiences during the NCIMS HACCP Pilot, please mark the response that most closely reflects your opinion regarding each statement. During the onsite visits by the NCIMS Evaluation Team, you will have an opportunity to provide your reasons for, and comments about your responses.

1). Under the NCIMS HACCP Pilot, we are able to monitor and assure the safety of Grade A dairy products we produce at least as well as (or better than) we were able to do under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

2). We now monitor the safety of Grade A dairy products we produce more completely under the NCIMS HACCP Pilot than we did under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

3). Under the NCIMS HACCP Pilot, the Regulatory agency is able to verify and assure the safety of Grade A milk products from our milk plant at least as well as they could under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

4). The current State NCIMS HACCP listing provides at least as much verification and assurance of the ongoing safety of Grade A milk products from my plant as did State Ratings made under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

5). Under the NCIMS HACCP pilot, Grade "A" products increased in product shelf life?
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

If shelf life increased, what was the per cent increase? ____________________________________

6). Under the NCIMS HACCP pilot, we had fewer consumer complaints.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

7). Under the NCIMS HACCP pilot, we had a decrease in withheld or nonconforming product.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

8). Under the NCIMS HACCP pilot, we observed constantly improved sanitation monitoring results.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

9). The latest official FDA audit of the HACCP listing for my plant provides at least as much verification and assurance of the ongoing safety of Grade A milk products from my plant as FDA check ratings have done under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree ( ) NA

10). "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are monitored by our plant at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

11). "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are evaluated by the State and the FDA at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

12). Under the NCIMS HACCP pilot, the ability to implement corrective actions improved resulting from better training and a deeper understanding of product safety issues based on root cause analysis and internal verification programs.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

13). The HACCP pilot has had no impact on my job responsibilities.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

14). The NCIMS HACCP program has had a positive impact on non-Grade "A" products produced at this plant.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree ( ) NA

15). Buyers reduced their HACCP requirements or auditing frequency since we became listed under the NCIMS HACCP pilot program.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

16). The relationship with my state regulators improved over the course of the HACCP pilot.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

17). The relationship with my FDA Regional Milk Specialist improved over the course of the HACCP pilot.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

18). The use of a paid consultant was helpful in development and implementation of the NCIMS HACCP program.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree ( ) NA

19). The NCIMS Technical Assistance Team was a valuable resource.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree ( ) NA

20). I think volunteering for the HACCP pilot was the right decision for my plant.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

E. Summarize the approximate time investment by plant personnel in the HACCP pilot to date:

Training (include trainer & trainee):		HACCP Document Preparation:	_______ hours
HACCP Team	______ hours	Baseline Survey	_______ hours
Production Personnel	______ hours	Advisory Visit(s) by State & FDA	______ hours
45 Day Audit( Prep. & Audit)	______ hours	State Listing Audit(s)	______ hours
4 Month Audit (Prep. & Audit)	______ hours	FDA Check Audit (Prep. & Audit)	_______hours
Verification Activities	_______hours	Validation Activities	_______hours
State Program Evaluation by FDA	_______hours	HACCP Written Program Review & Updating	______hours
Other (Identify)HACCP Operation	______ hours
Other (Identify)Correcting deficiencies identified during the various steps of the program			_____ hours

F.

1. Did the Hazard Analysis change since you were listed? Yes _____ No ____
2. What was the approximate extra cost beyond normal operational expenses under the traditional PMO program for participating in the HACCP pilot? $______________________
3. List any hazards evaluated during the hazard analysis that may not have been addressed under your traditional PMO safety program. _________________________________________________________________________________
4. How would you improve the HACCP pilot program (on back)?

---

Attachment 3. State regulatory Program Director's Questionnaire

STATE PROGRAM DIRECTOR'S QUESTIONNAIRE

In order for the NCIMS HACCP Pilot Committee Evaluation Team to fully evaluate the pilot we ask that you answer the following questions. We intend to minimize the on-site evaluations so we do not unduly interfere with the operations or inappropriately influence the study. Because we will depend heavily on reports, documents and questionnaires, we would appreciate your prompt response. Please complete this questionnaire using MS WORD then save it with a different name and e-mail it as an attachment to rgraham@dhhmail.dhh.state.la.us.

Name_____________________________________________ State_______________________________

Part I

Listed below are a series of statements with which you might agree or disagree. Beneath each statement are five possible responses to each of these statements. Based on your experiences during the NCIMS HACCP Pilot, please mark the response that most closely reflects your opinion regarding each statement.

Note: During the onsite visits by the evaluation team, you will have an opportunity to provide your reasons for, and comments about the responses you provide.

1). State Regulatory audits, state listing audits and FDA audits performed on Pilot Plant(s) in your state provide at least as much verification and assurance of Grade A products as did the inspections, ratings and check ratings under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

2). If the NCIMS HACCP pilot is accepted and implemented as an alternative to the current system, the State Listing Officer should be prohibited from having direct regulatory responsibility for a HACCP listed milk plant in which they will perform the listing audit.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

3). The current State NCIMS HACCP plant listing(s) provides at least as much verification and assurance of the ongoing safety of Grade A milk products (for each plant that has been listed that I audit) as State Ratings made under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

4). We are able to evaluate the "other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

Part II

Please complete the following:

1. Recognizing the additional advisory, training and monitoring time involved in the start-up of any new program, the amount of time required to regulate plants under HACCP in the future will be ( ) more ( ) the same ( ) less than that required under the traditional system.
2. Estimate the amount of time (in hours) spent, thus far, on the following activities involved with the Phase II portion of the Pilot Study (Subsequent to January 28, 2002) (If not applicable indicate "NA")
   1. Baseline Survey _____
   2. Advisory Visits to plant _____
   3. State Listing Audits _____
   4. 4 Month Audits _____
   5. "Follow-up" Audits _____
   6. FDA Audit _____
3. What were your one greatest benefit and your one greatest concern derived from this voluntary alternative system?
4. What would you suggest to improve HACCP as a voluntary alternative to the traditional system?

---

Attachment 4. State Regulatory Milk Plant Regulatory Auditor's Questionnaire

State Regulatory Auditor's Questionnaire
(State Person Responsible for Routine Regulatory Oversight)

In order for the NCIMS HACCP Pilot Committee Evaluation Team to fully evaluate the pilot we ask that you answer the following questions. We intend to minimize the on-site evaluations so we do not unduly interfere with the operations or inappropriately influence the study. Because we will depend heavily on reports, documents and questionnaires, we would appreciate your prompt response. Please complete this questionnaire using MS WORD then save it with a different name and e-mail it as an attachment to rgraham@dhhmail.dhh.state.la.us.

Name_____________________________________________ State_______________________________

Part I

Listed below are a series of statements with which you might agree or disagree. Beneath each statement are five possible responses to each of these statements. Based on your experiences during the NCIMS HACCP Pilot, please mark the response that most closely reflects your opinion regarding each statement.

Note: During the onsite visits by the evaluation team, you will have an opportunity to provide your reasons for, and comments about the responses you provide.

1). The regulatory audits of the Pilot Plant(s) I performed provided at least as much verification and assurance of the safety of Grade A milk products as did the traditional inspections I have performed on similar milk plants. (Answer this question ONLY if you have performed one or more audits of the Pilot Plant.)
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

2). If the NCIMS HACCP pilot is accepted and implemented as an alternative to the current system, the State Listing Officer should be prohibited from having direct regulatory responsibility for a HACCP listed milk plant in which they will perform the listing audit.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

3). Under the NCIMS HACCP Pilot, the milk plant is able to verify and assure the safety of Grade A milk products that they produce at least as well as they could under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

4). The current State NCIMS HACCP plant listing(s) provides at least as much verification and assurance of the ongoing safety of Grade A milk products (for each plant that has been listed that I audit) as State Ratings made under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

5). The latest FDA audit of the State NCIMS HACCP listing for plants I audit have provided at least as much verification and assurance of the ongoing safety of Grade A milk products from those plant(s) as FDA check ratings of State ratings have done under the traditional system (complete this question ONLY if one or more FDA audits have been completed).
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

6). We are able to evaluate the "other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

7). "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are monitored by the milk plant at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

8). "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are evaluated by FDA Milk Specialists and State Listing Officers at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

Part II

Please complete the following:

1. 1. Recognizing the additional advisory, training and monitoring time involved in the start-up of any new program, the amount of time required to regulate plants under HACCP in the future will be
   ( ) more ( ) the same ( ) less
   than that required under the traditional system.
2. 2. Estimate the amount of time (in hours) spent, thus far, on the following activities involved with the Phase II portion of the Pilot Study (Subsequent to January 28, 2002) (If not applicable indicate "NA")
   1. Baseline Survey _____
   2. Advisory Visits to plant _____
   3. State Listing Audits _____
   4. 4 Month Audits _____
   5. "Follow-up" Audits _____
   6. FDA Audit _____
3. What were your one greatest benefit and your one greatest concern derived from this voluntary alternative system?
4. What would you suggest to improve HACCP as a voluntary alternative to the traditional system?

---

Attachment 5. State Listing Officer's Questionnaire

STATE LISTING OFFICER'S QUESTIONNAIRE
(State Person Responsible for NCIMS Ratings and HACCP Listings)

In order for the NCIMS HACCP Pilot Committee Evaluation Team to fully evaluate the pilot we ask that you answer the following questions. We intend to minimize the on-site evaluations so we do not unduly interfere with the operations or inappropriately influence the study. Because we will depend heavily on reports, documents and questionnaires, we would appreciate your prompt response . Please complete this questionnaire using MS WORD then save it with a different name and e-mail it as an attachment to rgrahm@dhhmail.dhh.state.la.us.

Name_________________________________________ State________________________________________

Part I

Listed below are a series of statements with which you might agree or disagree. Beneath each question are five possible responses to each of these statements. Based on your experiences during the NCIMS HACCP Pilot, please mark the response that most closely reflects your opinion regarding each statement.

Note: During the onsite visits by the evaluation team, you will have an opportunity to provide your reasons for, and comments about the responses you provide.

1). The Listing audit of the Pilot Plant(s) I performed provided at least as much verification and assurance of the safety of Grade A milk products as did the traditional Sanitation Ratings I have performed on similar milk plants. (Answer this question ONLY if you have performed one or more audits of the Pilot Plant.)
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

2). If the NCIMS HACCP pilot is accepted and implemented as an alternative to the current system, the State Listing Officer should be prohibited from having direct regulatory responsibility for a HACCP listed milk plant in which they will perform the listing audit.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

3). Under the NCIMS HACCP Pilot, the milk plant can verify and assure the safety of Grade A milk products that they produce at least as well as they could under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

4). Under the NCIMS HACCP Pilot, the Regulatory agency is able to verify and assure the safety of Grade A milk products from a milk plant at least as well as they could under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

5). The latest FDA audit of the State NCIMS HACCP Pilot listing(s) for plant(s) I have audited provides at least as much verification and assurance of the ongoing safety of Grade A milk products from those plants as FDA check ratings of State ratings I have done under the traditional system (complete this question ONLY if one or more FDA audits have been completed).
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

6). "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling etc., are evaluated by the State regulatory agency as well under the NCIMS HACCP pilot alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

7). "Other" NCIMS requirements, such as Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are monitored by the milk plant at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

8). "Other" NCIMS requirements such as Appendix N. drug monitoring, use of raw milk from only Grade A listed sources, product labeling etc., are evaluated by FDA at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

Part II

Please complete the following:

1. Recognizing the additional advisory, training and monitoring time involved in the start-up of any new program, the amount of time required to regulate plants under HACCP in the future will be ( ) more ( ) the same ( ) less than that required under the traditional system.
2. Estimate the amount of time (in hours) spent, thus far, on the following activities involved with the Phase II portion of the Pilot Study (Subsequent to January 28, 2002) (If not applicable indicate "NA")
   1. Baseline Survey _____
   2. Advisory Visits to plant _____
   3. State Listing Audits _____
   4. FDA Audit _____
3. What were your one greatest benefit and your one greatest concern derived from this voluntary alternative system?
4. What would you suggest to improve HACCP as a voluntary alternative to the traditional system?

---

Attachment 6. Regional Milk Specialist Questionnaire

Regional Milk Specialist NCIMS HACCP Pilot Questionnaire

In order for the NCIMS HACCP Pilot Committee Evaluation Team to fully evaluate the pilot we ask that you answer the following questions. We intend to minimize the on-site evaluations so we do not unduly interfere with the operations or inappropriately influence the study. Because we will depend heavily on reports, documents and questionnaires, we would appreciate your prompt response. Please complete this questionnaire using MS WORD then save it with a different name and e-mail it as an attachment to rarbaugh@ora.fda.gov

PART I

Listed below are a series of statements that you may either agree or disagree with. Beneath each question are five possible responses to each of these statements. Based on your experiences during Phase II of the NCIMS HACCP Pilot, please mark the response that most closely reflects your opinion regarding each statement.

Note: During the onsite visits by the evaluation team, you will have an opportunity to provide your reasons for, and comments about the responses you provide.

1. FDA audit(s) of the State NCIMS HACCP Pilot listing for plant(s) that I have performed provide at least as much verification and assurance of the ongoing safety of Grade A milk products from the milk plant being audited as FDA check ratings I have performed of similar milk plants under the traditional system. (complete this question ONLY if you have performed one or more FDA audits of NCIMS HACCP listings during Phase I or II of the pilot.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

2. Under the NCIMS HACCP Pilot, the milk plant can verify and assure the safety of Grade A milk products that they produce at least as well as they could under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

3. Under the NCIMS HACCP Pilot, the Regulatory Agency is able to verify and assure the safety of Grade A milk products from a milk plant at least as well as they could under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

4. The current State NCIMS HACCP Pilot listing provides at least as much verification and assurance of the ongoing safety of Grade A milk products from the plant(s) listed as State Ratings made under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

5. "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are evaluated by the State regulatory agency at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

6. "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are monitored by the milk plant at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

7. "Other" NCIMS requirements, such as, Appendix N drug monitoring, use of raw milk from only Grade A listed sources, product labeling, etc., are evaluated by the State HACCP Listing Officer at least as well under the NCIMS HACCP alternative as under the traditional system.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

8. The training you received January 2002, in Baltimore, was adequate to prepare you for your responsibilities under the HACCP Pilot Study.
( ) Strongly disagree ( ) Disagree ( ) No opinion ( ) Agree ( ) Strongly agree

PART II

Please complete the following:

1. 1. Estimate the amount of time (in hours) spent on the following activities: (NA = Not applicable)
   1. Baseline Survey ____
   2. Advisory Visit(s) to milk plant ____
   3. State Listing Audit ____
   4. 4 Month Audits ____
   5. FDA Audit ____
   6. State Program Evaluation ____
2. What were your one greatest benefit and your one greatest concern derived from this voluntary alternative system?
3. What would you suggest to improve HACCP as a voluntary alternative to the traditional system?

---

Evaluation of the NCIMS HACCP Pilot Program Phase II Expansion