Guide To The Performance Audit Process
(Candidate & Auditor Guide, Retail Level I) For
State, Local & Tribal Food Safety Inspection Officers

Based on FDA’s Voluntary National Retail Food Program Standards
 Standard 2, Trained Regulatory Staff
http://www.cfsan.fda.gov/~dms/ret-toc.html

Posted: April 26, 2007

This guide provides the background leading up to the audit, purpose of the audit, candidates role, auditors role, food establishment selection criteria, implementation and scoring system, and question & answers.

Background Leading up to the Performance Audit -
FDA’s Voluntary National Retail Food Program Standards are updated through the Conference of Food Protection. In our joint effort to develop a nationally recognized training program, Standard 2, Trained Regulatory staff, http://www.cfsan.fda.gov/~dms/ret3toc.html provides such a training plan for those new hires or those individuals newly assigned to the regulatory retail food program. The training courses may be completed through the ORA U Retail Food Level I Curriculum on-line* http://www.fda.gov/ora/training/orau/state/RFCurric.htm or equivalent means (other sources of training meeting Standard 2).

During the training period, Standard 2 provides for the use of the “Assessment of Training Needs” (ATN) worksheet for the trainee’s application and transfer of web instruction to the office and field environment. The ATN is utilized during the 25 joint inspections when the trainee takes the lead and is observed and mentored by the jurisdiction’s training officer, supervisor or designee. This process re-enforces the competences expected to be performed during routine inspection activities and provides on-going feedback to the trainee and trainer.

After completing all of the training and joint and independent inspections in the curriculum, the trainee is now eligible as a candidate for the Retail Food Level I Performance Audit. The performance audit criteria consist of 10 elements of competency derived from the ATN worksheet. (The performance audit criteria contain those elements that in addition to being important are measurable and occur at a frequency that would be reasonably expected to be observed during an inspection. On the other hand, the ATN has elements that are important but some are hard to measure and/or occur infrequently.)

*Note: For state, local & tribal regulators to register on-line for free access to web courses, go to: http://www.fda.gov/ora/training/  and click on “State, Local and Tribal Regulators” and follow the instructions.

Note: This is a draft voluntary audit “tool” being offered to those jurisdictions that may choose to use it. Management should delegate responsibilities (i.e. establishment selection, audit scheduling, concurrence of the audit, etc.) to first line supervisors.  Please provide any comments or questions to ORAU@fda.hhs.gov  
     

Purpose of the Performance Audit –
The Level I Audit is designed to ensure that State, Local and Tribal Food Safety Inspection Officers (FSIO) have the knowledge and skills necessary to demonstrate their ability to apply the competencies required during an actual inspection. The audit mechanism provides each candidate an opportunity to demonstrate his or her expertise by successfully conducting a thorough and comprehensive audit. As part of ORA U’s voluntary Retail Food Level I Curriculum, the performance audit “tool” may be used by those jurisdictions choosing to use this field process to assess an individual’s newly acquired knowledge, skills and abilities.  

Candidate’s role during the audit –
The candidate (person being audited) is responsible for independently conducting an assigned inspection while being evaluated by the auditor.  The candidate should refrain from asking the auditor questions pertinent to the inspection (e.g. advice, assistance, etc.), but should feel free to explain his/her actions to the auditor before and during the audit. These explanations help the auditor understand the candidate’s approach to the inspection and reduces the risk of the auditor drawing inaccurate conclusions about the candidate’s actions.  If unique or unexpected circumstances are encountered during the audit, the candidate may seek appropriate guidance from his/her supervisor (or designee) while keeping the auditor informed of these contacts.

Performance Auditor’s role during the Audit –
The performance auditor assesses the candidate’s ability to conduct an inspection using the audit criteria and plays no role in conducting the inspection.  The candidate should conduct the inspection as if the auditor were not present.  The auditor needs to be as unobtrusive as possible.  The auditor may ask questions of the candidate to better understand or clarify the rationale for the candidate’s actions.

Performance Auditors should be (Supervisors) designated by management who have completed our ORA U DHRD web course “Basics of Auditing for Regulators” which is being developed to assist these auditors in their role http://www.fda.gov/ora/training/orau/state/intro.htm

  The Effective Performance Auditor Should:

• Be able to think logically without undue influence from personal bias. 
• Possess a level of technical competence regarding (State, Local, Tribal) policies and regulations that translate into respect from their peers. 
• Have a history of following and enforcing (State, Local, Tribal) policy and regulations versus utilizing their own criteria and interpretations.
• Be able to work with others in a cooperative, goal-oriented manner while recognizing the individual qualities of people.
• Be able to fail candidates who do not demonstrate the required competencies. 
• Have the ability to be impartial to any reputation that a candidate may have and judge the candidate’s competency on facts revealed during the performance audit.
• Possess good communication skills (includes the ability to communicate opinions, observations and conclusions in a courteous and respectful manner).

Food Establishment Selection Criteria for the Audit
Management should select the establishment to be used for the audit.  In all cases, the food establishment selected for the audit should reflect the work covered during the candidate’s training.  The establishment that you select should provide the opportunity to assess all audit criteria elements.

Audit Implementation and Scoring System

1. To pass the performance audit, the candidate must:
   The candidate must demonstrate competency on ALL 10 Audit Criteria Elements to pass the audit. An observed practice should lead to an element failure – and thus audit failure – if it would cause one of the following outcomes:
   1. The Department’s ability to take regulatory action in response to significant violative condition(s) is substantially compromised (i.e. the inability to demonstrate competency on the element results in compromising the health departments ability to process a case (“case killer”).
   2. Fundamental regulatory procedures are not successfully demonstrated. 
    
   If an element contains many judgments or actions that must be properly demonstrated by the candidate (as many elements do), and one or more of these judgments or actions are found to be deficient, the auditor must determine if the observed deficiencies constitute a pattern of behavior that demonstrates a widespread lack of understanding or lack of competency in the element.  Additionally, if any one of these judgments or actions is so fundamental to the core of an element that demonstrating competency on that judgment or action would be necessary to successfully show competency on that element, then an inability to demonstrate that competency would, by itself, warrant failure of the element.

2. Termination of the Performance Audit: 
   The performance auditor should, in a timely manner, notify the Health Department Director or candidate’s immediate supervisor of any audit termination.  Management should be responsible for determining how to complete the audit.  The Performance Auditor may end the audit for any of the following reasons:
   1. Sickness or personal emergency involving the candidate or performance auditor. 
   2. In the judgment of the auditor, the candidate’s performance clearly indicates that he or she is not ready for a performance audit.  This would result in an audit failure.
   3. The candidate’s performance jeopardizes possible regulatory action on the involved establishment.  This would result in an audit failure. 
      If during the audit (prior to completion of the audit), the auditor determines with certainty that the candidate has failed an element, the auditor may terminate the audit, based on the criticality of the failure. 
   4. The establishment is not suitable for the audit (i.e. audit is too complex, highly violative, etc.)

      NOTE: If the termination of the performance audit was due to candidate illness or personal emergency, the Health Department Director (or delegate, as determined by the Director) should reschedule the audit to a time when the personal emergency is no longer an impediment to conducting the audit.

      If the termination of the performance audit was based on auditor illness or personal emergency, the Director (or delegate) should work to reschedule the audit as soon as possible. 

3. Re-Audit
   When management chooses to provide a re-audit, the Health Department Director (or delegate as determined by the Director) will decide when the candidate will be re-audited, and should coordinate the scheduling of all re-audits.

   If the candidate fails the Certification audit, the Director may work with the candidate and the candidate’s first line supervisor to develop and implement a remediation plan to address deficiencies observed during the audit.  Suggestions on what the remediation plan may include are listed below. 

   • Attending a training course related to operational procedures,
   • Assignment of training inspections,
   • Reading assignments of current reference documents,
   • Other actions as necessary to improve performance in the areas noted in the failed or terminated audit. 

   At the discretion of the Director, the re-audit may be a “Directed re-audit”, i.e. an audit focusing on any previously failed element(s), or a “Full re-audit”, i.e. an audit addressing all audit criteria elements.

   Failing an element that had been passed during a previous audit:
   In cases where a Directed re-audit is performed and the candidate fails to demonstrate competency of an element he/she had demonstrated competency on during a previous audit, the auditor must decide whether to fail the candidate on that element – and thus the re-audit.  However, in determining failure of that element, the auditor should assess whether the failure was due to a lack of understanding of the element or an oversight on the part of the candidate.  A lack of understanding would indicate a lack of competency and thus justify a failure whereas an oversight may not.  The auditor may consult the candidate’s supervisor in determining pass or failure of the element.

   Note: If the candidate fails three audits, the Health Department Director decides the course of action. 

4. Additional Reporting Requirements for Audit Failures        
   If the candidate fails the audit, the performance auditor should provide a written rationale for those failed elements.  The written rationale shall be recorded on, or attached to the Level I FSIO Audit Results Summary Form.          

Steps to the Performance Audit

Overall responsibility for the implementation of this program within each state rests with the FSIO (State, Local, Tribal) department’s management.  Management may want to delegate responsibilities to first line supervisors (i.e. establishment selection, audit scheduling, and concurrence of successful completion of the performance assessment audit). 

Step 1 – The candidate (new inspector) works with his/her first line supervisor to complete all requirements listed in the FSIO (State, Local, Tribal) Level I Training Curriculum.

Step 2 – The candidate notifies his/her first-line supervisor when he/she completes step 1.

Step 3 – The supervisor confirms that the candidate has completed his/her Level I training curriculum.

Step 4 – The candidate’s supervisor notifies the Department Director that the candidate has completed the Level I Training Curriculum.

Step 5 – The Department Director (or designee) coordinates the scheduling of the candidate’s performance audit.  This coordination includes the selection of a performance auditor and appropriate establishment for the audit. 

Step 6 – The candidate performs the audit and the auditor audits the candidate.  The performance auditor may use the Level I Certification Audit Worksheet to take audit notes. 

Step 7 – At the conclusion of the audit, the performance auditor completes the FSIO (State, Local, Tribal) Audit Results Summary Form and Level I FSIO (State, Local, Tribal) Auditor Feedback Form and distributes the originals and copies of all audit forms as follows:

1. We suggest that a copy of the Level I FSIO Audit Results Summary Form be  provided to the Department’s Director and to the candidate’s supervisor.
2. We suggest the original Level I FSIO Audit Results Summary Form should be provided to the candidate.

           
Step 8 – The Department Director (or designee) reviews the Level I Audit Results Summary Form and verifies that all required training has been accomplished. 

If the Candidate Fails the Performance Audit, cited below are suggested options:

Remediation & Re-Audit

1. The Director reviews the results of the audit and determines remediation of failed elements.
2. After remediation is complete, the Director schedules a re-audit.  At the Director’s discretion, the re-audit may be a “Directed re-audit” i.e. audit focusing on any failed element or a “Full re-audit”, i.e. an audit addressing all audit criteria elements.
3. If the candidate fails a second audit, the Director may permit another re-audit (a third audit).  If the candidate fails three audits, the Director decides course of action.

Performance audit Questions and Answers
 

1. Should there be a review of the final inspection report by the auditor?
   Yes, we suggest the audit should include a review of the final inspection report by the auditor.  The auditor has witnessed the entire inspection so the review of the final inspection report closes the audit.
   
   
2. Where will the final feedback of the audit go?
   See the document titled “Steps to Level I FSIO Performance Assessment”. 
   
   
3. Will the candidate be aware of what the audit criteria is beforehand or the establishment they will be auditing?
   All the Level I FSIO (State, Local, Tribal) Audit documents and forms are posted on the DHRD ORAU web site and the candidate is encouraged to become familiar with all the related documents.
   
   
4. Can the same auditor do the re-audit, if a re-audit is necessary? 
   Yes, however, we suggest it is up to the discretion of the Health Department Director to make that decision.  Reference the Audit Implementation and Scoring System section above.  The candidate’s Health Department Director should have the authority to decide which auditor audits each employee, and he/she may permit an auditor to audit a candidate one or more times. 
   
   
5. How soon does the candidate get feedback from the audit or auditor? 
   Depending on the nature of the comments, the employee should get feedback as soon as possible.  If the auditor feels that his/her comments are minimal, the auditor can give feedback immediately.  If the auditor needs to review his/her comments, we recommend feedback should occur within 72 hours.  If comments require discussion with the Director, feedback may be postponed to 5 days.
   
   
6. If the Level I FSIO Training Curriculum requirements are not finished, should the inspector be audited?
   The audit should not be conducted until all training requirements are complete as referenced in Standard 2 of FDA’s Voluntary National Retail Food Program Standards.



7. How will the feedback from the audits be used?
   Information from the Audit Results Summary Form should be used by jurisdictions to track trend data as to the specific element(s) that are being failed. 
   
   
8. Can there be a better definition for discontinuance of an audit or examples of what constitutes discontinuance?  Does the employee get penalized? 
   See the Audit Implementation and Scoring System section above. Sections A & B explain the conditions for termination of the Audit.  Note: The candidate must demonstrate competency in all elements of the audit criteria.  Section C of the document explains the re-audit options.  As stated under section C “at the discretion of the Director, the re-audit may be a “Directed re-audit”, i.e. an audit focusing on any previously failed element(s), or a “full re-audit”, i.e. an audit addressing all audit criteria elements”. 
   
   
9. Is it really necessary to have an auditor different than the section supervisor? 
   We encourage Health Department Directors to designate a different auditor other than the candidate’s immediate supervisor; however, the final decision is up to the discretion of the Health Department Director.    

Audit Criteria Procedures, Guidance and Forms Involved with the Audit Process – The audit will be conducted according to the Level I FSIO Performance Assessment Audit Criteria and procedures documents developed by the Level I FSIO Advisory Group. All documents will be posted under ORA U State Training.  It is advantageous for the candidate (person being audited) and the auditor to take some time to review all the related documents posted on the web in order to feel more comfortable about the audit process and know what to expect.  Documents for review include:

1. Guide to the Performance Audit Process, Retail Level I FSIO – This guide provides the background leading up to the audit, purpose of the audit, candidate’s role, auditor’s role, food establishment selection criteria, implementation and scoring system, steps to the audit process, and question & answers.
2. Audit Criteria, Retail Level I FSIO - This lists the 10 elements the Auditor uses to assess the candidate during the audit process. 
3. Audit Failure Reference Guide, Retail Level I FSIO - This Reference Guide is designed to help clarify the competencies covered by each element of the Audit Criteria.  Please note; this is only a “Guide” and the listed examples are not all inclusive. 

Forms that you may find helpful:

4. Audit Worksheet Form, Retail Level I FSIO – [PDF 10 kb; Word , 37 kb] This document is available to the Performance Auditor to record notes or observations made during the audit.  Completed worksheets are not “official” documents and we suggest the worksheets not be provided to the candidate.  We suggest a copy of the Audit Results Summary Form be provided to the candidate.   
5. Audit Results Summary Form, Retail Level I FSIO – [PDF 58 kb; Word , 36 kb] This form is used by the Auditor upon completion of the audit to record pass/fail of the audit. 
6. Auditor Feedback Form, Retail Level I FSIO – [PDF 60 kb; Word , 36 kb] This form is completed by the Auditor and may be used to share information with his/her management concerning the audit process. 

Please provide any comments or questions to ORAU@fda.hhs.gov