1-8.

Overview of training development management/planning.

Through planning, a manager will develop a realistic estimate of the resources required to accomplish evaluation projects, establish milestones, and allocate the available resources to the project. Initial TD management planning begins with an education/TD requirement that is a result of a needs analysis, or a new/revised training strategy.

Types of
planning

   a. Planning serves as a top-level tool to manage education/training requirements and resources. Proponent schools develop two types of plans:

      (1) A Training Development Project Management Plan (TDPMP) is developed for every education/TD process or product (see table 1-1 for procedure). This plan is simple and informal, or detailed and complex. The TDPMP will tell the "who", "what", "when", "where", "why", and "how", and ultimately estimate the cost of every project.

            (a) The TDPMP documents workload and resource requirements for the duration of a specific project. Examples include-

• Conduct a job analysis of the 84B30.

• Design/develop a drill.

            (b) The TDPMP identifies manpower and resourcing requirements for the specific project to include such things as-

• Personnel.

• Milestones.

• Costs.

• Material.

• Temporary duty (TDY).

• Any other factor required for the TD project.

        (2) Proponent TD Plan is a proponent's internal, living document that includes all requirements (resourced and unresourced). If a new TD requirement is identified or an existing one is changed, adjust the TD Plan. The TD Plan is a rollup of the requirements outlined in the TDPMP. It is a long-range document covering multiple years, and provides data to various resource, budget, and manpower reports, such as the Monthly Status Report, Installation Contract, Program Objective Memorandum, and Command Operating Budget.

            (a) The TD Plan documents TD workload and resource requirements for the programming, planning, budgeting, and execution years. The details within the TD Plan will increase each year.

            (b) The TD Plan addresses all required collective and individual training products (e.g., resident and nonresident courses, training support packages (TSP), manuals, etc.) as well as all TD processes (e.g., mission analysis, evaluation, etc.). Table 1-2 provides the procedures.

   b. For further information, see TRADOC Reg 350-70, chapters II-2 and II-3.

1-9.

Electronic support. Training developers should take advantage of TRADOC-provided automation tools available to support the applicable TD process. The automation tools and programs available change over time, both in what they can accomplish, and ease in use. Keep cognizant of these changes, and bring the need for upgraded automation requirements to the attention of the chain of command. A good source of current information is the Army Training Support Center website. Throughout this pamphlet there are references to the current automation tools, and guidance on how a tool will help perform a specific process/procedure. The current, primary training automated development tool is the Automated Systems Approach to Training (ASAT). Each Training/TD (task) proponent should have an ASAT point of contact (POC) in the TD organization that can assist with obtaining and using the program. Another resource is the ASAT Help Desk.

1-10.
Quality control

Quality control criteria. Each chapter in this pamphlet will include QC criteria for applying the evaluation process.

2-1.

Evaluation overview. The evaluation process is used to appraise quality (efficiency and effectiveness) of education/training, and TD. There is a process to follow to conduct an evaluation of training programs, products, and/or process, whether evaluating-

• The effectiveness of a training program, or the meeting of training objectives; appraising the effectiveness of new training;

• Equipment or environmental factors; or

• Extent that training and training products meet the needs of the Army (i.e., determine if training received is what is required in the field).

The evaluation process involves the following phases: planning, collecting, analyzing, reporting, and follow-up.

2-2.

Evaluator's role. The prevailing role of an education/training evaluator is to identify, articulate, and provide information to decisionmakers to assist them in making education/training decisions. These could include decisions on whether to continue training, whether and how to improve the training, and/or the cost effectiveness of the training. Evaluation is the process used to complete QC checks throughout the entire SAT process. As an evaluator, look for both strengths and weaknesses in the entire instructional system. Focus on-

• How well the graduates are meeting job performance requirements.

• Whether instruction is being provided that is not needed.

• Whether instruction that is needed is not being provided.

• Ways to improve the learner's performance on the job, as well as the instructional system.

• How well each instructional system component is contributing to overall instructional system quality. This includes, but is not limited to, lesson plans, instructors, equipment, training devices, interactive courseware, training schedules, audiovisual media, facilities, manpower, and costs.

• Whether the SAT process is being appropriately applied.

• How efficiently the education/training products meet the identified needs.

Support to decision-
makers

    a. Evaluators support decisionmakers at all levels by-

      (1) Collecting, analyzing, evaluating, and distributing feedback concerning such areas as:

          (a) Quality of current training and training support.

          (b) Sufficiency of doctrine.

          (c) Operability and maintainability of equipment and weapon systems from field user's viewpoint.

           (d) Readiness to meet new training requirements.

      (2) Providing standards and guidance for evaluating and accrediting Army training, training products, and institutions.

      (3) Identifying performance deficiencies.

      (4) Providing successful initiatives from the collection and analysis of trends data.

      (5) Ensuring quality of training by determining if the:

          (a) Instruction follows objectives and implementation procedures listed in the lesson plan, course management plan, student evaluation plan, and the approved program of instruction (POI).

          (b) Training aids used in classrooms support the objectives, are appropriate, understandable, and readable.

          (c) Environmental conditions contribute to a proper learning environment.

          (d) Instructor performance meets instructional standards.

          (e) Training development and training management are effective and efficient.

           (f) Collective training products are effective and efficient.

        (6) Ensuring staff and faculty have received required training.

Evaluation team make-up

    b. An evaluator is an independent observer that provides guidance and assistance while ensuring a quality process is applied, and quality products are produced. Conducting an evaluation is a team effort, guided by the training developer, who will function as the project leader. Executing an evaluation as a team effort is the most effective way to accomplish this process. Building the team may involve a matrix management approach. The team will consist of a project leader and subject matter experts (SMEs) (e.g., training developers, soldiers in units, instructors, task performers, and their supervisors, etc.). Depending on the purpose of the evaluation, the other team members will vary (i.e., ad hoc teams are built on the requirements of what is being evaluated). There are several major players that perform on every team.

      (1) Evaluators on the evaluation team primarily consist of a training developer (an expert in evaluation), and experts in other areas, depending upon the evaluation initiative.

        (a) The training developer, a GS-1750 (Instructional Systems Specialist), is normally in charge of the project. This is the individual trained in the conduct of evaluations. This person is the "training development" SME.

        (b) Ensure the "content/technical" SMEs are master experts in the military occupational specialty (MOS), area of concentration (AOC), or other areas being evaluated.

      (2) A difficulty encountered when setting up this team is selecting master SMEs. There are three levels of SMEs-apprentice, journeyman, and master. Make sure master training developers and master content area SMEs are on this team.

Team member roles

    c. Everyone taking part in an evaluation is part of the evaluation team. TRADOC Reg 350-70, paragraph II-3-4, provides the basic policy on TD teams. A variety of people are needed during the evaluation process, but the number and mix of personnel will vary based on the evaluation.

Evaluation team roles

      (1) The evaluation team shall ensure the evaluation:

        (a) Incorporates all elements that make up a SAT.

        (b) Determines if the right things are being trained; students learned; and training transferred to the job, met the unit's needs, and was efficient.

        (c) Provides support to decisionmakers at all levels.

        (d) Is thorough and comprehensive.

        (e) Is technically correct.

        (f) Results in a quality product by applying QC measures.

        (g) Complies with TRADOC TD guidance and policy.

        (h) Meets milestone requirements.

Training developer role

      (2) The training developer (GS-1750) is a key player on the evaluation team, who usually leads and manages the internal evaluation efforts (within the proponent school) and, depending on the purpose, may lead the external evaluation efforts. This role includes-

        (a) Keeping all people involved in the evaluation process informed of progress, problems encountered, developments, changes, and constraints.

        (b) Being responsive.

        (c) Providing results in a timely manner.

        (d) Keeping the project management plan updated, as appropriate.

        (e) Providing guidance to the team members concerning how they are to perform their responsibilities.

        (f) Providing evaluation training to the assigned team members.

        (g) Providing QC of the processes performed and products produced.

        (h) Providing assistance to those organizations being evaluated.

        (i) Conducting a standards-based evaluation (i.e., not checklist-based).

        (j) Conducts data collection and analysis; identifying deficiencies and efficiencies.

        (k) Providing recommendations to actual or potential problems/deficiencies.

        (l) Sharing identified efficiencies with appropriate organizations.

        (m) Assisting with the center/school's self-assessment efforts.

        (n) Ensuring all necessary corrective actions were completed.

SME role

      (3) The SME is the content or technical expert on the team. This SME is, or should be, the master performer of the action/activity being evaluated. No matter what the job-a training developer, a combat/doctrine developer, or an instructor-the SME is involved directly in the evaluation function. The SME is specifically responsible for the accuracy and completeness of the technical content, and the comprehensiveness of the content presented. The SME may perform other duties the team leader assigns, such as data collection.

Additional members

      (4) Depending on the purpose of the evaluation, the composition of the evaluation team may include, in addition to the training developer and SME, the following:

        (a) Visual information specialists.

        (b) Installation support person.

        (c) Safety Officer.

        (d) Resource management personnel.

Evaluation areas

    d. Evaluators should routinely evaluate all aspects of the production, management, and conduct of the Army education/training system. This includes such areas as:

      (1) Enlisted, officer, warrant officer, and DA civilian training.

        (a) Evaluate quality of instruction, standardization, test objectives, etc., to prepare students to perform effectively.

        (b) Ensure horizontal and vertical integration of tasks and training.

        (c) Ensure progressive and sequential training.

        (d) Evaluate quality of individual training strategies and Career Development Model.

      (2) Unit Training Products.

Evaluation areas

        (a) Evaluate quality of unit training strategies (unit long and short-range training strategies).

        (b) Review Combined Arms Training Strategies (CATS), Mission Training Plan (MTP), drills, exercises, Warfighting TSP, and Training Aids, Devices, Simulator and Simulations (TADSS) to ensure-

• Product usefulness.

• Technical/doctrinal content accuracy.

• Design and development are consistent with analysis data.

• Effective training sequence.

• Product meets needs.

• Safety, environment, and risk assessment/management considerations are incorporated.

• Deficiencies identified were corrected.

Note: Unit feedback on MTP, Soldier Training Publication (STP) and TSP are supposed to come through the automated unit training management program (i.e., Standard Army Training System/Unit Training Management Configuration).

      (3) Combined Arms Training Strategies.

        (a) Army Training Strategy (Headquarters (HQ) TRADOC only).

        (b) Unit Long Range Training Strategy.

        (c) Unit Short Range Training Strategy.

        (d) Individual Long Range Training Strategy.

        (e) Individual Short Range Training Strategy.

        (f) Self Development Training Strategy.

      (4) Threat. Evaluate to determine school's compliance with TRADOC regulations on threat, i.e., ensuring valid, accurate, and consistent threat portrayal is in the training and TD areas.

      (5) Doctrinal development. Evaluate the organization, procedures and management of the development process, and the integration of doctrine into training.

      (6) School evaluation system. Evaluate feedback systems and how training institutions use feedback to verify/validate decisions, or cause change.

      (7) Education/training management. Evaluate the management of implementation of education and training.

      (8) Training Resource Management.

         (a) Evaluate how well the school aligns resources with mission/training requirements to-

• Accomplish training student workload to standard.

• Allocate resources to support Commanding General, TRADOC priorities.

• Identify tradeoffs, and impacts of these tradeoffs.

• Consider efficiencies and economies.

• Apply savings to meet unresourced priorities.

        (b) Evaluate training resources involving programming and budgeting, manpower management, ammunition, equipment, and facilities.

        (c) Evaluate assignment and management of contracts.

        (d) Evaluate management of personnel assignments and staff and faculty training.

      (9) Training Requirements Analysis System (TRAS). Determine how well the institution is managing TRAS implementation and process to include:

        (a) TRAS document implementation and updates.

        (b) Maintenance of milestone schedules.

        (c) Meeting programming and budgeting windows for resources.

      (10) Systems Approach to Training process. Ensure the appropriate application of minimum essential elements of the SAT process.

      (11) Staff and faculty training. Evaluate the adequacy and relevancy of institution's training and development programs for the staff and faculty.

      (12) Automation systems which provide-

        (a) Proponent approved functionality (operational) requirements.

        (b) Seamless share data.

        (c) Automated systems easy to operate.

        (d) Multiple levels of help, to include use of program, technical assistance, and embedded training.

        (e) The capability to maintain the integrity of current data.

        (f) Key databases in the production cycle (available through the TD automated support system).

Remember, evaluators must have access to decisionmakers, and the evaluation findings, conclusions, and recommendations must be credible to be useful.

2-3.

Evaluation process description. Evaluations assess the quality of training and TD. Evaluators conduct evaluations by analyzing the current status of unit and individual performance, training products, programs, and processes by using a 5-phased process (planning, collecting data, data analysis, providing recommendations/reporting of findings, and following up on recommendations to ensure implementation). This process produces valid and reliable results that identify training deficiencies. Findings provide the basis for corrective recommendations through the chain of command. Findings also identify those areas of the training program performing efficiently and effectively.

Evaluation findings

    a. Evaluation efforts will determine whether:

      (1) Training and TD efforts have accomplished their intended purpose (i.e., the objective of the training has been met).

      (2) The right soldier was trained (i.e., all students fell within the target audience).

      (3) All students met specified entry-level standards.

      (4) Accreditation was performed properly, and accreditation data was obtained.

      (5) The TD process met minimum essential requirements of analysis, design, development, and implementation.

      (6) Training products met unit needs.

      (7) Soldiers' training met unit needs.

      (8) Appropriate quantity of training was received.

       (9) Soldiers needed instruction they did not receive.

      (10) Soldiers received instruction they did not need.

      (11) Training institutions are adhering to command training and education guidance.

Note: An overview of each evaluation phase is provided in paragraph 2-3b through f below.

Planning overview

    b. The first step in doing anything is proper planning. Evaluations are no exception and require thorough planning. Some routine evaluation duties, such as conducting a classroom observation, reviewing a test, or analyzing a group of student end-of-course critiques, do not require in-depth evaluation plans. Explain procedures for performing these routine duties in a local standard operating procedure, or equivalent document. However, major evaluations (i.e., in-depth evaluations of school training program, products, or processes) require an evaluation plan-the end product of the planning phase. This phase of the process is discussed in detail in chapter 3.

Data collection overview

    c. The data collection process involves determining what type of data is required, what data to collect (student performance, student feedback, audit trails, supervisors, graduates, etc.), where/who data is collected from (source), how much, and how data is collected (method/technique/instrument). Next, structure and develop the data collection method/instrument to collect data. The final step is to administer the instrument/technique to collect the data. This phase of the process is discussed in detail in chapter 4.

Analysis overview

    d. Analysis is the process of reviewing, synthesizing, summarizing, and processing evaluation data collected to develop initial findings concerning the issue being evaluated. Analysis reduces the huge volumes of raw data collected into a series of initial findings. The method used will depend on the type of data collected. This phase of the process is discussed in detail in chapter 5.

Reporting findings and recommend-
ations overview

    e. After evaluation data is collected and analyzed, the next step is to identify major findings and recommendations. Once identified, write a report to include:

      (1) References.

      (2) Background/problem.

      (3) Purpose of evaluation.

      (4) Summary of data collection procedures.

      (5) Results - major findings, conclusions, and recommendations. This phase of the process is discussed in detail in chapter 6.

Follow-up overview

    f. The follow-up phase determines if recommendations were implemented. This phase should take place within one year of the evaluation approval. This phase of the process is discussed in detail in chapter 7.

Feedback system

    g. Evaluators receive feedback from students, supervisors, soldiers, instructors, commanders, training developers, and affected organizations. Provide feedback to the same. The entire evaluation process is based on feedback, either formal or informal. Informal feedback consists of unsolicited information, which is transmitted either directly or indirectly to an action office, verbally, or in writing. Formal feedback, however, is solicited, specific in content, and transmitted through channels by means of a document designed specifically for the purpose (i.e., a survey or questionnaire). The feedback system, therefore, includes either the instruments or mechanisms used to elicit the information, and the means by which it is transmitted to the appropriate action office. It is the evaluator's responsibility to ensure that the feedback system is functioning, and evaluation results are relayed back to the appropriate action point in the system. Formal or information feedback, via external and internal evaluations, may drive future evaluation plans.

3-1.

Planning overview. This chapter addresses the planning phase of the evaluation process. It will assist evaluators in planning an evaluation and creating the Master Evaluation Plans (MEP) and Evaluation Project Management Plans. Add or delete additional actions, as the specific evaluation conducted requires. Evaluations that are not well-planned or managed will result in the collection of information that cannot be trusted (i.e., invalid and of no use). The Quality Assurance Office (QAO) is responsible for the development of MEPs and Evaluation Project Management Plans. However, involvement from other directorates and departments is essential in their development.

3-2.

Planning an evaluation. The first step to any evaluation is planning, which is probably the most difficult step. Everything that follows depends on how well the evaluation is conceptualized. This entails how well thoughts, ideas, and/or intuitions are expressed. In the planning process, assess all major goals/objectives, needs, resources and capabilities, and/or any other areas, which are eventually represented in an evaluation plan. There are several reasons to carefully plan an evaluation and document the plan in a written evaluation plan before conducting an evaluation. Planning-

• Helps ensure there is a valid need for doing the evaluation (i.e., reduces the chance of doing an evaluation that is not needed).

• Focuses the intent of the evaluation and prevents the evaluation from getting "off track".

• Forces the evaluator to "think through" the entire evaluation and plan for all actions required.

• Identifies and optimizes the use of the limited resources available for doing the evaluation.

• Ensures that everyone involved in the evaluation is informed of, and knows, their responsibilities.

• Identifies and prioritizes initiatives.

Types of evaluation plans

    a. Thorough planning is imperative to the success of a well-conducted evaluation. The complexity of the evaluation determines the amount of planning required. Routine evaluations will not require an in-depth evaluation plan. However, major evaluations of school training programs, products, or processes will require an in-depth evaluation plan. The end product of a well-instituted planning phase is an effective and efficient evaluation plan. There are two types of evaluation plans proponents are involved in developing.

      (1) Evaluation Project Management Plans are the individual plans developed for each evaluation conducted. They support the MEP. These plans are informal, simple, and unwritten (the project requirement may exist in a database, but not as a formal report) or they are a formal, very detailed, complex plan. Job Aid 350-70-4.3a provides a format of an Evaluation Project Management Plan. Guidance for developing a project management plan is provided in JA 350-70-4.3b.

      (2) The MEP is the planning document that provides all evaluation requirements for the next fiscal year (FY), and projections for the following 3-5 years. Evaluation requirements outlined in Evaluation Project Management Plans are included in the MEP. The Combined Arms Center (CAC) and Army Accessions Command (AAC) QAOs and each TRADOC center/school will prepare and provide the MEP to HQ, TRADOC by September for the upcoming FY. Centers/schools should send a courtesy copy of their MEP to CAC and ACC QAOs. These plans are mandatory per TRADOC Reg 350-70. Job Aid 350-70-4.3c provides a format of a MEP.

Planning procedural steps

    b. There are six major steps involved in planning an evaluation (see table 3-1). The training developer will perform these steps to successfully plan an evaluation. An Evaluation Project Management Plan is the outcome of this procedure.

Note: The evaluation plan (steps 5 and 6) is a by-product of planning an evaluation.

    h. Conducting literature search and reviewing training feedback may assist in determining-

      (1) If the reason/concern/problem/issue has been previously identified (determine if this is a one-time situation, or represents a recurring trend).

      (2) If the reason/concern/problem/issue has been previously studied. If the information required to address the problem/issue has already been addressed, it reduces the possibility of duplicating something that has already been done.

      (3) If there are other related training issues that need evaluating as part of the evaluation.

      (4) If any existing studies pertinent to the evaluation exist.

Note: The establishment and maintenance of a feedback system will assist the organization in organizing feedback collected from different sources.

    i. Collect and review training documents to become familiar with the training programs, products, and/or processes being evaluated. Job Aid 350-70-4.3b (Step 4) provides a list of the different training documents that may need reviewing. Basic questions to ask during this step to assist in collecting existing documentation are:

      (1) Is there a written tasking from the assistant commandant or appropriate tasking authority to do an evaluation?

      (2) Are there written comments from the field commanders? Who were the field commanders that made the comments? And who recorded the commander's comments?

      (3) Is there a trip report indicating field problems?

      (4) Is there other documentation indicating a training or education problem?

Evaluation plans

     j. Development of evaluation plans is discussed in paragraph 3-3 below.

3-3.

Evaluation project management plans.

    a. A manager must develop a realistic estimate of the resources required to accomplish evaluation projects, establish milestones, and allocate the available resources to the project. Evaluation plans will achieve this by identifying the total data/information required, and specifying the methodology for collecting; processing, and analyzing that data/information; and validating the measuring instruments. These plans answer the basic questions:

      (1) Why is the evaluation being conducted?

      (2) What will the evaluation accomplish?

      (3) How is the evaluation done?

      (4) Who will conduct and support the evaluation?

      (5) What resources will the evaluation require?

      (6) When will the evaluation be completed?

      (7) What references are used during the evaluation?

      (8) Who will receive the final report?

Elements of the evaluation plan are discussed below. These elements provide additional insight into the basic questions addressed in an evaluation plan.

Elements of evaluation plans

    b. There are specific critical elements that all Evaluation Project Management Plans must include. Each basic question listed above has associated elements to address for a successful evaluation. Each essential element is discussed in JA 350-70-4.3b (Step 5). The evaluation plan elements are outlined in general terms in table 3-5.

3-4.

Quality control criteria for planning. TD/task proponents perform QC actions as an inherent part of the SAT process. When planning an evaluation, ensure:

       a. Areas selected for evaluation were based on one or more of the following:

           (1) Identified deficiency/discrepancy.

           (2) Scheduled evaluation.

           (3) A request from an internal or external source.

        b. Purpose of the evaluation is clearly stated.

        c. Scope of the evaluation describes precisely what is covered.

        d. A thorough literature search was conducted to identify relevant information.

        e. The final evaluation plan was developed and includes:

            (1) Purpose.

            (2) Background.

            (3) Scope.

            (4) Objectives.

            (5) Essential elements of analysis.

            (6) Methodology.

            (7) Evaluation recommendations and responsibilities.

            (8) Resource and support requirements.

            (9) Timelines.

            (10) Feedback plans of results.

       f. Master Evaluation Plan was revised, to include all evaluation project management plans, as they were developed or modified.

Chapter 4
Collecting Evaluation Data

4-1.

Data collection overview. This chapter will discuss the data collection phase of the evaluation process, to include selecting, using, and defending appropriate methods of collecting data. It will provide guidance on designing and using surveys and interviews, as well as the use of observations.

4-2.

Data collection procedures. During the planning phase for evaluation, identify the information required to address the overall evaluation effort, as well as how to collect the required information. Data collection is the process of gathering, collating, and preparing data for the purpose of processing (analyzing) to obtain desired results. To determine the worth of any training, training product, or process, the data is collected to analyze. Relevant data should come from several sources, with more than one method used to collect data. General data collection sources and methods are document reviews, individual interviews, group interviews, surveys, tests or time trials, and personal observations. The intent is to collect sufficient raw data to ensure a successful analysis. The techniques or instruments used will depend on the type of data required for collection. See table 4-1 for general procedural guidance for collecting data.

 a. Information collected is based on:

          (1) Area of interest.

          (2) Questions that must be answered.

          (3) Information needed to answer these questions.

Job Aid 350-70-4.4a, paragraph 1a, provides general guidance on achieving this requirement.

Note: If an evaluation is conducted on the TD process, collect data based on the QC measures provided in the individual TRADOC pamphlets (i.e., analysis, design, development, validation).

           (2) Combat Training Center (CTC) rotation feedback.

           (3) Lesson learned collectors.

           (4) Internal training review report.

           (5) Supervisor feedback (surveys/questionnaire/interview).

           (6) Graduate feedback (surveys/questionnaire/interview).

           (7) Student feedback (surveys/questionnaire/interview).

           (8) Instructor feedback (surveys/questionnaire/interview).

           (9) Unit commander feedback (surveys/questionnaire/interview).

           (10) Subject matter experts.

           (11) Monitoring of training and instructors (surveys/questionnaire/ interview).

           (12) Training developers.

           (13) Doctrine and combat developers.

           (14) Field visits or unit feedback.

           (15) Comparison between what was supposed to be trained versus what was trained (validated training).

           (16) Unsolicited feedback.

           (17) Analysis of trainee test results.

           (18) Analysis of validity of assessment/test employed.

           (19) Second party training review reports.

           (20) Accident reports.

           (21) Conferences and seminars.

           (22) Document reviews.

Technical approach to data collection

      c. There are two technical approaches to collect data that may be used-quantitative or qualitative.

           (1) Quantitative data indicates an amount (how much or how many) and is measured on a numerical scale. This method is used most frequently. A quantitative approach is:

               (a) Objective.

               (b) Reliable.

               (c) Easy to use.

               (d) Not as time consuming as qualitative.

For example, a runner's time in a 10-kilometer race is a quantitative variable, because it measures the amount of time it took to run 10 kilometers.

      (2) Qualitative data is usually in the form of words, not numbers. It can be thought of as the perception about the quality of something. Qualitative data either labels sampling units, or places them into categories. In general terms, such data is identified or named by some quality, category, or characteristic. Such data are nominal scales (as in agree, disagree, or no opinion in survey instruments). When qualitative data is analyzed, it usually features the proportion that lies in a given category. For example, what proportion of the survey respondents agreed training developers are best qualified to develop ITPs?

Types of data collection methods

   d. Data collection instruments contain some style of questions designed in a systematic, highly defined approach. Job Aid 350-70-4.4a, paragraph 1b, provides general guidance on achieving this requirement. Their purpose is to obtain consistent data that is compared, summarized, and, if the data is quantitative, subjected to statistical analysis. Use a single or combination of data collection methods to collect data. Obtain evaluative data from sources previously discussed. Methods for collecting data are discussed below. Details are provided for the first four, as they are the primary methods used.

         (1) Questionnaires/surveys collect large samples of information from individuals associated with, or affected by, the training programs, products, or process being evaluated (students, graduates, soldiers' first line supervisors, and instructors). Structured questionnaires are most valuable in obtaining straightforward factual information. A well-prepared and administered questionnaire will yield valid data. An appropriate timeframe for administering questionnaires/surveys to graduates, or first line supervisors, is usually no sooner than 6 months after soldier has graduated from the course. Carefully prepared, properly distributed, objectively executed, and critically analyzed questionnaires can provide constructive information on:

         (a) The ability of recent graduates to perform specific tasks for which they were prepared.

         (b) The specific nature of instructional deficiencies, as perceived by the student or graduate.

         (c) Details of the jobs actually being performed by the graduates.

         (d) Instruction not needed by the graduates in their jobs.

         (e) Other areas that require analysis.

Note: For external evaluation, Army Research Institute (ARI) has developed the Automated Survey Generator (AUTOGEN) Software Program that creates the survey. This software is discussed in more detail in chapter 9.

       (2) Interview guides, like questionnaires, collect information from individuals associated with, or affected by, the training programs, products, or process being evaluated. When interviewing, generally a list of questions is used (i.e., an interview guide), the questions are asked verbally, and the participants' answers are recorded. The questions are not necessarily in a mandated order, but should ensure the data is collected in a systematic and standardized way. Depending on the participant's response (i.e., a thorough answer to a question that is asked later has already been provided), the order of the questions may need to change, which requires knowledge and understanding of the questions. Although interviews are economical, flexible, and a good means of collecting information, maintain neutrality as an interviewer, and do not affect the participant's perception of the question or answer given. However, interviews allow evaluators to gain a great depth of understanding of problems and issues. Interviews can also yield anecdotal examples and vignettes that bring life to the problems or issue at hand.

        (3) Observations are used during the implementation of training to assure training is being delivered in the right sequence, and conditions and standards are being met. This method also involves sending evaluators to observe and interview soldiers and their supervisors on the job. Direct observation of soldiers as they perform their job, combined with interviews, provides a useful source of task information. Job aids (which are also referred to as worksheets, observation forms, or checklists) assist in collecting information during the observation of training programs, products, or processes. Use HQ TRADOC Form 350-70-4-1-R-E when observing training programs, observing individual or unit performance, or reviewing training documents, products, or processes. Use this form to record TD, training management, and instructor observations. In addition to the reproducible format at the back of this pamphlet, a FormFlow version is also available to provide the observer the capability to digitally record observations. The Staff and Faculty Development Program also uses Instructor Performance Checklists to evaluate basic instructor performance, classroom instructors, video teletraining instructors, small group instructors, and After-Action Review (AAR) performance. These checklists are found in TRADOC Reg 350-70, chapter III-4.

         (4) Tests or timed trials measure the learner's ability to perform critical tasks to standards, and/or the learner's obtainment of supporting skills and knowledge. Test results are also used to determine the effectiveness of training, and identify areas of improvement.

         (5) Student critiques.

         (6) Instructor questionnaires or interviews.

         (7) Examination of training documentation/publications.

         (8) Expert panel comments.

         (9) Job performance evaluations.

        (10) Major training exercises.

        (11) Job Aids.

        (12) Reports (course, evaluation and/or company).

Design data collection instruments

     e. Detailed procedural guidance for developing data collection instruments are found in JA 350-70-4.4a, paragraph 2. Remember to consider early in the design of collection instruments how the measures are scored, who will score it, and how the data is analyzed.

      (1) Development and administration of questionnaires and structured interviews (JA 350-70-4.4a, paragraphs 2c through 2e(4)).

      (2) Additional guidance on interviews (JA 350-70-4.4b and JA 350-70-4.4c).

   f. When preparing to develop a questionnaire or interview, constantly apply the following criteria:

     (1) To what extent might a question influence the respondents to show themselves in a good light?

     (2) To what extent might a question influence a respondent to be unduly helpful by attempting to anticipate what the researcher wants to hear or find out?

     (3) To what extent might a question be asking for information about/from respondents that they are not certain, and perhaps not likely, to know about themselves?

The validity of questionnaire and interview items is limited by all three considerations above.

Methods to administer data collection instruments

   g. Once the data collection instrument is developed, consider to whom and how it is administered. (Determining the sample size is discussed in the next section.) Various methods to employ are listed below.

      (1) Mailed questionnaire.

      (2) Personally administered questionnaire.

      (3) Electronically administered questionnaire.

      (4) Telephonic interview.

      (5) Face-to-face interview.

      (6) Video teleconference interview.

Regardless of the data collection instrument used, administer those designed to research training transfer to the job, to the graduate or supervisor, at least 6 months after the graduate's return to duty.

Sample size for surveys

     h. Paramount to providing accurate data to senior leadership requires determining how many completed surveys are needed to produce a reliable report. Additionally, provide senior leadership with the confidence that the information collected is representative of the target audience. Table 4-2 provides steps to determine how many completed surveys are needed. Job Aid 350-70-4.4d assists with accomplishing procedures in table 4-2.

Standard sampling techniques

   i. Since it is frequently impossible to study an entire population, evaluators rely on sampling to acquire a section of the population from which data is collected. Sampling is considering a segment of a target population that is representative of that target population. If the sample is not a true representation of the population, then the data analyzed may be erroneous. The most commonly used sampling procedures are:

     (1) Simple Random Sampling-a sampling technique in which each individual in a population is chosen by chance, and each member of the population has an equal chance of being included. The group of subjects (the sample) is selected from a larger group (the population).

    (2) Stratified Random Sampling - a sampling technique that divides the population into categories (strata) and then data is collected from the strata by simple random sampling. These categories are based on certain characteristics relevant to the questionnaire (i.e., age, training level, or gender).

Raw data collected

j. The end product of the collection phase is raw information. This raw data is in the form of completed questionnaires, interview guides, observation checklists, and other completed data collection instruments. Analyze this raw data.

4-3.

Quality control criteria for data collection. Training development/task proponents perform QC actions as an inherent part of the SAT process. Regardless of the collection data instrument used, ensure-

     a. Everyone involved understands the purpose of the evaluation.

     b. Data sources were relevant to provide appropriate, pertinent, and reliable information.

     c. Data collection instruments were appropriately designed to collect required data.

     d. Data sources provided appropriate, pertinent, and reliable information.

     e. Instrument was validated prior to distribution.

     f. Acceptable sample size has been determined to collect sufficient data.

     g. Appropriate sampling technique has been used to select either a simple, or stratified, sample of the target audience.

5-1.

Evaluation analysis overview. This chapter discusses the conduct of the data analysis phase of the evaluation process. This will assist evaluators with analyzing findings to identify trends and systemic problems. This chapter includes reviewing, summarizing, and analyzing raw data, as well as interpreting the results of the analysis.

5-2.

Analysis description. The analysis process transforms large volumes of raw data collected into usable findings. Simply stated, analysis is the process of reviewing, summarizing, and processing information to develop initial, sound findings/recommendations concerning the issue being evaluated. (Note: Do not confuse this analysis with the analysis phase of the SAT.) Before starting an analysis of collected data, ensure-

• Data has been collected from a sufficient and appropriate sampling/reliable source.

• Adequate data samples have been collected to validate the reliability of the findings.

• Special attention is given to notes made by respondents on questionnaires or answers to supplemental questions included in the questionnaire.

• Data that contains the halo effect (indiscriminate rating of all items positively) or central tendency (indiscriminate rating of items in the center of the scale) is used with caution.

• Data for electronic analysis has been collected in an easily processed form.

Analysis procedural steps

a. Table 5-1 provides the steps a training developer should perform when conducting any analysis.

The end product of the analysis is a series of initial findings that are addressed in the Evaluation Report.

Types of data analysis methods

      b. Evaluators may analyze data in various ways. The data is either in the form of qualitative data (expressed in narratives or words), or quantitative data (expressed in numbers). The type of data collected will depend on the objectives of the evaluation. These are determined when planning the evaluation, and ultimately developing an evaluation plan.

Review data

     c. Ensure data is valid and reliable. Triangulation (using multiple methods to study the same thing) can corroborate evidence, and increase validity, especially for qualitative findings. Examples of events that could result in invalid and unreliable data:

         (1) Different data collectors doing interviews or observations (resulting in different interpretations).

        (2) Different interpretations of notes recorded during observations if not interpreted by the evaluator conducting the observation.

        (3) Questionnaire answers that are incomplete, illegible, or not understandable.

Data integrity

     d. Check data for integrity as a part of every evaluation, especially when there were several data collectors involved, questionnaires were mailed, or an unstructured data collection method was used. Closely check data for integrity when there is less control on how it is collected. When reviewing data for integrity, ensure:

       (1) Responses are complete-a blank next to a question could mean do not know, refused to answer, or the question was not applicable.

       (2) Responses are understandable-either the data collector's written response to an observation, or the answer provided by the survey taker.

Adequate samples

    e. Ensure adequate data samples are collected to validate the reliability of the findings. When reviewing data for validity, ensure:

        (1) Responses are consistent-different questions, pertaining to the same subject, on the same instrument, are consistent (ask the same question in two different ways, to see if the same answer is obtained). Also, when using a rating scale for rating a list of items, look for patterns of responses that may indicate the respondent did not seriously answer the question.

       (2) Responses are uniform-if different data collectors for administering interviews or observations are used, make sure that collectors follow uniform procedures for collecting and recording data.

       (3) Responses are appropriate-if a response does not pertain to the purposes of the evaluation or the question asked, discard the responses. If integrity problems are not resolved, discard the data.

Summarize data

   f. When data is summarized, simply condense the data for analysis.

      (1) The easiest and most accurate method for summarizing large amounts of quantitative data is to use an automated statistical program.

      (2) Summarizing qualitative data is much more difficult to accurately and effectively summarize. Job Aid 350-70-4.5 provides guidance for summarizing qualitative data.

Analyze data

    g. Before analyzing any data, ensure all quantitative data is entered into a computer, and all qualitative data is summarized and condensed into categories. Keep in mind exactly why the data is being analyzed (i.e., identify what specific questions are needed).

    h. Statistical analysis is required for analyzing quantitative data. (Note: Statistics simply show things about the data that are otherwise not seen.) There are two types of statistics:

      (1) Descriptive statistics give measurements that simply "describe" the data collected. This includes such measures as:

        (a) Mean - the average value.

        (b) Range - the extreme values.

        (c) Standard deviation - the degree to which values are dispersed.

      (2) Inferential statistics "infer" something about the total population from which the data was collected, usually by a random process. It includes such measurements as-

        (a) t tests-a statistical analysis used to infer something about two different populations, based on data collected from a sample of each of the populations.

        (b) Chi-square-a statistical test commonly used to compare observed or actual frequency with expected or hypothesized frequency of two variables.

        (c) Analysis of Variance-a statistical analysis used to determine whether there are any statistically significant mean differences between two or more groups, or one or more variables, or factors.

Individual descriptions on how to use these various methods for analyzing data are not provided. If the evaluation plan requires using unfamiliar statistical analysis, consult with a statistician or evaluator experienced in statistics.

Interpret the analysis results

     i. Interpret the analysis in common sense terms, and be able to explain the results. Interpretation of analysis is one of the most difficult steps in this phase of evaluation. Keep in mind the purpose of the evaluation as data results are interpreted. Annotate all trends identified, and include in the final report. Qualitative data is often considered less objective than quantitative data, but can provide very useful information, if procedures are followed, especially when looking at themes and relationships at the case level. Quantitative data, though more scientific, requires statistical manipulation to represent findings.

As with analysis, interpretation of results may call for the use of unfamiliar statistical procedures. Solicit the assistance of a statistician, or more experienced evaluator, when using unfamiliar statistical methods.

5-3.

Quality control criteria for evaluation analysis. TD/task proponents perform QC actions as an inherent part of the SAT process. When analyzing data, ensure-

    a. The integrity of the raw data, by verifying the completeness and understandability of responses.

    b. Accuracy of data summaries was verified.

    c. Data analysis was appropriate for the type of data collected.

    d. Results of the analysis were interpreted to provide usable, accurate, and pertinent recommendations.

6-1.

Evaluation report preparation overview. This chapter provides guidance on how to develop a report, which captures the results of an evaluation (phase four of the evaluation process). An evaluation report will include findings, conclusions, and recommendations. This chapter will assist in preparing, staffing, and distributing the report/recommendations.

6-2.

Evaluation report descriptions. Formal and informal reports are an ongoing process during the life of an evaluation; do not consider them as just an "end of evaluation" product. Establish the primary audience during the planning phase, and write the report to that audience's interest. The report should be concise and presented in a way that will ensure the evaluation information is used to its fullest potential. The report summarizes the results of the evaluation (i.e., findings, conclusions, and recommendations). The report is provided to senior leaders for their information and guidance.

6-3.

Report preparation steps. Evaluators should perform the steps in table 6-1, which provides the general procedural guidance for preparing the evaluation reports.

6-4.

Prepare draft report. The evaluation report is a vital document that summarizes the results of the evaluation (i.e., findings, conclusions, and recommendations). From the findings, develop conclusions, and for each identified problem area, provide specific recommendations for management to consider. There are different ways of reporting information, depending on how the report is used, the target audience, and the impact that the evaluation will have. Be as concise and simple as possible, while ensuring all required information is included.

Types of evaluation reports

     a. The length and format of an evaluation report can vary significantly. Factors to consider are the audience for whom the report is intended, complexity of the evaluation, and/or the impact the evaluation results will have on the organization. For purposes of standardization, the reports are categorized into two types-Executive Summary and Detailed Evaluation Report.

Executive Summary

       (1) The Executive Summary is a synopsis of the evaluation, and requires a response to the evaluation recommendations. The Executive Summary is a memorandum designed for senior leadership that includes:

             (a) Background.

             (b) Purpose and objectives.

             (c) Methods.

             (d) Major findings, recommendations, and who is responsible to implement (or lead the implementation of) recommendations.

             (e) Suspense date for a response to recommendations.

Job Aid 350-70-4.6a provides a format of an Executive Summary. For most evaluations, the Executive Summary is sufficient. Keep on file the supporting data and documentation required to explain and support findings.

Detailed Evaluation Report

        (2) Detailed Evaluation Reports are lengthy, formal evaluation reports that explain in detail those findings included in the Executive Summary listed above. A detailed report is required if:

             (a) Intended audience is outside the installation, and the evaluation specifics (to include supporting data) cannot be explained in person.

             (b) Evaluation will have a very significant impact on training/training resources.

             (c) Evaluation involves very complex data collection or analysis methodologies that require explanation, and are supported with data.

      (3) In addition to the items in an executive summary, the detailed report includes:

           (a) An Executive Summary.

           (b) A discussion of limitations.

           (c) A list of assumptions.

           (d) An identification of Essential Elements of Analysis.

           (e) Appendices for data summaries, which may include Data Collection Instruments, the Evaluation Plan, etc.

           (f) A memorandum requesting concurrence of the director on the recommendations, and organization responsible for implementing recommendations. Format the memorandum similar to paragraph five of the Executive Summary cited in JA 350-70-4.6a. A format for a Detailed Report is provided in JA 350-70-4.6b.

Report preparation hints

      b. Whether an Executive Summary or a Detailed Evaluation Report is prepared, keep the report as brief as possible. The following suggestions will assist in preparing either report.

        (1) Only include information that the intended reader needs.

        (2) Keep it simple.

        (3) Do not use acronyms unknown to readers.

        (4) Do not write at a reading level drastically above, or below, the reader.

        (5) Explain in simple terms any complex data collection, or analysis methodologies.

        (6) Stay focused on the Essential Elements of Analysis when writing the report.

6-5.

Staff draft report for review/concurrence.

    a. The procedures for staffing a draft report should begin within the organization, and then to involved organizations. The steps are as follows:

       (1) Distribute the draft report internally to individuals within the office to review the report content.

       (2) Provide the director a copy of the draft report for review/approval.

       (3) Ensure director signs a memorandum to distribute to organizations responsible for implementing recommendations. (Note: If an Executive Summary is prepared, the director will sign it. If a Detailed Evaluation Report is prepared, the director signs the accompanying memorandum.)

       (4) Distribute draft report to all organizations affected by the evaluation for their review and concurrence (or nonconcurrence) on recommendations made. Request a response back within 30 days.

    b. If nonconcurrence comments are received, resolve any disagreements prior to sending the final report to the decisionmaker. Use coordination meetings, if necessary, to resolve disagreements.

    c. If no agreements are reached on nonconcurrences, send the report with explanations of all nonconcurrences to the decisionmaker for a final decision.

6-6.

Obtain final approval of recommendations. In this phase, a final report is provided to the decisionmaker (i.e., assistant commandant) for final approval of the overall report and the individual recommendations.

Draft report

     a. Send a memorandum to the decisionmaker, along with the report and copies of the responses received as a result of the staffing from the Staffing Draft Report step. Job Aid 350-70-4.6c provides a sample format for the memorandum sent to the decisionmaker.

     b. After the decisionmaker has approved or disapproved the recommendations, distribute the final report.

6-7.

Distribute report/recommendations for action, and conduct follow-up check. In this phase, the final report is distributed to all organizations that are responsible for implementing the approved recommendations, with a copy of the memorandum from the decisionmaker, showing final approval of recommendations.

      a. Provide a suspense date for written responses to responsible organizations.

     b. Responses must include:

         (1) Actions they have taken thus far.

         (2) Milestones for future actions they will take.

      c. Inform the organization that a follow-up check is conducted to ensure that they have taken the actions, and provide a predetermined time to conduct this check-up.

6-8.

Quality control criteria for preparing evaluation reports. Training development/task proponents perform QC actions as an inherent part of the SAT process. When developing an evaluation report, ensure-

    a. The final evaluation report was focused on the objectives of the evaluation.

    b. The final evaluation report includes, at a minimum:

      (1) Findings.

      (2) Conclusions.

      (3) Recommendations.

    c. Reports were staffed to all appropriate organizations.

    d. Final report was approved by chain of command before distribution.

    e. Final report was distributed to all relevant organizations (i.e., all organizations that are responsible for implementing the approved recommendations).

7-1.

Evaluation follow-up overview. This chapter provides guidance on how to conduct a follow-up of the evaluation effort (phase 5 of the evaluation process). During this phase, it is determined if recommendations made as a result of the evaluation were actually implemented. This chapter will assist in conducting follow-ups, and includes-

• Preparing for a follow-up evaluation.

• Conducting the follow-up evaluation.

• Preparing and staffing the report.

7-2.

Follow-up description. The follow-up phase is sometimes considered the most important part of the evaluation process, but often is the most overlooked. Conducting a follow-up will determine if recommendations were implemented, and if they resulted in an improvement to the training. This phase should take place within one year of the evaluation approval. If the recommendations are not followed up, do not conduct the evaluation.

Follow-up steps

    a. Though the decisionmaker has signed off on the evaluation and approved the recommendations, there is still no guarantee that those responsible will implement the recommendations. Conduct a follow-up to ensure implementation. See table 7-1 for procedural steps for conducting a follow-up.

Input action milestones into tracking system

    b. During the reporting phase, determine actions the responsible organizations have taken so far, and their plans for future actions. To begin the follow-up, update information from these organizations to determine status of actions taken and planned. These actions are put into a tracking system.

   c. The tracking system:

      (1) Includes all present or future actions taken by an organization.

      (2) Is automated to simplify use of data collected.

      (3) Has the projected date for the follow-up. Projected date will depend on the milestones established for the actions, and on local policies and procedures for performing follow-ups.

      (4) Provides an audit trail on all actions taken as a result of the evaluation.

Conduct the follow-up

    d. The purpose of the follow-up is to ensure that those responsible have implemented the approved recommendations. How a follow-up is conducted is dependent on the actions themselves, resources available, and local policy for follow-ups. The primary goal of a follow-up is to ensure the organization has implemented the actions. Follow-up measures may include:

      (1) Meeting with personnel from the organizations.

      (2) Observing training.

      (3) Reviewing training documentation.

      (4) Examining training records.

    e. Reconcile any differences of opinion on actions that they have or have not taken.

    f. Enter new milestones into the tracking system if the organization has not implemented actions.

    g. Record action as complete in the tracking system when all corrective actions have occurred.

Prepare a follow-up report

    h. Write up the results of the follow-up in a Follow-up Report. A sample Follow-up Report is found in JA 350-70-4.7. Report should:

      (1) Summarize each action taken for each approved recommendation.

      (2) Be brief.

      (3) Include a new milestone, if more work is required.

Staff follow-up report

    i. Staff the follow-up report to inform everyone of the follow-up results, and to document the follow-up. Staff the report to:

     (1) Organizations responsible for implementing the recommendations.

     (2) Other affected offices.

7-3.

Quality control criteria for conducting follow-up evaluations. Training development/task proponents perform QC actions as an inherent part of the SAT process. When conducting a follow-up of an evaluation, ensure-

    a. Follow-on actions were entered into a tracking system. This tracking system includes-

      (1) Actions taken by all responsible organizations.

      (2) Projected dates for verifying collective action were implemented.

      (3) Dates for scheduled follow-up evaluation.

      (4) An audit trail of all actions taken as a result of the evaluation.

    b. Recommendations were implemented.

    c. A follow-up report has been prepared, detailing the corrective action taken for each approved recommendation.

   d. Follow-up report was staffed to all appropriate organizations.

8-1.

Internal evaluation overview. This chapter provides "how-to" guidance for conducting internal evaluations. Internal evaluations cover the evaluation of-

• Overall training development process.

• Student learning.

• Instructional materials.

• Personnel.

• Instructional resources.

• Implementation.

• Training products.

• Staff and faculty education/training.

• Instructor performance.

• Infrastructure requirements.

• Course Management Plan.

• TRAS documents.

• Training Support Products.

8-2.

Internal evaluation description. The purpose of an internal evaluation is to improve the quality and effectiveness of the instructional system, by providing sufficient, high-quality data to decisionmakers upon which they can make sound, informed decisions about the training and education. During an internal evaluation, gather internal feedback and management data from the education/training instructional system environment. Periodic internal evaluations may identify weaknesses/problems, as well as strengths, of the TD and instructional system. Internal evaluation is a deeper requirement than checking instructor techniques and method of instruction. It is a check of the quality of the content in regards to what is being taught, and what the students are assimilating. In an internal evaluation, make a comparison between the course objectives and standards applied in the training environment, and the objectives and standards specified in course development documents. In addition, evaluate school's/center's control of the total training environment, and promptness in moving graduates to units, to include proper application of the TD process.

    a. Internal evaluations focus on the TD process and the measurement of learning that was gained from the training program, in an effort to continually improve instructional quality and effectiveness.

Where conducted

    b. If an internal evaluation on the application of the SAT process (i.e., analysis, design, development, implementation, and/or evaluation) is conducted, then conduct the evaluation in the TD environment. The QC checks outlined in the related TRADOC pamphlets will assist in accomplishing an evaluation of these areas. If an internal evaluation of the implementation of training is conducted, then conduct the evaluation wherever the training is being implemented:

      (1) Training institution.

      (2) TASS Training Battalion.

      (3) Distributed Learning Facilities.

      (4) Unit.

      (5) Home.

NOTE: Internal Evaluations are NOT conducted ONLY in resident schools.

Objectives of internal evaluation

    c. Internal evaluations provide school/decisionmakers with a method of assuring that training and training products are correctly developed and implemented IAW the appropriate standards of SAT. The school's/decisionmakers' responsibilities include ensuring that:

      (1) A systems approach is being appropriately applied to the analysis, design, development, implementation, and evaluation of all training and training materials (i.e., products and/or programs).

      (2) Records/documents that clearly explain the decision process are maintained.

      (3) Analysts have interacted with school/center personnel charged to conduct the evaluation function, to ensure that feedback data required for analysis are identified in evaluation plans.

      (4) Training development, and training functions, are effective and efficient.

      (5) Task inventories from new or modified equipment development processes have been provided to the appropriate office.

     (6) Access to information from analysis of collective training requirements, such as MTP and standard drills, are available.

     (7) Collective training programs including the appropriate MTP, and standard drills are developed that will support units/organizations for which they are proponent.

      (8) Staff and faculty personnel receive training relevant to their duty assignment. As a minimum, staff and faculty training will consist of an introduction to a SAT and, when required, courses where instructional/facilitation techniques are certified.

      (9) The instructional base is providing the appropriate/intended training, and using the appropriate training products (POIs and lesson plans/TSPs are examples).

      (10) The instructional system is producing a qualified graduate in an effective and cost-efficient way.

       (11) Quality control mechanisms are in place for developing and implementing training and training products.

        (12) Objectives of the training have been met.

        (13) The infrastructure (e.g., TD facilities, classrooms, shop areas, learning facilities, billets, training areas, and ranges) adequately supports all phases of the SAT.

8-3.

Internal evaluation procedures. Table 8-1 provides the general procedural guidance for conducting an internal evaluation. The "how-to" procedures for a majority of the steps below are discussed in-depth in chapters 3 through 7.

Project manage-
ment plans

    a. Plans to conduct an internal evaluation are detailed in Evaluation Project Management Plans. A guide for preparation of Evaluation Plans is provided in chapter 3.

Feedback channels

     b. It is essential to establish a process that ensures feedback is provided to the appropriate individuals/organizations. It is also imperative that feedback is constantly evaluated and applied, to check and improve the training system. Feedback helps:

      (1) Ensure training materials are maintained that reflect current doctrine, conditions, equipment, and procedures.

      (2) Ensure intended educational objectives have been met.

      (3) Provide information to students on individual performance.

      (4) Identify substandard performance or trends as early as possible for individual assistance or corrective action.

     (5) Assess the effectiveness of training and instructional methods used in training implementation.

     (6) Provide information to base shifts in emphasis and allocation of scarce resources.

Checklists

   c. Job Aids listed in paragraph 8-4g below provide assistance in evaluating products and processes.

Data collection

   d. Data collection techniques used during internal evaluations are discussed in chapter 4, to include questionnaires/surveys, interviews, observations, and tests.

Analyze data

   e. Detailed guidance for analyzing data is provided in chapter 5. In addition to the procedures outlined in chapter 5, an analysis of internal evaluation data should specifically include the following measures:

      (1) Review the POI and instructional materials (lesson plans/TSPs, instructor guides, student guides, and other instructional materials are examples) to determine whether they are current, adequate, and in agreement.

   (a) Compare terminal and/or enabling objective standards with the standards in the POI, to determine if the requirements of the POI are being met.

   (b) Compare lesson plan/TSP with course being taught to determine if they are the same.

      (2) Compare stated training resource requirements with actual resources, to determine if adequate resources are available to support, operate, and/or maintain the instructional system.

      (3) Review instructor records to ensure instructors are qualified to teach the POI.

      (4) Review test and measurement data to determine if students are meeting the terminal and enabling objectives.

      (5) Analyze test and measurement instruments to determine if they are valid and reliable.

      (6) Employ QC checklists and JAs for the various TD phases.

Evaluation reports

    f. Provide results of data analyses to the appropriate elements within the school to influence the development process. Detailed guidance for preparing and distributing reports is provided in chapter 6.

Follow-up

    g. It is important to determine if deficiencies identified during an internal evaluation have been corrected. The steps for conducting a follow-up evaluation are provided in chapter 7.

8-4.

Areas to consider/review during internal evaluations. Internal evaluation will assist in establishing and maintaining the desired level of QA. Therefore, it is not excessive to have a system of in-process reviews of key areas. Depending on the circumstances, and the directives of proponent evaluation policies and plans, the evaluator needs to review various documents/data during an internal evaluation. This will ensure all required data to conduct a successful internal evaluation is available. Some areas to consider are listed below:

Docu-
mentation
to review

    a. A record of training needs assessment, to determine if training is required, or some other aspect of the installation/unit environment needs addressing, e.g., personnel management, maintenance and logistical support, and equipment availability and operability.

    b. The specific target population description (developed during job analysis), to ensure training and training support materials is developed with the user in mind. Internal evaluations should check how well the analysis, design, development, and implementation processes and products match the target population description, and the needs of field users.

    c. Records/documents and data sampled to determine compliance with HQ TRADOC and local policy and guidelines. There are certain documentations of critical "decision points" in the TD process which internal evaluators should check, as a minimum, to see how well the decisions stand up to the tests of soundness of rationale, and validity of selection. This may include, but is not limited to:

      (1) Critical task selection.

      (2) Training site selection.

      (3) Job performance measure description.

      (4) Training method and media selection.

      (5) Validation of tests, training materials, and courses.

    d. Course control documents to determine if there are any discrepancies between the planned course, and the course that was actually implemented. Study each component and procedure authorized, and/or required by these documents.

    e. Resource documents to ensure all required resources are available and sufficient. Recommended corrective actions should consider system constraints. Evaluate actions taken and whether:

      (1) Facilities (instructional and support) are available and adequate.

      (2) Equipment and training devices (instructional, support, and test and measurement), and supplies are available and meet system requirements.

      (3) Human resources (instructional developers, instructors, students, and courseware maintenance personnel) are available.

      (4) There is adequate time allotted for the instruction (adequate course length, and sufficient time to maintain the course).

      (5) Adequate funds are available to support, operate, and maintain the course.

Other areas to review

    f. Several methods to use for collecting internal evaluations data are described below.

      (1) Visit instructional facilities. Make enough visits of sufficient length, and ensure there is a representative sampling to evaluate or check:

        (a) The quality of implemented instruction. Ensure that the visit is long enough to observe samples of representative instruction for the entire course.

        (b) Hardcopy instructional materials, such as instructor and student guides, workbooks, and reference materials for quality and availability.

        (c) Equipment, training devices, instructional media, and

training aids for condition, operation, maintenance reliability, and appropriateness.

        (d) Instructional literature for availability and quality.

      (2) Evaluate instructor performance. Instructors must show acceptable application of instructional system technology, and their activities should conform to those specified in the lesson plan. Ensure instructor records are current, and show amounts of in-service training and special training completed. Ensure:

        (a) The instructors follow the lesson plan, and teach to the standard.

        (b) The instructors use instructional media properly, detect student problems and respond to student needs, and are qualified to teach.

        (c) The instructors' records are current, and show required amount of in-service training and special training are completed.

       (d) Noted weaknesses on instructor evaluation forms have been corrected.

      (3) Measurement (Testing Program). Ensure that measurement programs are not compromised. If they are, the tests cannot provide useful data/feedback on student performance. To avoid tests being compromised, monitor the test and measurement program, to ensure quality of the test and measurement items, and student performance; and evaluate instruction in terms of student performance. Test and measurement instruments are the performance measures that determine student achievement of course objectives. A satisfactory measurement program should:

        (a) Provide students and instructors with goals.

        (b) Inform each student of their progress in meeting program objectives.

        (c) Establish a permanent record of each student's achievement, and make it available to the student.

       (d) Identify any need for a remedial program.

       (e) Identify students not meeting course standards, so appropriate action is taken.

       (f) Provide feedback data to establish a constant quality control check on the instructional system.

Note: Monitoring the measurement (test) program is a very important aspect in verifying instructional quality. But, it is just one aspect of internal evaluation. Do not exclude other aspects, such as visiting classrooms, or checking equipment adequacy and audiovisual aids. Internal evaluation analyzes all actions that occur in system operation.

JA

    g. Job Aids 350-70-4.8a through 8d provides four checklists for conducting internal evaluations on the following functional areas:

      (1) SAT Process (JA 350-70-4.8a).

      (2) Training Institutions/Facilities (JA 350-70-4.8b).

      (3) Products (JA 350-70-4.8c).

      (4) Training Development Management (JA 350-70-4.8d).

8-5.

Outputs of internal evaluations. Possible outputs of internal evaluations may include and subsequent corrective actions:

    a. Reports with identified deficiencies, and corrective actions and follow-ups on identified deficiencies.

    b. Efficient and effective individual training, training programs, and products.

    c. Needs assessment.

    d. Use of data for improvement of trainee performance and revision of learning materials.

    e. Validated course developed in accordance with (IAW) the SAT process; and validated evaluation instruments.

    f. Certified instructors, and qualified evaluators and training developers

    g. Updated Master Evaluation Plan and supporting TD Project Management Plans as required.

8-6.

Internal evaluation issues/concerns.

Possible causes of problems

a. Though an instructional system has been validated before implementation, students may still have difficulty with instruction during the day-to-day training. Managers, training developers, instructors, as well as evaluators, need to consistently review and address these possible problems. Problems may include-

    (1) Instructors that do not follow the POI or lesson plans.

    (2) The POI is different in some ways from the course being implemented.

    (3) Training resources that are required to support, operate, and/or maintain the POI differs in some respects from the resources that have been allocated.

    (4) The training resources are inadequate for the student to master specific terminal or enabling objectives.

    (5) Training materials are not correlated with the test and measurement instruments, the terminal or enabling objectives, or the instructional content identified in the task and learning analyses.

    (6) Students do not have the prerequisites required.

Managerial questions

    b. Competent management is key to an effective evaluation. Management has the overall responsibility for ensuring that all components of the evaluation are fully integrated. Internal evaluations are specifically focused to obtain answers to the questions "How good is the training?" "Are students learning?" "Has the training development process been applied?" and "Do we need to change anything?" Questions that managers seek to answer via internal evaluations are found at JA 350-70-4.8e.

Higher head-
quarter issues

    c. During internal evaluations, deficiencies may be found over which the school does not have complete control (i.e., classroom is overcrowded). Correction of deficiencies may depend on support from higher HQ. In such cases, it is imperative to bring these issues to the attention of the senior leadership, and report them to higher HQ. Maintain documentation of proponent reports to higher HQ, and their response to issues.

8-7.

Quality control criteria for internal evaluations. Training development/task proponents perform QC actions as an inherent part of the SAT process. When conducting an internal evaluation, ensure:

     a. Internal evaluation project management plans were prepared in a timely manner to impact the resource requirements.

    b. Feedback channels were established that:

       (1) Were accessible to units and school.

       (2) Collected usable, pertinent data.

       (3) Distributed evaluation results to appropriate organizations/personnel.

    c. Appropriate checklists of JAs were completed in support of the evaluation.

    d. As appropriate, training and testing events were observed, and recommended improvements discussed with proponent.

    e. Data collection was completed following the QC criteria in

chapter 4.

    f. Data was analyzed sufficiently to develop viable training/TD recommendations following the QC criteria in chapter 5.

    g. Evaluation reports were prepared and distributed to appropriate organizations following the QC criteria in chapter 6.

    h. Recommended improvements were thoroughly discussed with training/TD proponents.

    i. Follow-ups were conducted to verify that proponent took action on recommendations.

9-1.

External evaluation overview. This chapter provides "how-to" guidance for conducting external evaluations. External evaluations cover the evaluations of education/training products used in the unit, and the capability of soldiers to perform after receiving specified training.

9-2.

External evaluation description. External evaluation determines if soldiers can meet job performance requirements, need all the instruction they received, or need any additional instruction they did not receive. This process gathers data from the field to assess graduate's on-the-job performance in a job environment, and assess if the soldier can satisfy real-world job performance requirements. Evaluators must realize that the responses to the surveys are opinions of supervisors/soldiers in a specific unit configuration that may or may not relate to wartime or battlefield requirements, and may or may not be in a peacetime environment. Likewise, it is important to compare what the field says is being done with regard to a particular task, with what other documentation indicates should be done to support a particular unit mission, and/or equipment configuration, or operations capability.

External evaluation definition

    a. External evaluation is the evaluation process that provides the means to determine if the training received meets the needs of the operational Army. This evaluation ensures the system continues to effectively and cost-efficiently produce graduates who meet established job performance requirements. External evaluations are considered a quality improvement operation, ensuring soldiers and training products continue to meet established job performance requirements, as well as continually improve system quality.

Where conducted

    b. External evaluations gather data from the field to assess soldiers' on-the-job performance. A misconception often made is that external evaluations are anything conducted outside of the proponent schoolhouse. This is not true. External evaluations are conducted on soldiers and/or supervisors after the individual has graduated from a course and is performing their job/duty in the unit.

Objectives of external evaluations

     c. External evaluations assist in learning how well graduates meet job performance requirements. When conducting external evaluations, look for both strengths and weaknesses of the training system. External evaluations will help determine-

      (1) How well the graduates are meeting job performance requirements.

      (2) Whether training is being provided that is not needed.

      (3) Whether any needed training is not being provided.

      (4) Ways to improve the graduate's performance as well as the training system.

9-3.

External evaluation procedures. Table 9-1 provides the general procedural guidance for conducting an external evaluation. This process relies on input from the job environment (field) to establish how well the soldiers are performing. Data is gathered and analyzed from outside the instructional environment. The "how-to" procedures for a majority of the steps below are discussed in chapters 3 through 7.

9-4.

Unit training evaluation. In the context of this pamphlet, when unit training evaluation is discussed, the focus of TRADOC proponents is on the training (and doctrinal) materials provided to support training in units; the purpose is NOT to evaluate the unit. The assessment of unit training and proficiency is, and must remain, a unit responsibility. However, TRADOC must work in close cooperation with the unit to provide the products (MTPs, STPs) necessary to support unit evaluation efforts. Evaluation of the education/training products used in the unit is critical to determine the effectiveness of collective and individual task performance and products. The intent of a training/education evaluation within the unit is to improve training and task performance proficiency.

Factors to consider during an external evaluation

     a. A unit evaluation of education/training products (i.e., students, products, etc.) may require assessing one or several factors to determine the effectiveness and efficiency. Such factors to consider include:

       (1) Proponent's training programs and products.

       (2) CALL trends.

       (3) Feedback from units (i.e., commander).

       (4) Responses provided to unit feedback.

       (5) Feedback provided to training developers for needs analysis.

Combat Training Center interface

    b. Many proponent schools can no longer afford to send evaluation teams to units to evaluate the validity and effectiveness of training/training products. Therefore, CTCs have become a critical source of that information. The CTC program provides highly realistic and stressful joint and combined arms training, IAW Army and Joint doctrine. Combat training center rotations and reviews provide an invaluable source of collective training feedback that may impact the determination of unit missions, critical collective tasks, and collective task analysis data used in the development of collective training products. Any feedback from CTCs should trigger the proponent to revisit analysis and product revisions. Evaluators should ensure:

      (1) Training scenario missions and unit training products are tactically sound, based on approved doctrine, and developed IAW all provisions of TRADOC Reg 350-70 and the SAT process.

      (2) Interface with CTCs to receive unit task training performance feedback that identify the need or requirement to develop or revise training scenario missions IAW TRADOC Reg 350-70 and the SAT process.

      (3) Support of the TRADOC Remedial Action Program (T-RAP) (TRADOC Reg 11-13) by the conduct of detailed reviews of priority issues in an effort to determine solutions in all doctrine, organizations, training, materiel, leadership and education, personnel, and facilities areas.

       (4) Review of CALL trends (provided by the CTCs Concept of Operations collection efforts) and ensure application, as appropriate, of lessons learned to training and doctrinal products.

Checklists

    c. Job Aid 350-70-4.9a provides assistance in conducting external evaluations.

9-5.

Description

    a. The AUTOGEN is a cost effective, user-friendly, automated survey development program to give each school their own survey development, data collection, and analysis capability. The AUTOGEN is currently comprised of 2 modules, one to accomplish job analysis surveys, and the other to conduct external evaluation surveys. The conduct of an external evaluation is critical to acquiring feedback from course graduates and their supervisors at least 6 months after graduation for use in improving the quality of Army education/training. Additionally, the feedback will assist the proponent to ensure the training meets the needs of the operational force. The AUTOGEN is now available for download from the ARI AUTOGEN download site. Use password 'qualityjob' (one word, not case sensitive).

    b. The AUTOGEN provides-

      (1) The proponent the capability to quickly and effectively obtain valid individual task performance data directly from active and RC soldiers, and their supervisors, in field units for a specific job, or for an entire MOS/AOC.

      (2) A DCSOPS&T-approved template and incumbent background information that captures Frequency of Performance and Task Training Emphasis data.

      (3) The proponent the capability to quickly and effectively build an external evaluation survey to assess the effectiveness and efficiency of provided individual education/training courses.

ARI server

    c. The Army Research Institute has made available a server for administering AUTOGEN-generated job analysis and external evaluation surveys. This user-friendly site requires minimal interface with ARI to load surveys and download the survey answer file (data). It is anticipated that this capability will provide easier access to AC, as well as RC, soldiers, increasing their participation. Distribute surveys using either the ARI server, or a center/school server. However, if the ARI server is used to distribute an AUTOGEN-generated survey, the Survey Control Number is automatically generated and provided to ARI. The ARI server will automatically maintain (back up daily) the survey data and supporting files.

Recom-
mending AUTOGEN changes

     d. Forward recommended changes to the AUTOGEN program to Commander, TRADOC (ATTG-CD (QAO)), 5 Fenwick Road, Fort Monroe, VA 23651-1049 by 1 May of each FY. A panel of HQ TRADOC and ARI representatives will prioritize suggestions and fund according to requirements and available resources. Centers/schools are notified of all changes/updates to the software, which are downloaded by the school's designated AUTOGEN POCs (previously identified to HQ TRADOC). Recommendations are considered in terms of meeting the needs of all, not just a few, schools.

9-6.
Outputs of external evaluations

Outputs of external evaluations. Possible outputs of an external evaluation and subsequent actions may include:

    a. Reports with identified deficiencies and corrective actions, and follow-ups on identified deficiencies.

     b. Efficient and effective individual and collective training, training programs, and products.

     c. Needs assessment.

     d. Use of data for improvement of trainee performance and revision of learning materials.

9-7.

Possible causes of problems

    a. Possible problems that may be identified during external evaluations include:

       (1) Criterion test(s) do not measure graduate's ability to meet job performance requirements.

      (2) Terminal or enabling learning objectives do not reflect job performance requirements.

      (3) Job performance requirements were incorrectly identified during job and task analyses.

      (4) Job performance requirements changed after job and task analyses.

Managerial questions

    b. Competent management is key to an effective evaluation. Management has the overall responsibility for ensuring that all components of the evaluation are fully integrated. External evaluations are specifically focused to obtain answers to the questions "How good are our graduates and training support products" and "Do we need to change anything?" Questions that managers may seek to answer via external evaluations are found at JA 350-70-4.9b.

9-8.

Quality control criteria for external evaluations. Training development/task proponents perform QC actions as an inherent part of the SAT process. When conducting an external evaluation, ensure-

    a. External Evaluation Project Management Plans were prepared in a timely manner to impact the resource requirements.

    b. Senior leader approval/sponsorship was obtained.

    c. Feedback channels were established that:

       (1) Were accessible to units and school.

       (2) Collected usable, pertinent data.

       (3) Distributed evaluation results to appropriate organizations/personnel.

     d. Appropriate checklists and JAs were completed in support of the evaluation.

     e. Data collection instruments were prepared and data collection conducted following the QC criteria in chapter 4.

     f. Visitation plans were prepared and followed.

    g. Data were analyzed sufficiently to develop viable training/TD recommendations.

    h. Evaluation reports were prepared and distributed to appropriate senior leaders and organizations following the QC criteria in chapter 6.

    i. Recommended improvements were thoroughly discussed with training/TD proponents.

10-1.

Accreditation overview. This chapter provides an overview of accreditation as it applies to all TRADOC assigned and affiliated training institutions, and "how-to" guidance for centers and schools to conduct a self-assessment. Standards for self-assessing the following areas are discussed:

    a. Professional Military Education (PME), also known as Institutional Leader Development.

    b. Initial Military Training (IMT).

    c. Combat Training Center programs.

10-2.
Description

Accreditation description. Accreditation is the TRADOC Commander's formal recognition given to a training institution, which gives authority to conduct (or continue to conduct) education/training. It is the result of an evaluative process that certifies an institution's training program, processes, personnel, administration, operations, and logistical support (infrastructure) are adequate to support training to course standards and that training institutions are adhering to TRADOC Command Training Guidance and directives. Accreditation of all AC and RC training institutions are reevaluated every 3 years.

    a. Accreditation is a QA function that helps to assure the command that training and education provided meet the competency needs of today's Army, and the objective force. Accreditation assures:

      (1) Standardized training and training products are doctrinally correct, and set the correct standards for the Army.

      (2) Staffs, faculties, and observer/controllers are trained to standard, and provide quality instruction.

      (3) Institutional infrastructure meets required standards.

      (4) Training program provides relevant, realistic training to meet opposing force (OPFOR)/Contemporary Operational Environment (COE) requirements.

      (5) Training institutions are preparing to meet the training and education needs of the Stryker and Future Forces.

      (6) Feedback to senior leaders regarding significant training issues.

Accreditation approach

    b. Teams of both military and civilian educators conduct all accreditation visits. Do not consider these visits as inspections only; the teams also provide staff assistance. Presently, PME and IMT is accredited at proponent schools and affiliated TASS battalions, Army Training Centers, and Noncommissioned Officer Academies (NCOAs). When sufficient resources are available, the accreditation will also include functional courses, and any other training conducted by Army institutions. Organizations leading the accreditation effort are:

      (1) Commander, AAC - Recommends IMT accreditation to Commander, TRADOC.

      (2) Commander, CAC - Recommends PME and CTC Program accreditation to Commander, TRADOC.

      (3) Proponent schools will continue to accredit RC training institutions.

      (4) Currently, the U.S. Army Sergeants Major Academy will continue to accredit Phase 1 of the Advanced Noncommissioned Officer's Course and Basic Noncommissioned Officer's Course, and all of the Primary Leader Development Course. The focus of the accreditation effort is on conduct of training; training support; proponent functions; and command training guidance and directives. Focus of CTC program accreditation is on Operations Groups, OPFOR/COE, TADSS, and facilities.

Guidelines for TRADOC accreditation

    c. The Guidelines for TRADOC Accreditation establishes the policies, procedures, objectives, and responsibilities for accreditation of IMT and PME, as well as accreditation of CTC programs. The standards against which the programs will be evaluated are listed in these guidelines.

Accreditation standards guide

    d. The Standards Guide contains the TRADOC accreditation standards, with references, evaluation criteria, mandatory comments, and guidance for evaluators and training institution staffs preparing for accreditation. There are two different Accreditation Standards Guides, one for self-assessment and accreditation of IMT/PME, and the other for the accreditation of the CTC program. The IMT/PME Accreditation Standards Guide provides the basis for conducting formal accreditation evaluations. The CTC Program Accreditation Standards Guide includes separate standards guides for each of the four CTC pillars evaluated.

Evaluation of Accreditation standards

    e. HQ TRADOC Form 350-70-4-2-R-E, Record for Evaluation of Accreditation Standards, is a form for entering all of the approved standards evaluated, and for noting accreditation evaluation findings for each. The reproducible version of this form is available at the back of this pamphlet. Base remarks on the guidance in the Accreditation Standards Guide and, if appropriate, attach detailed notes pertaining to the evaluation of particular standards.

Accreditation report formats

    f. There are several accreditation reports; either received from other organizations, or reports the team is responsible for writing. These include:

      (1) A memorandum of Notification of Accreditation Status for the training institution prepared by HQ TRADOC. This is based on recommendations from the CAC and AAC QAOs. This memorandum will advise the training institution of its accreditation level (i.e., status). Job Aid 350-70-4.10a provides the sample format design for the Notification of Accreditation Status.

       (2) The center/school QAOs are responsible for conducting accreditations of the RC TASS Training Battalions. They are to provide a memorandum to their commandant that provides accreditation findings and recommendation of accreditation status. A format of this report is shown in JA 350-70-4.10b.

       (3) The commandant of the proponent school will provide a memorandum to the commander of the RC Training Battalion, awarding accreditation status. A format sample of this memorandum is provided at JA 350-70-4.10c.

10-3.

Self-assessment description. Training institutions conduct self-assessments to ensure standards established by HQ TRADOC have been met. The self-assessment process is based on the Accreditation Standards and Guide established by HQ TRADOC.

    a. Conducting a self-assessment provides the institution an opportunity to assess its situation prior to an official accreditation visit. A self-assessment will:

       (1) Demonstrate that the institution meets the accreditation standards established by HQ TRADOC QAO.

       (2) Give visibility for, and provide an objective, critical evaluation of the institution's performance (strengths, weaknesses, and challenges), and specific recommendations for improvement.

      (3) Raise higher headquarters issues (HHI) that are beyond the scope of the training institution to the appropriate level.

       (4) Identify ways for the school to sustain strengths, correct weaknesses, and improve training.

       (5) Analyze the resources and effectiveness of the institution in fulfilling its mission.

       (6) Serve as a baseline for measuring progress in the coming decade, and provides a sound basis for institutional planning and improvement.

       (7) Demonstrate that performance, competence, and achievements of students who complete programs are commensurate with the certificates, diplomas, and degrees awarded by the institution.

Self-
assessment procedures

     b. Prior to an accreditation visit, the education/training institution will complete a self-assessment based on the Accreditation Standards Guide. Job Aid 350-70-4.10d provides the detailed guidance for preparing and conducting a self-assessment, and preparing a self-assessment report. A self-assessment is conducted on every course within a 3-year cycle.

Self-assessment cover letter and report

     c. A self-assessment report is prepared and provided to the Accreditation Team 60 days prior to the accreditation. The report identifies the results of the self-assessment. Included in the report are the training institution's strengths and weaknesses, areas of deficiency, planned initiatives, and HHI. It also transmits documents needed by the accreditation team for review. The format design of the self-assessment cover letter and report is provided at JA 350-70-4.10e.

10-4.

Quality control criteria for accreditation. Training development/task proponents and training institutions perform quality control actions as an inherent part of the SAT process. When conducting a self-assessment and reporting the results, ensure-

     a. Findings of previous self-assessments are considered.

     b. Personnel at all levels of the institution are involved in conducting the self-assessment.

     c. The self-assessment report does identify strengths, weaknesses, and challenges.

    d. The self-assessment report includes recommendations for improvement.

    e. The self-assessment report includes all the information and documents described by JA 350-70-4.10e.

    f. Higher headquarters issues are reported to the appropriate headquarters.