Detailed Information on the
Department of Defense Training and Education Programs - Basic Skills and Advanced Training Assessment

Assessed the effectiveness of the programs and realigned resources as necessary to ensure the best possible balance and return on investment.

Reviewed performance measures used to manage the program and revised them as necessary to develop the most useful and accurate indicators.

Continuing to assess the effectiveness of the programs and realign resources as necessary to ensure the best possible balance and return on investment.

Continuing review of performance measures used to manage the program and revising them as necessary to develop the most useful and accurate indicators.

Measure: Cost per student year equivalent at Service professional development education activities. (Average of all services.)

Explanation:Professional development education activities are identified in Budget BA-3 documents. Activities include Army War College, Naval War College, Air War College, School of Advanced Airpower Studies, Naval Justice School, senior enlisted academies, and others. Cost from the Professional Development Education appropriation code.

Measure: Personnel Readiness of Military Forces. DOD measures the readiness of units to perform their missions. DOD's long term goal is to have sufficient numbers of highly trained and ready units. Many of DOD's missions are long-term in nature and demand high readiness for these units. This readiness or "C-rating" metric is based on a commander's assessment of his/her unit's ability to perform its mission. The metric measures whether the unit has sufficient number of well trained personnel as well as capable equipment and other activity indicators. The unit-level "C-ratings" are aggregated to determine the readiness of larger units to perform critical military missions (e.g. executing war plans). DOD military and civilian leaders and Congress use these ratings to determine whether military readiness to perform these critical missions is adequate. In the last several years, this measure has consistently reported that units are at high readiness."

Explanation:C-level indicates the capability of personnel to perform military missions. Unit commanders assign C-level ratings based on the commander's judgment whether he/she has sufficient amounts of trained and ready troops. The following are C-level definitions: C-1 Minor deficiencies with negligible impact C-2 Some deficiencies with limited impact C-3 Significant deficiencies preventing some training mission C-4 Major deficiencies that preclude satisfactory institutional training mission accomplishment. The unit-level "C-ratings" are aggregated to determine the readiness of larger units to perform critical military missions (e.g. executing war plans). Tha "actual" C-rating is classified and shown here as "n/a".

Measure: Commander's assessment of institutional training readiness for Specialized Skill Training - - reported as C-level, which includes quantitative and qualitative measures. (Average of all Services; Army is Active Component only.)

Explanation:C-level indicates the capability of U.S. military training institutions to perform their mission. Activity commanders assign C-level ratings based on TW-level ratings and the commander's judgment (which includes an assessment of Institutional Training Mission Support - manpower, equipment and facilities). The following are C-level definitions:

Is the program purpose clear?

Explanation: The purpose of basic skill and advanced training is stated in clear and concise terms in DoD Directive 1322.18, "Military Training" (September 2004): 4.1 Training programs that effectively support force readiness shall be provided for the Total Force. 4.2. All personnel and components shall train on their Mission Essential Tasks (METs) to established standards??. 4.3. Members of the Department of Defense shall receive, to the maximum extent possible, timely and effective individual, collective, unit, and staff training necessary to perform to standard during operations. The Basic Skill and Advanced Training program includes the following : ?? Specialized skill training, which provides military members with initial job qualification skills or new or higher levels of skill in their current military specialty or functional area. The focus is on "occupational" training that relates to skills that individual military members require to perform wartime missions. (Definition from Institutional Training Readiness Report (ITRR).) ?? Flight Training that provides the individual flying skills needed by pilots, navigators and naval flight officers. These are undergraduate flight training programs culminate in an officer being categorized as a "designated" or "rated" officer. (Definition from ITRR.) ?? Professional military education prepares military officers for leadership through training. The training provides breadth of view, diverse perspectives and critical analysis, abstract reasoning, comfort with ambiguity and uncertainty and innovative thinking, particularly with respect to complex, non-linear problems, such as uncertainty during battle. (From CJCSI (Chairman, Joint Chiefs of Staff Instruction) 1800.01B.) This category includes the Defense Acquisition University (DAU) at Ft. Belvoir, VA; National Defense University (NDU) at Ft. McNair, DC; a war college or equivalent for all four Services; and various other activities. ?? Training support, which is Service-wide support costs of the training establishment. Examples include equipment maintenance, control and supervision over ranges and training facilities, travel and per diem costs for students, and other costs. Elements of the program are defined at the service level. An example at the Service level is found in the Air Force Strategy for On The Job Training: "(Conduct) training programs providing realistic and flexible training producing a highly skilled, motivated force capable of carrying out all tasks and functions in support of the AF [Air Force] mission. These programs should provide the foundation for AF readiness." The individual activities within Basic Skill and Advanced Training also have governing instructions, charters, and similar documents that state their purposes in greater detail.

Evidence: DoDD 1322.18 "Military Training" (September 2004). Institutional Training Readiness Report (ITRR), January 2005, page E-11. CJCS Instruction 1800.01B (30 Aug 2004) [section 4, Chairman's Vision]: ?? ??The Armed Forces must also examine doctrine, organizations, training systems, material procurement, leadership preparation, personnel programs and facilities to ensure military superiority. ?? The men and women of our Armed Forces are the nation's most important strategic resource. Only a force of dedicated, highly educated, and well-trained men and women capable of leveraging new ideas will succeed in the complex and fast-paced environment of future military operations. Mission statements are published in strategic plans, regulations, and other documents. For example: ?? Army Regulation (AR) 350-1 (9 Apr 2003) outlines Army Training and Education requirements; AR 350-17 (31 May 1991) outlines the non-commissioned officer development program; AR 621 series of regulations address education (various dates). ?? Air Force Doctrine Document (AFDD) 1-1 (18 Feb 2004), Air Force Instruction 36-2201 (4 Feb 2005) [pg. 5], and other statements guide the development of leadership and other capabilities of Airmen through a deliberate process which includes education and training. AFDD 1-1 emphasizes that education and training are critical components of force development. ?? Navy's OPNAV Instruction 1500.56A (16 Nov 1994) implements DoD military training policy.

Does the program address a specific and existing problem, interest, or need?

Explanation: Military personnel require a range of specialized training. They need technical training to operate equipment unique to the military; process training to perform complex tasks; and professional education to prepare them for the challenges of higher ranks. The training requirements and the number of graduates needed are determined by the military services, the customers of the program. The military services derive their need for trained personnel from their mandate under Title 10 of the U.S. Code to organize, train, and equip the nation's armed forces. Multiple sections in United States Code require the Department of Defense to establish specific training programs and/or schools to prepare servicemembers for various military occupational specialties (MOS). Furthermore, certain sections in law also establish minimum training and/or education guidelines for specific MOSs. The program is built through a collaborative process with customers. Specific training requirements are based on force structure, doctrine, missions, and other factors. Some needs are identified in the Universal Joint Task List, a statement of required capabilities that is evolving into a standard reference. The program has the capability to adjust as needs change. Defense Acquisition University (DAU), which is established in law, trains the acquisition workforce; personnel served by the program are required by law to be certified and trained. DAU and its components train the acquisition, technology, and logistics (AT&L) workforce - a community with diverse knowledge requirements that evolve rapidly with technology. Among the career fields are: Contracting, Purchasing, Facilities Engineering, Life Cycle Logistics, and Information Technology. Offerings include required certifications, assignment-specific courses, and other training. National Defense University (NDU) addresses national security through teaching, research, professional exchange, and outreach. NDU's structure and components are outlined in law. NDU evolved in the 1970s and 1980s to encompass separate colleges under a single organization. It is the preeminent entity for educating "military and civilian leaders through teaching, research, and outreach in national security strategy, national military strategy, and national resource strategy; joint and multinational operations; information strategies, operations, and resource management; acquisition; and regional defense and security studies." (Quote from NDU Mission statement.)

Evidence: Flight Training -- (A) Section 2003 of Title 10 states, "To be eligible to receive an aeronautical rating as a pilot in the Army or Air Force or be designated as a naval aviator, a member of an armed force must successfully complete an undergraduate pilot course of instruction prescribed or approved by the Secretary of his military department." (B) Section 9303 of Title 10 states, "The Secretary of the Air Force shall establish and maintain?? (1) one or more schools for the training and instruction of aviation cadets; and (2) courses of instruction for aviation students at one or more established flying schools." Medical Training -- Section 2112 of Title 10 states, "There is hereby authorized to be established within 25 miles of the District of Columbia a Uniformed Services University of the Health Sciences (hereinafter in this chapter referred to as the "University"), at a site or sites to be selected by the Secretary of Defense, with authority to grant appropriate advanced degrees. It shall be so organized as to graduate not less than 100 medical students annually." Examples that show there are existing needs for Basic Skill and Advanced Training include: ?? Army Soldier Manning Decision Review (SMDR) and Training Resource Arbitration Panel (TRAP). In accordance with AR 350-10, they adjust training requirements of the Army based on operational and manning needs. ?? Navy Fleet Inventory Manning Requirements determine input plans for "A" and "C" schools based on the Skilled Personnel Project for Enlisted Retention (SKIPPER) and other models. Defense Acquisition University (DAU) Overview Briefing (October 2004), from www.dau.mil, "About DAU." National Defense University (NDU) -- (A) Section 2165 of Title 10 states, "There is a National Defense University in the Department of Defense." (B) CJCS Instruction 1800.01B [pp. A-1, A-A-5, B-1]. (C) Mission Statement from www.ndu.edu. Naval Postgraduate School -- Title 10, Section 7041 states, "There is a United States Naval Postgraduate School for the advanced instruction and technical education of commissioned officers of the naval service in their practical and theoretical duties."

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Overall, the courses and activities in military Basic Skill Training and Advanced Training are focused on needs unique to U.S. armed forces. The training and education provided are not available in other programs. Within the program, each Service conducts skill training as required by the unique specialties or equipment of the Service, but they work together to the extent possible so there are not redundant activities within the program. The Interservice Training Review Organization is a formal effort to remove redundancies through joint or co-located training whenever possible. One result of efforts to remove redundancies can be found in the Marine Corps: approximately 60% of courses attended by Marines are accomplished at non-USMC schools. Another example of how the integrating programs to reduce redundancy can be found in Air Force and Navy integration of flight training. Going through the four phases of training: ?? The Navy conducts initial flight training for the Navy and Marine Corps. This leads to increased warfighting effectiveness and training efficiency, per Naval Transformation Roadmap 2003. ?? Air Force Navigators and Student Naval Flight Officers (SNFOs) complete primary training under a combine syllabus that is taught in Navy squadrons at Naval Air Station Pensacola. ?? Navy SNFOs and Air Force Navigators selected for large, multi-engine aircraft complete advanced training with the Air Force. Navy SNFOs and Air Force Navigators selected for tactical jets continue onto Navy intermediate training. ?? Final training is performed by the respective Service because they operate different types of aircraft in different environments Program directors also search for ways to remove redundancies and leverage other efforts as long as program requirements are met. For example, Navy computer network operators attend Cisco Academy for training. The Navy also leases small arms marksmanship trainers for firearms training. Defense Acquisition University (DAU) and National defense University (NDU), the senior military colleges in the program, have unique missions with curricula specified by the Joint Chiefs of Staff.

Evidence: ITRO co-location review is described in the ITRO Procedures Manual of 1 April 2004, pp. 2-6, 3-3, and others. DAU and NDU are established by specific regulations, yet still work to reduce redundancies. For example, NDU has cooperative arrangements with other institutions whereby the appropriate portion of required education is provided by the other institution -- leveraging the core competencies of each to produce the desired outcome.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program emphasizes continuous improvement. It is regularly evaluated by a variety of agencies and incorporates lessons from real-world experience. A detailed evaluation of the effectiveness of DOD's training institutions is DoD's Institutional Training Readiness Report (ITRR), an unclassified annex to the Quarterly Readiness Report to the Congress. The ITRR reflects favorably on Basic Skill and Advanced Training activities. The ITRR breaks Specialized Skill training into Initial Skill Training, Skill Progression Training and Functional Training. The Army's effort to remove flaws is illustrative. Several Army agencies, Army Staff, and Major Commands oversee the conduct of training. In addition, a comprehensive quality assurance program is administered by the Training and Doctrine Command with the help of Army Agencies and Major Commands. The Marine Corps training places emphasis on critical skills and pertinent recurrent training to optimize the members' quality of support for the unit's Mission Essential Task List. Reviews of training standards and associated courses are conducted tri-annually, and ad hoc reviews are conducted as required to ensure compliance with any changes in governing policy, doctrine and/or requirements. NDU is evaluated by the Process for Accreditation of Joint Education (PAJE) and various accrediting bodies (e.g.; Middle States Association of Colleges and Schools) to ensure program quality.

Evidence: Assessment of institutional training readiness that incorporates quantitative and qualitative measures. The Commander will assign C-level ratings based upon training workload (TW) level ratings and the commander's judgment (which will include an assessment of Institutional Training Mission Support - manpower, equipment and facilities). The following are C-level definitions. C-1: Only minor deficiencies with negligible impact on capability to perform required institutional training mission. C-2: Some deficiencies with limited impact on capabilities to perform required institutional training mission. C-3: Significant deficiencies that prevent institution(s) from performing some training missions. C-4: Major deficiencies that preclude satisfactory institutional training mission accomplishment. On target ratings: "If any training category is rated C-3 or below, a narrative description of the problem, its causes, and plans to correct the problem will be provided. Commander's comments are also required when the C-level rating for a category of training is higher or lower than the corresponding TW-level rating." The de facto target is C-2 or C-1. (Note: These "C" ratings are UNCLASSIFIED) The most recent ITRR reported the following C-ratings: ?? Army (Active Component) Initial Skill Training: C-2 ?? Navy Initial Skill Training: C-1 ?? Army (Active Component) Skill Progression Training: C-2 ?? Navy Skill Progression Training: C-1 ?? Army (Active Component) Functional Training: C-2 ?? Navy Functional Training: C-1 ?? Army (Active Component) Flight Training: C-1 ?? Navy Flight Training: C-1 ?? Air Force Initial Skill Training: C-1 ?? Marine Corps Initial Skill Training: C-2 ?? Air Force Skill Progression Training: C-1 ?? Marine Corps Skill Progression Training: C-2 ?? Air Force Functional Training: C-1 ?? Marine Corps Functional Training: C-2 ?? Air Force Flight Training: C-1 The results of effectiveness evaluations can be found in published reports such as PAJE and accreditation reports for the Defense Universities, the ITRR, and Service-specific reporting for other activities. PAJE Standards for accreditation are in CJCSI 1800.01B, Enclosure F. Accreditations are conducted for each military college falling under its jurisdiction. Standards for NDU's accreditation by Middle States Commission on Higher Education may be found at: www.msche.org/ publications/ Characteristicsbook050215112128.pdf . (Excerpt included as evidence.) NDU's accreditation was reaffirmed in March 2002 in a "Statement of Accreditation Status." A periodic review report is due June 1, 2007. NDU's IRM College was renamed a Center of Academic Excellence in IA Education in 2003. Standards may be found at: www.nsa.gov/ia/academia/ caeCriteria.cfm?MenuID=10.1.1.2. Renewal is due in 2006.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: As noted above, the military services are required to organize, train and equip military forces. As "customers" for this program, the military services need adequate numbers of trained and ready troops for their units. After thier initial training, troops receive additional training from this program. Ultimately, all of the troops that graduate from this rigorous training join military units. DoD allocates resources to provide trained recruits to the military units. Training resources include instructor staff, facilities, equipment and funding. The number of sufficiently trained graduates needed from this program drives funding requirements. Officials can adjust funding if the needs for trained graduates by units change. Detailed budget documents ensure that resources support each program's purpose. DOD and OMB review the needs and performance as part of the their work building the President's Budget funding request for this program. The military services provide data justifying that resources are adequately targeted. The DoD Comptroller also conducts a mid-year review of all programs. During this review, the effectiveness of these programs are examined, and the Department can adjust funding levels to ensure that adequate resources support the program. The Institutional Training Readiness Report (ITRR) provides an annual recap of how effectively the resources supported the program purpose. It uses a Training Workload measure (TW), that shows the percent of planned training actually conducted. This metric indicates whether DoD used resources for training as intended. The military services use complex models to conduct detailed planning and tie resources to planned outputs. An example is the Army Institutional Training Resource Model (ITRM), which provides a readiness justification for funding requirements. ITRM documents the student load needed to support the force structure and the resources needed to provide the requisite training

Evidence: DoD budget exhibits for Army and Air Force Officer Acquisition demonstrate how the funding for accession training is programmed, budgeted, and obligated in a targeted manner by each service. For example, the Army exhibit for Officer Acquisition--which includes the US Military Academy and Officer Candidate School (OCS)--shows the amount actually spent in FY2004, requested in the Budget in FY2005, and planned for FY2006 and 07. Detailed descriptions of the funded operations are provided. As for the budgeting process, Service programming guidance memoranda serve to prioritize funding requirements. At DAU, for example, student throughput has gone up 176%, student travel costs down 45%, annual training cost per student decreased 271%, and faculty staffing down 10% since FY99 with a relatively flat budget. An example of how Services determine required resources is the Army Institutional Training and Resource Model (ITRM), which identifies funding requirements for institutional training. Reports such as NDU's Annual report to Chairman JCS provide details.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Basic Skills and Advanced Training program uses long-term performance measures at all levels to support the purposes of the program. DOD has developed performance measures six years into the future for training and other activities. These performance measure plans are tied closely to funding requirements. The measures and funding plans are revised annually. The following three long-term performance measures cover the outcomes of most DOD activities in the basic skill and advanced training program: Measure 1 - Completion Rate for Flight Training. Flight training is a key enabler of American military power and is the largest budgetary component of the basic skill and advanced training program. It measures the completion rate for students by comparing the actual number of students that graduate from training to the planned number. Students that successfully graduate from this rigorous training program are of the highest caliber and are considered ready to perform missions and join their units in the field. The standards against which these troops are trained are rigorous and have been developed by military services based on lessons learned from combat and operational experience. The long-term goal takes into account the needs of field units for adequate numbers of trained and ready graduates. It also accounts for normal student attrition. Measure 2 - Percentage of Enrollees Completing Special Skills Training. It indicates how well the basic skill and advanced training program is producing trained and ready graduates with technical and mid-grade enlisted leaders. It measures the completion rate for students by comparing the actual number of students that graduate from training to the planned number. Students that successfully graduate from training have met the high standards in their fields and are considered ready to perform missions and join their units in the field. The standards against which these troops are trained are rigorous and have been developed by military services based on lessons learned from combat and operational experience. The long term goal takes into account the needs of field units for adequate numbers of trained and ready graduates. It also accounts for normal student attrition. Measure 3 - Personnel Readiness of Military Forces. DoD measures the readiness of units to perform their missions. DoD's long term goal is to have sufficient numbers of highly trained and ready units. Many of DoD's mission are long-term in nature and demand high readiness for these units. This readiness or "C-rating" metric is based on a commander's assessment of his/her unit's ability to perform its mission. The metric measures whether the unit has sufficient number of highly-trained personnel as well as capable equipment and other activity indicators. DoD military and civilian leaders and Congress use these ratings to determine whether military readiness to perform these critical missions is adequate.

Evidence: Service Budget Activity Three (BA-3) OP-5 exhibits for Specialized Skill Training show input, output, and workload through FY 2007. All Services submit Budget Activities as part of their Congressional Budget Justification Books; Army and Air Force are provided as examples. C-level indicates the capability to perform required institutional training mission. Activity commanders assign C-level ratings based on TW-level ratings and the commander's judgment (which includes an assessment of Institutional Training Mission Support - manpower, equipment and facilities). The following are C-level definitions: C-1 Minor deficiencies with negligible impact C-2 Some deficiencies with limited impact C-3 Significant deficiencies preventing some training mission C-4 Major deficiencies that preclude satisfactory institutional training mission accomplishment. The unit-level "C-ratings" are aggregated to determine the readiness of larger units to perform critical military missions (e.g. executing war plans). Army Institutional Training Program Assessment Form prepared as part of the POM for "People, Leader Development, and Institutional Training." The NDU Strategic Action Plan (2000), NDU Report of Self-Study (Oct 2001) and NDU Strategic Framework (working document) also contain specific measures that pertain to its mission and performance. Elements of the program at all levels have long-term performance measures. Some examples include: ?? An Army Program Assessment Form prepared as part of the POM process describes "People, Leader Development, and Institutional Training." The metric is the number of students graduated per number of students programmed in the Army Program for Individual Training (ARPRINT). ?? Air Force performance measures are found in diverse documents such as: Projected Manning Categories for Career Fields/AFSCs; MAJCOM SORTS/Mission Capability Rates Reports (Classified); AETC Decision Support System (ADSS); Field Evaluation Questionnaires (FEQs); and others. ?? The National Defense University (NDU) Strategic Plan states strategic goals in addition to mission and vision values. NDU built its most recent accreditation self-study around these goals and found that they adequately allowed the university to achieve its purpose, but they could be expanded to include new organizational elements of NDU. The accreditation reviewers found that NDU is more focused on mission and goals than most other educational institutions. Still, NDU determined that most of its performance measures were more project management-oriented rather than strategic. FY05 finds NDU in the process of reexamining its goals and measures and revising them into a new strategic plan with annual and long-range measures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Long - term targets discussed above are established by senior Service leaders based on mission needs. In many cases military missions are recurring in nature and are tasked to units for extended periods (e.g., Navy deployments in support of overseas presence missions). Units tasked with performing military missions need a continuous stream of adequate numbers of trained personnel equipped with certain skills to replenish ranks or enhance unit capabilities. The personnel and training requirements to staff these units are integrated by the military services and DOD into long term goals and budget plans. For Measures 1 and 2, the target of 100% reflects the need for program to produce all of trained and ready graduates that have been planned. If fewer graduates are produced than planned this could have an adverse impact on military readiness to perform missions. The goals are carefully calibrated to balance unit needs with training program capabilities. Other examples: Defense Acquisition University has long-term enabling strategies to achieve its goals with annual measures in place and tracked; will complete development of long-term outcome measures by September 2005. National Defense University (NDU) focuses on continuous quality in order to meet periodic accreditation assessments. NDU is developing a strategic plan with more long-term goals with measures and targets.

Evidence: Targets and timeframes for the measures "Completion Rate for Flight Training" are shown in the Institutional Training Readiness Report (ITRR), dated January 2005. Budget documents show programmed numbers and actual completion numbers for Percentage of enrollees completing specialized skills training. In reference to the other examples discussed in the explanation section, targets and timeframes are normally published in a variety of formats and reports. For example: ?? The Army Program Assessment Form for "People, Leader Development, Institutional Training" (part of the POM process). ?? Some of DAU's targets /goals are stated in its strategic plan. ?? NDU Strategic Framework (working document) and NDU Academic Assessment Elements (Version: March 2004) address long-term targets/goals and timeframes as applicable.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: DoD's annual performance measures demonstrate progress in achieving the program's long-term goals. The annual measures gauge ability of all of DoD's training institutions to meet the specialized skill training requirements identified by the military services. The measures also show that these DoD training institutions can produce adequate numbers of graduates. In addition, to the three measures described in question 2.1, the following two short-term annual measures show this annual progress. Measure 4 - Assessment of Institutional Training Readiness. This measure assesses the readiness of staffing, equipment, and facilities at DOD training institutions to support the education of service members in specialized skills. These ratings are denoted as "C-ratings." The "C-ratings" show whether institutions are in varying states of readiness C-1 (very ready) or go down to C-4 (not ready) to provide training. Measure 5 - Graduates of Specialized Skill Training. The number of graduates required from training programs. This number is adjusted annually. Other examples of annual performance measures that support long-term goals are as follows: ?? The USAF Air University (AU) Strategic Plan 2005-2010 ties very specific measures to each of its strategic goals and objectives. ?? NDU has annual performance measures that enable them to work toward the university goals -- particularly those related to the teaching mission. These measures include student output and annual curriculum reviews, etc. The university is revising its plan to ensure consistent linkage to long-term measures.

Evidence: Institutional Training Readiness Report (ITRR January, 2005) showing C-level, which is commanders' assessments of readiness: Tables 1-1, 2-1, 3-1, and 4-1. Budget documents show "Number of Graduates from Specialized Skill Training Activities": the number of programmed graduates (objective) against the actual number of graduates each year. USAF Air University (AU) Strategic Plan 2005-2010.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Annual targets for Basic Skills and Advanced Training are established by senior Service leaders based on mission needs. Military forces must remain rigourously trained for military missions that are recurring in nature over extended periods (e.g., Navy deployments in support of overseas presence missions). In order to create ready forces, the Services' training institutions must maintain high standards for their own capability to train troops. Services set their baselines and targets to ensure the highest level of military readiness. The military's high performance in its operations world-wide indicate that the baselines and targets for training readiness are high and are being met. The annual output measure regarding commanders' assessment of institutional training readiness (Measure 4) has a target of C-2 or better. Below C-2, commanders must submit an explanation of the causes of problem(s) and their plans to improve the situation. So a rating of C-2 or C-1 indicates the activity can continue to support the Basic Skill and Advanced Training program. Measure 5 is an output measure which has graduate production as its enduring annual target. This measure reflects a requirement of the program to support the force structure in-place with adequate numbers of highly-trained personnel who have met the high standards for their training programs. Within the program, Service staffs and directors also establish baselines and targets for activities. These are based on assessment of demands and collaboration with customers. Finally, activities such as Defense Acquisition University (DAU) identify performance baselines and targets at their level. DAU has a 2005 Performance Plan that identifies specific tasks and targets.

Evidence: One annual performance measure is "Number of Graduates from Specialized Skill Training Activities, and the other annual measure is the readiness rating of institutional training. The targets for each of these measures are located in the Performance Measures Tab of this report. ITRR shows next-year targets throughout. The Army process and responsibilities are described in AR 350-10, Chapter 2. An example of how the process works can be found in the Army - as explained in Army Regulation 350-10, Management of Army Individual Training Requirements and Resources. The Army Staff establishes baselines and targets based on guidance from the Secretary of the Army and Army Chief of Staff. This guidance is published to the Army Staff as the Army Plan, which consists of: Army Strategic Planning Guidance, Army Planning Priorities Guidance, Army Program Guidance, and Army Campaign Plan. Army Staff Director of Training (within G-3/5/7) and Director of Military Personnel Management then provides refined guidance in a memorandum to their staffs and others involved with managing individual training requirements and resources. The ARPRINT mission database is the baseline (accessed through ATRRS (EXPLAIN IN ENGLISH)). It is developed through a collaborative process between the staffs of the Army Staff Director of Training (?), DCS G-1 (EXPLAIN IN ENGLISH), Reserve Components, and Major Commands and other Army activities that perform institutional training. The Director of Training sets targets (measures of success) based on historical data vetted through the Army Executive Council as an Army Performance Management Measure for the Annual Defense Report Performance Measure. DAU FY05 Annual Performance Plan shows that targets are set for completion within the fiscal year.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The main program partners are the recruiting organizations within each military service. These recruiters provide the new recruits that the accessions training program turns into troops with rudimentary military skills. Without adequate numbers of recruits, the accessions training program cannot meet its long term or short term goals. NDU maintains partnerships with a number of universities as a part of its education and outreach missions. For example, when information assurance (IA) became critical to DoD's national security strategy, the University's Information Resources Management College (IRMC) made this a thrust of its curriculum. The college applied for and was certified as a NSA/DHS National Center of Academic Excellence in IA Education (CAE). Since that time, IRMC has developed a number of partnerships with other CAEs to allow for cooperative degree programs. These partnerships are documented in Memoranda of Understanding (MOU) that specify common goals. Additionally, both parties commit to maintaining their CAE status. NDU IRMC reps meet with their educational partners several times a year to discuss IA issues, evaluate program success, and examine means to further promote IA education. An example MOU is presented, as well as the NDU Strategic Framework citing collaboration and partnerships as one of the institution's values. A final and important category of program partners is contractors. Contractors work toward annual/long-term goals based on requirements as stipulated in letters of agreements, statements of work and contracts. Contractors are often used to perform work that helps meet program goals but the necessary resources are not available within DoD. Generally, contracts specify, through statements of work, the deliverables and services that contractors must provide and the expectations for quality of work that must be met to fulfill contractual requirements. Periodic performance reviews are routinely scheduled to ensure contractors are meeting pre-defined timelines and milestones for productivity. Letters of agreement are well-defined, and expectations are specified and mutually agreed upon by all parties concerned.

Evidence: Institutional Training Readiness Report (ITRR), page E-10, "Reporting Organizations," and Tables 3 and 4. Annual Defense Report to the President and the Congress, 2004, pages 10-11. An example Memorandum of Understanding (MOU) is presented. This one is between the University of Texas at San Antonio and the National Defense University, Information Resources Management College. The schools agreed to a cooperative mission by providing extended academic opportunities. The MOU outlines four goals in this collaborative arrangement. This MOU is good for three years. National Defense University Strategic Framework (excerpt).

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Due to the importance of this mission, activities are evaluated by internal and external organizations. Assessments include a progress check on all performance tasks. As an example, the Air Force provided this partial list of evaluations for Basic Skill and Advanced Training activities: GAO; AF Audit Agency; Staff Assistance Visit (SAV); Training Standardization and Evaluation Visit (TSEV); Inspector General; Local Evaluation Programs; AFOMS; Studies and Analysis Squadron. Various Air Force documents are provided as evidence - typical of all Services. One of the most powerful independent evaluations is the Inspector General (IG) inspection or visit. All Services have IG branches; the Army Regulation is presented here as typical. IG activities are well-planned and thorough, as evidenced by the detailed descriptions in the regulation. major commands (such as Army Training and Doctrine Command) submit semi-annual reports up the chain of command "summarizing the significant activities of auditors, investigators, inspectors, and their efforts to curb fraud, waste, and mismanagement." Basic Skill and Advanced Training is also evaluated by the GAO. An example is "MILITARY PERSONNEL: Actions Needed to Better Define Pilot Requirements and Promote Retention" (GAO/NSIAD-99-211). In addition, independent analyses from contracted companies such as RAND evaluate their mission performance. Internal processes (instructional systems design and course content review boards) stress continual review. The Services also solicit evaluations from members who have attended comparable training conducted outside the Service. More specific examples: ?? The Navy undertook the Executive Review of Navy Training (ERNT), in winter 2000, to examine Navy training and make recommendations to improve it. As a result, the Navy has established the Naval Personnel Development Command (NPDC) as the single entity responsible for individual skill training (outside flight training). ?? DAU uses the comprehensive Enterprise Performance Review and Analysis (EPRA) program to evaluate and inform leadership. ?? NDU is evaluated by the Process for Accreditation of Joint Education (PAJE) and Middle States accreditation processes -- fully independent and, in the case of Middle States, identical to the processes at civilian universities. Finally, Board of Visitors examination provides an independent view of university from outside of the academic community.

Evidence: Air Force Northwest region Operating Instruction 36-3, Region Staff Assistance and Commander's Visit. Army Regulation 20-1, Inspector General Activities and Procedures. GAO report GAO/NSIAD-99-211 excerpts. The Navy undertook the Executive Review of Navy Training (ERNT), in winter 2000, to examine Navy training and make recommendations to improve it. As a result, the Navy has established the Naval Personnel Development Command (NPDC) as the single entity responsible for individual skill training (outside flight training). Chairman of the Joint Chiefs of Staff Process for the Accreditation of Joint Education (PAJE) reports for three activities within Basic Skill and Advanced Training: industrial College of the Armed Forces, National War College, and Joint Forces Staff College.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The DoD Planning, Programming, Budget and Execution Process (PPBE) requires budgeting based on resources needed to achieve both annual and long-term goals. OP-5 exhibits submitted by each Service provide detailed information on long-term performance goals at the activity group / sub-activity group (AGSAG) level. These exhibits detail the program's indirect and direct costs. OP-5s provide a level of information that would qualify as complete and transparent. Section I of the OP-5 describes the activity, such as Specialized Skill Training. Section IV provides performance criteria in terms used by the Services and Congress to justify the program. Army and Marine Corps OP-5s are provided as examples. The Program Objective Memorandum (POM) clearly outlines resource requirements. Budget models are determined using historical program and financial data in combination with known or perceived changes to program elements (i.e. number of users, location of user populations, and future directives). Both annual and out year POM requests are tied to information provided by the forecasting models and identified needs of the programs. A complete accounting of the needs is available. Budget request documents for have charts available that indicate item by item the results of the forecasting models and requirements provided by the owners of the program. Such documents include the OP-5 and OP-14 in the FY06 President's Budget and include results of program expansion, enrollment, etc. The Monitoring the Status of the Force (MSOF) briefing, presented quarterly to Under Secretary of Defense (Personnel and Readiness) includes a measure of Budget and Performance Integration that tracks at a high level how well planning, evaluation, and budget staff work with program managers to create an integrated plan/budget, and monitor its implementation. Further, DoD Supplementary FY2007-FY2011 Program and Budget Submission Guidance states that the Deputy Secretary of Defense has directed each Component (Service) to use performance-based measures for its budget formulation. This should significantly improve management. Activities also use strategic plans and the strategic performance goals therein to justify budget requests. National Defense University (NDU) compiles budget requests from its components. They are largely based on numbers of students directed to NDU by the Services and OSD, but also include outreach and research program requirements. It should be noted that budgets provided through NDU funding do not cover all program costs. Rather, NDU funding is supplemented through funding obtained from other DoD and non-DoD sources. Obtaining this funding is not transparent to the university. It requires a large amount of coordination, justification, etc.

Evidence: Army and Marine Corps OP-5 Exhibits for Specialized Skill Training. All Services submit OP-5s for their budgets. An example OP-5 is Army Specialized Skill Training, which shows detailed information about changes in the program and their connection to its budget. The goals include personnel and workload, but go beyond these to show relevant portions of the AGSAG; in this case curriculum development. The OP-5 for Professional Development Education includes initiatives like distance learning. Monitoring the Status of the Force briefing slide showing an assessment of Budget and Performance Integration.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Elements and activities within DoD Training and Education are making progress on strategic planning that include plans of action and milestones to correct areas of concern. Information technology enhancements will facilitate improvements in analysis and planning - areas that were previously inefficient and slowed productivity. Examples of improved strategic planning include: ?? Navy recently established the Naval Personnel Development Command (NPDC) as the single entity responsible for individual skill training (outside flight training). NPDC aims to improve "mission accomplishment by providing Sailors with the tools and opportunities to learn, grow, lead and excel," with a host of initiatives to accomplish this. One of the most significant is the 5 Vector Model, which will provide continuity to a career-long learning environment for personal and professional development. ?? Air Force is working on improving information collection and handling as a means of improving strategic planning. An example is the Student Registration and Recording System. ?? The annual planning process at Defense Acquisition University (DAU) includes a formal SWOT analysis (strengths, weaknesses, opportunities, threats) of its training program. ?? National Defense University (NDU) is revising its strategic plan to better comply with regulations such as the Government Performance and Results Act (GPRA).

Evidence: Secretary of Defense Annual Report to the President and the Congress, page 9. The NPDC "Commander's Intent" statement describes the command's origins and intent. It clearly shows the Navy's commitment to improving training starting at the strategic level, by integrating training as a crucial part of helping sailors achieve their career goals. It also takes advance of the latest technologies to allow sailors to learn at any location, so deployments will not delay their career advancement. From www.npdc.navy.mil. Air Force Communications Agency journal "Intercom," Aug 2004 article, "AETC: Commitment to Excellence" from public.afca.af.mil/Intercom/ DAU Baldrige Opportunities for Improvement includes specific changes to the planning process to improve predictions and strategies. One of 43 opportunities (DAU 2004 Remedial Plan "Opportunity for Improvement, #21O2") is described below:  Description: While DAU identifies key strategic objectives in the form of goals for the six-year planning horizon of 2004-2009 and these are used to set one-year objectives and goals, objectives/goals/targets are not indicated for future years. Without a longer-term, defined timetable, it may be difficult for the senior leaders, faculty, students, and stakeholders to measure the rate of accomplishment towards goals.  Remedial Plan: A measurement system will be created that measures progress toward achieving out-year goals.  Proposed Timeline: An approved system will be in place by 30 September 2005.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Services conduct periodic reviews to ensure units are operating in compliance with established directives and policies, and to determine progress being made toward long-term goals and performance requirements. As noted previously, all of the Services have contributed to and used the Institutional Training Readiness Report (ITRR). Each services monitors its training programs performance based on other key parameters. In the ITRR, Commanders assess institutional training readiness for Specialized Skill Training. C-levels indicate the capability to perform required institutional training mission. Activity commanders assign C-level ratings based on Training Workload-level ratings and the commander's judgment (which includes an assessment of Institutional Training Mission Support - manpower, equipment and facilities). NOTE: In future years, DoD can draw on ratings from the Defense Readiness Reporting System (DRRS), which is under development. Training organizations are responsive to data such as student throughput and class loading to maximize performance. They adjust their programs as practicable. In addition, they respond to senior leader guidance to ensure the delivery of the right skills at the right time. For example:: ?? Marine Corps uses the Training and Education Command (TECOM) Integrated Management System (TIMS) to track and assess throughput. ?? Air Force uses capacity resource analysis to identify gaps between requirements and actual production, and presents the information to Air Force leaders for resolution. Capacity resource analysis is an integral part of the process through which the AF develops flying training inputs to the PPBE process. Within most training activities, course reviews are conducted. These include content review boards at set intervals for maximum effectiveness and efficiency. Graduates are surveyed at the completion of each course, to ascertain the effectiveness of course material. Lessons learned flow continuously and from many sources: combat skills feedback, deployments, industry changes, etc. Additionally, smaller course adjustments are implemented continuously to ensure proper curriculum maintenance. On a monthly basis DOD carefully monitors the progress of its recruiting establishment which is a vital partner to meeting this program's goals. DOD monitors and reports to Congress and OMB, whether the military services are recruiting enough troops to fill positions required in accessions training. Organizations such as Defense Acquisition University (DAU) have automated these processes for real-time information retrieval through a tool known as DataMart. National Defense University (NDU) surveys the supervisors of its graduates to help determine course value.

Evidence: An example of C-levels that were assessed in 2004 can be found in the 2004 ITRR Tables 1-1, 2-1, 3-1, and 4-1; use of this information for management is shown in the Army and Marine Corps inputs, pages E-18 and E-25, respectively. NOTE: In future years, DoD can draw on ratings from the Defense Readiness Reporting System (DRRS), which is under development and scheduled for full operational capability October 2007. One example is Army Audit Report A-2003-0415-FFF, Air Defense Artillery School, prepared by the Army Audit Agency 27 Aug 2003. The report shows how the Army discovered new soldiers were being "over-trained." The report recommended [pg. 5] that the Army implement assignment-oriented training, which could save $41M over the period FY2004-FY2009. A similar report covering the Army Signal Center, A-2004-01212002-0555-FFF (16 Sep 2002), suggested that assignment-oriented training would save $104M over the same period [pg. 5]. Marine Corps Systems Approach to Training (SAT) manual, 4 Jun 2004, Appendix D on Sample Questionnaires. Quarterly recruiting and retention update to OMB/Congress NDU PAJE and MSA accreditation reports, NDU self-studies, internal component assessment reports, component performance plans and curriculum committee notes. NDU surveys include one given to supervisors of graduates of the Joint Forces Staff College (JFSC). NDU reported that in summer 2004 ICAF planned a major revision to their curriculum, but postponed it when surveys indicated the existing course was effective.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The military service secretary bears ultimate responsibility under Title 10 of the U.S. Code for organizing, training and equipping military service troops. Within each service, high-ranking military officers oversee the service-unique training commands which train personnel in basic and specialized skills. Activity managers within these commands are held accountable for cost, schedule, and performance results as they apply. Tracking mechanisms such as weekly, monthly, and/or quarterly performance reports are required, and accountability is built into individual performance plans. Gaps between requirements and production are identified to Service leaders. Activity commanders that are not meeting the mission can have funds withdrawn and redirected based on Service priorities and needs. Field activities use the Federal Acquisition Regulation (FAR) to encourage savings as defined in FAR - - Part 48 or terminate contracts as outlined in FAR - - Part 49. Component instructions and references cite these DoD directives and instructions and use the FAR in rewarding and penalizing other government partners and contractors as the situation dictates. For the Services, DoD's mid-year review examines funding execution data for efficiency and effectiveness. It could reveal additional resources to meet new manning requirements brought forth, or adjust the amount of resources brought forth. All programs go through a mid-year review, during which the funding that was allocated is compared to the funding that has been executed. DD Form 1002 (example included as evidence) provides a detailed summary of program execution to ensure funding levels are appropriate for all programs. As a result of mid-year review, managers and partners are held accountable for the program's cost, and any deviations in program execution can be corrected.

Evidence: DOD Directive 7045.14 specifies that the Planning, Programming, Budgeting and Execution process (PPBE) will involve centralized planning and decentralized execution. Centralized planning defines strategies, goals, budgets, and such; activity commanders are held accountable for execution. DoD Instruction 5010.37 policy states that the DoD components shall manage, provide resources, and evaluate activities based on output performance requirements and standards documented in performance work statements. The Efficiency Review (ER) shall be the basis for continued and directed efforts for productivity, performance, efficiency and effectiveness improvement. The ER process supports the DoD Productivity Program under DoD Directive 5010.31. DoD Directive 5010.31 states that the productivity program will focus management attention on achieving maximum Defense outputs within available resource levels by systematically seeking out and exploiting opportunities for improved methods of operation, in consonance with the Defense preparedness mission. Top level program partners are the commands in each Service responsible for executing the training and education mission: ?? TRADOC - The Army's Training and Doctrine Command, headquartered at Fort Monroe, Virginia. ?? NETC - The Naval Education and Training Command is headquartered in Pensacola, Florida. ?? AETC - The Air Force's Air Education and Training Command, headquartered in San Antonio, Texas. ?? TECOM - The Marine Corps' Training and Education Command, headquartered in Quantico, Virginia. As shown by the Institutional Training and Readiness Report (ITRR), these entities are held accountable for their performance and evaluated by a set of standards that include the Training Workload rating (TW) and the C rating that indicates readiness to perform their mission. Institutional Training and Readiness Report (ITRR), reporting organizations, overview of measures, reports of assessments. FAR - Part 48, Value Engineering, page 1; and Part 49, Termination of Contracts, page 1. An example of a report that all Services submit is Marine Corps appropriation status report, Form 1002.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Since these programs are funded with DOD's Operation and Maintenance (O&M) accounts, which expire annually, the program must obligate it in a timely manner. These O&M accounts have Congressionally-mandated restrictions on transferring funds. No more than $15 million in a funding category within O&M can transferred to another activity without notifying Congress. In addition to routine monthly reviews of execution by financial personnel at all training activities, an annual comprehensive mid-year funding execution review is performed by DOD and OMB. This review determines whether training and other programs are obligating funds in a timely manner. The review also determines what corrective actions are required to properly align remaining resources with projected needs for the balance of the year. DD Form 1002 provides a detailed summary of program execution to ensure funding levels are appropriate for all programs. As a result of mid-year review, timely obligation of funds is ensured, and their proper use is confirmed. Activities have financial management offices responsible for ensuring that they obligate funds in accordance with spending plans and Financial Management Regulations."

Evidence: Office of the Secretary of Defense (OSD) Comptroller iCenter web site explaining budget execution (www.dod.mil/comptroller/icenter/budget/budgetexecution.htm). DoD Financial Management Regulation, DoD 7000.14-R (excerpts as noted). Marine Corps Form 1002 excerpt, as an example of a form all Services submit. Included are Basic Skill and Advanced Training Budget Activity sheets for Army and Navy as examples. FY2005 sheets show the estimated training costs for FY04 and the FY2006 sheets show the actual training costs for that same year. NDU has monthly budget meetings with reconciliation, as well as semi-annual reviews with OSD (Comptroller).

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: As described previously, DOD and OMB conduct annual mid-year reviews to ensure that training and other programs are executing funding well and achieving their program goals. The Services continually measure program efficiency and effectiveness. For example, one measure is cost per student year which is used for resourcing decisions and to "cross-fertilize" among the services methods for improving the efficiency and effectiveness of their training programs. Measure 6 shown in the Measures tab displays "Cost per Student Year." This efficiency measure is used by the services to determine the cost per student year equivalent at service professional development activities. In addition, the Inter-Service Training Review Organization (ITRO) works to achieve efficiencies across all Services. The ITRO constantly reviews cost effectiveness and efficiencies, and improves them through course consolidations, collocations, and standardizations. An example of the efficiencies achieved within Basic Skill and Advanced Training program is the Joint Specialized Undergraduate Pilot Training (JSUPT). This is an example of the Services using collaboration, collocation, course consolidation and standardization to effectively and efficiently train military pilots. Based on this effort, primary military pilot training is conducted at either Vance AFB, OK or Naval Air Station Whiting Field, FL. Advanced training in multi-engine turbo props is provided at Naval Air Station Corpus Christi, TX. Helicopter training is provided by the Army at Fort Rucker, AL. The use of a common training activity for providing training for foundational aviation skills reduces training costs that are associated with manpower, facilities, aircraft procurement and associated maintenance costs, etc. The Navy and Marine Corps have always collaborated and consolidated efforts to train pilots at Navy flight training activities where possible. Another example of ITRO results is that Marine Corps personnel attend the Army Field Artillery School for training. In addition, Army soldiers that attend the school receive both basic and specialty training. ITRO reviews are based on mission training requirements. A mission-based review is necessary to ensure the training meets mission requirements and the demands of increasingly complex technology, combat systems, and training platforms. In addition, training efficiency is maximized by integrating current instructional technology design and technology. Curriculum Review Boards (CRBs) are conducted when changes to consolidated curricula impact resources or Service training standards. Life cycle management of the training is conducted every 3 years. At the level of an individual Service, the Air Force uses Training Technology working groups, Most Efficient Organization (MEO) teams, Contracted Reengineering Effort, Air Force Audit on 7-level training, and other programs to improve their skill training programs. And at the level of one activity, National Defense University (NDU) encourages leaders to conduct cost comparisons and trade-offs to achieve cost savings. NDU uses benchmarking to compare factors such as teaching hours, faculty pay, IT expenditures etc.

Evidence: Marine Corps budget document Form 1002, submitted as an example of a document all Services use during mid-year reviews. Interservice Training Review Organization (ITRO) references: All-Service regulation on Interservice Training, and ITRO Procedures Manual of 1 April 2004, Chapter 5. NDU uses comparison data (e.g.; National Center for Educational Statistics [NCES] Digest of Educational Statistics), where relevant, to examine teaching loads, etc. in comparable institutions.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Interservice Training Review Organization (ITRO) provides a framework for collaboration on training course curricula as well as coordination for school seats. ITRO functions are described in the all-Service regulation that defines the organization - it works to improve effectiveness and efficiency, and by its all-Service nature the ITRO is an example of coordination and collaboration. The ITRO was established in 1972 and operates through a series of committees. The Executive Board is composed of senior training officers (General and Admiral rank) from the Services including the Coast Guard. A Deputy Executive Board and Steering Committee perform high-level planning and other required duties. ITRO studies are performed by major committees and subcommittees formed as needed, composed of representatives from each Service, and directed by the Steering Committee. One example of collaboration is the Joint Specialized Undergraduate Pilot Training (JSUPT), which began in 1992. "Joint" in this instance refers to the fact that more than one Service is included for training. Primary training for Navy and Air Force pilots is conducted at either Vance Air Force Base (AFB), OK, or Naval Air Station Whiting Field, FL. Advanced training in multi-engine turbo props is provided at Naval Air Station Corpus Christi, TX. Helicopter training is provided by the Army at Fort Rucker, AL. Professional Development Education entities also provide strong examples of collaboration. The Professional Military Education (PME) review process, mandated by Chairman, Joint Chiefs of Staff, hinges on the Military Education Coordination Council (MECC) which includes senior members of all Service universities and colleges and is thus another prime example of collaboration. The MECC meets at least once annually. Defense Acquisition University (DAU) and National Defense University (NDU) have both established strategic partnerships with other government agencies, professional organizations, and academic institutions - lists are provided as evidence. For the longer term, the DoD Training Transformation Implementation Plan is designed to provide dynamic, capabilities-based training for the DoD in support of national security requirements across Services; Federal agencies; international coalition; international organizations; and state, local, and nongovernmental organization. Overall, Training Transformation will accomplish the following objectives: ?? Strengthen joint operations by preparing forces for new war fighting concepts ?? Continuously improve joint force readiness by aligning joint education and training capabilities and resources with combatant command needs ?? Develop individuals and organizations that think joint intuitively ?? Develop individuals and organizations that improvise and adapt to emerging crises ?? Achieve unity of effort the from a diversity of means. A Government Accountability Office (GAO) report on Training Transformation looks to the future by stating that when Training Transformation is fully operational in 2010, it will have "established a robust network of training capabilities that are integrated throughout the department to provide enhanced joint individual and unit training focused on commanders' needs and linked to readiness assessments."

Evidence: All-Service regulation on Interservice Training (28 Aug 1998), section 4 describing Interservice Training Review Organization (ITRO) functions. ITRO Procedures Manual (1 Apr 2004), Introduction, paragraph 5. Air Force fact sheet on Joint Specialized Undergraduate Pilot Training (April 2003); source: www.af.mil/factsheets. Chairman Joint Chiefs of Staff Instruction (CJCSI) 1800.01A, Enclosure C. (1 Dec 2000) Defense Universities provide some illustration of the degree of collaboration: ?? DAU has established over 85 partnerships and agreements (www.dau.mil page on Strategic Partnerships). For example, a strategic partnership with Grambling State University centers on DAU course work that Grambling will allow members of the Department of Defense (DoD) Acquisition, Technology and Logistics (AT&L) workforce to transfer towards Grambling State University bachelor and master degrees. Another partnership involves Old Dominion University (ODU), and includes a Masters Certificate Program in Government Contracting for qualified members of the DoD AT&L workforce. ?? NDU has agreements with over 20 public and private universities for cooperative degrees and participates in numerous other collaborative efforts including consortiums, joint research, advisory boards, etc. Source: NDU 2004 Annual Report to CJCS. NDU's Center for Technology and National Security co-hosted a series of workshops with the National Security Council and Department of Homeland Security. DoD Training Transformation Implementation Plan Executive Summary (9 Jun 2004) and GAO-05-548, MILITARY TRAINING: Actions Needed to Enhance DOD's Program to Transform Joint Training (June 2005).

Does the program use strong financial management practices?

Explanation: DOD's financial management weaknesses have been well documented and affect all activities and organizations within the program. Specifically, organizations throughout DOD do not have a clean financial audit opinion at this time. To the greatest extent possible, however, financial managers follow extensive Service-level guidance and procedures, and their organizations undergo regular and rigorous audits and inspections. In addition, the Services are addressing existing financial management weaknesses. The Navy for example conducts an Operational Risk Management (ORM) assessment annually and is striving for an unqualified opinion on its FY07 financial statements. Marine Corps Training and Education Command (TECOM) internal budgeting and accounting procedures and financial interface with Headquarters Programs and Resources (P&R) appear sound. The Financial Management section annually undergoes an Internal Management Control Program analysis, along with an inspection by the Resource, Evaluation and Analysis section of Marine Corps Combat Development Center Comptroller. In addition, P&R constantly monitors TECOM's monthly obligation rates against the spending plan to ensure resources are not wasted.

Evidence: A copy of a recent audit of the Navy's General Fund and Working Capital Fund is included as evidence of financial management 'at work'. All Services undergo similar type audits as part of their financial management efforts. As a specific example, Defense Acquisition University (DAU) has implemented cost accounting, time accounting, and budgeting tools as part of their financial management system.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The annual Institutional Training Readiness Report (ITRR) and Personnel Readiness of Military Forces measures have clear standards to evaluate the readiness of military training institutions to meet their mission and the readiness of trained personnel, respectively. These production levels are based on the skills requirements articulated by senior military leadership. Both the ITRR and the Readiness measure help senior military leaders determine if the quantity and quality of training is sufficient. ?? Service leaders use the ITRR C-ratings to monitor the readiness of training institutions to perform their training missions. If an institution receives a low "C"-rating, an action plan for improvement is required of the institution and reviewed. ?? Personnel Readiness of Military Forces is reported quarterly to Under Secretary of Defense (Personnel and Readiness). It includes several measures relevant to program management at a high level: Strategic Management of Human Capital and Financial Management. If personnel "C"-ratings are low for military units, senior DOD leaders immediately require remedial efforts by the military service to address the situation. If more training is required, unit personnel are very quickly assigned that training. In addition to these initiatives, program leaders and managers have taken significant steps to correct management deficiencies. For instance: ?? Air Force uses an earned value management system to track performance of the flight training program. The system shows that flight training programs are meeting their cost and schedule performance targets. ?? The Navy and Marine Corps acquired the Navy Marine Corps Intranet (NMCI) to function as a backbone to enhance learning, knowledge management and personnel development, in addition to improving interoperability, collaboration, mission accomplishment and related IT service needs. Navy Knowledge Online (NKO) is a super-portal that uses NMCI to help it serve as Navy's premier interactive education and training tool for sea warriors, is moving into another phase of service to the fleet. The Web site is a one-stop knowledge location for Navy education, training and professional growth management. The Army has developed a similar portal, Army Knowledge Online (AKO) with the same objectives in mind.

Evidence: Institutional Training and Readiness Report (ITRR) sections on training workload, with Marine Corps included as an example of ratings, explanation, and get-well plan. Personnel Readiness of Military Forces from the Measuring the Status of the Force (MSOF) brief (excerpt). DoD Supplementary FY2007-FY2011 Program and Budget Submission Guidance (excerpt) directing each Service to "use performance-based measures for 100% of its resources" as a step to address management deficiencies in the areas of budget and finance. Other developments that show addressing management: ?? The Navy has established the Naval Personnel Development Command (NPDC) as the single entity responsible for individual skill training (outside flight training). ?? Marine Corps' TIMS initiative is an example. ?? Defense Acquisition University (DAU) Baldrige remedial action plans specify steps to address management deficiencies. ?? National Defense University (NDU) Middle States Association Self Assessment, NDU Strategic Framework (working document). ?? The Army Knowledge Online (AKO) portal has more than 1.6 million registered users; Navy's NKO portal is being used by approximately 450,000 personnel.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Providing trained and ready personnel to military units is a critical part of military readiness. DOD's targets and baselines for this program are based on providing an uninterrupted stream of trained and ready troops. Targets and baselines for basic and specialized training are very high and do not change annually and over the long term because military readiness must be maintained at very high levels. The long-term goal for "Completion Rate for Flight Training" (Measure 1) is 90%. That goal has been exceeded for the last two consecutive years. The goal of 90% is based on the number of aviators planned to be trained, as described in the Institutional Training readiness Report (ITRR). "Percentage of enrollees completing Special Skills training" (Measure 2) is similar to the preceding measure, in that its goal is 90% of the number programmed to complete. Similarly, Measure 3 measures the unit's readiness is consistently high. This is due to the enduring requirement to have military forces remain at very high readiness.

Evidence: Targets and timeframes for the measure "Completion Rate for Flight Training" (Measure 1) are shown in the Institutional Training Readiness Report (ITRR), Jan 2004 and Jan 2005, Table 2, page E-7 in both reports. Flight training goals have been exceeded by at least 5% during the last two years as demonstrated by the data provided in the 2003 and 2004 ITRRs. Measure 2: Budget documents (OP-5 exhibits) show planned and actual values for "Percentage of enrollees completing Special Skills training." The baseline was set as FY2001, when completion rate was 103.6%. Measure 3: DoD's Quarterly Montioring the Status of Forces Report (classified document), show the quarterly and cumulative annual unit readiness ratings.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Due to the limited number of annual performance metrics, a complete answer to this question is not possible. The high readiness of U.S. forces indicates that DOD is achieving the goals even if comprehensive metrics are not available.

Evidence: Institutional Training Readiness Report (ITRR) , January 2004. BA3 budget documents show the programmed number of graduates as well as actual number of graduates from Specialized Skill Training activities, as well as the weighted professional development education cost per student.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Measure 5 shows that the program continues to improve its efficiency against the baseline level. The effectiveness of the program is demonstrated by the high level of readiness of U.S. military forces as evidenced in the Personnel Readiness of Military Forces metric (Measure 3). In some cases, renegotiation of contracts allows for the opportunity to identify and achieve efficiencies each year. Defense Acquisition University (DAU), for example, reports: student throughput has gone up 204%, student travel costs down 45%, annual training cost per student decreased 300%, and faculty staffing down 10% since FY99 - all with a relatively flat budget. A large portion of these savings comes from increased use of low-cost online training, which allows a greater number of students to take advantage of DAU's programs for a relatively small investment. The numbers of students in the Joint Professional Military Education schools of National Defense University (NDU) are predetermined. The school also has a pre-determined faculty-student ratio established by Congress and repeated in CJCSI 1800.1B. The required number of faculty members can't be reached through existing funding so NDU negotiated for agency chairs from non-DoD entities that send students to the University. This saves money and allows NDU to meet its goals. A balancing concern, however, is that using more non-academicians could impact accreditation.

Evidence: Chairman, Joint Chiefs of Staff Instruction (CJCSI) 1800.1B, identifying a challenge to professional military education community as: "develop and enhance joint* capabilities in a time of global war, finite resources and multiple commitments." (*Joint in this instance refers to capabilities involving more than one Service.) The provided reference includes a schedule of accreditation for all Service professional military education activities.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Not applicable. There are no other agencies that conduct training with purpose and goals similar to military basic skills and advanced training. While there are many technical schools, aviation schools, and other professional training programs, several elements mark DoD's program as profoundly different. They include: ?? Emphasis on technical qualifications that support weapons systems, as well as maintaining and operating these systems under combat conditions. ?? Flight training that involves military-unique environments such as formation flying, landing on aircraft carriers, and flying in hostile airspace. ?? Professional development and leadership training that involves combat-environment considerations.

Evidence: DAU cites the following as a few examples: CUX Awards, ASTD BEST award, Training Top 100, Chief Learning Officer Magazine awards, Brandon Hall awards, knowledge sharing awards, and CUBIC awards. NDU refers to Middles States Association and PAJE Accreditations; NSA and DHS Certification as a Center of Academic Excellence in IA Education; BOV meeting notes; University awards (e.g. Federal Library of the Year).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: All professional development activities must be accredited by the Joint Staff. This examines whether the training programs adhere to the military's requirements. While this accreditation shows that the level of instruction meets military requirements, it is not of sufficient scope to indicate that the overall program is effective. The Center for Naval Analysis has conducted numerous independent evaluations of Navy and Marine Corps training. These studies show that the Navy and Marine Corps have done a good job of providing adequate numbers of trained sailors and soldiers, but do not indicate whether those sailors and soldiers are adequately trained. The GAO, however, has indicated that the training programs of all military services have generally been successful in meeting individual service requirements. Nonetheless, GAO has noted that DoD needs to take actions to enhance its training programs to promote joint readiness. Defense Acquisition University (DAU) was accredited in 2003 and received a favorable Baldrige report as a result of national competition in 2004. DAU is benchmarked by several organizations (International Association for Continuing Education and Training, American Council on Education and Council on Occupational Education) for nationally recognized best practices in the area of Training. National Defense University (NDU) undergoes accreditation review by Middle States Association of Colleges and Schools (MSA) as well as the PAJE described above. Both reviews are thorough and allow assessment of the University against predetermined standards (measures). The reviews are performed on recurring schedules and involve both internal assessment by university personnel and outside assessment by peers. MSA evaluates the university against characteristics of excellence in higher education at large. Both reviews examine processes and products. Typical areas reviewed by MSA include strategic planning and use of resources. NDU is currently fully accredited by MSA and its JPME programs are currently fully accredited by PAJE.

Evidence: Center for Naval Analyses (CNA) reports indicate improvements in the amount of time it takes to train new sailors. Chairman of the Joint Chief of Staff (CJCS) Process for Accreditation of Joint Education (PAJE) schedule and results for National War College, Industrial College of the Armed Forces, and Joint Forces Staff College. Reference: CJCS Instruction 1800.01B, August 30, 2004, "Officer Professional Military Education Policy." Statement of Accreditation Status on National Defense University, March 2002. The next periodic review is June 1, 2007. The next self study evaluation 2011-2012. The Handbook for Institutional Self-Study outlines the many requirements for certification. Defense Acquisition University (DAU) received a 6-year accreditation in 2003. Also: 2004 Baldrige Feedback Report and the Annual Reports. Actions Needed to Enhance DOD's Program to Transform Joint Training. GAO-05-548.