Detailed Information on the
Defense Civilian Education and Training Assessment

1.Progress is being made to establish improved processes for measuring retention rate??an indicator of the return on investment in Defense Civilian Education and Training. The date of expected completion is September 2009. In FY 2007 a process was established to measure the retention rate for another program component. Now retention rate is being measured for 4 of 10 individual components of Defense Civilian Education and Training. Improvements were also made in existing processes to measure retention rate for other program components. As a result, the composite retention rate reported for FY 2007 is a more accurate and complete overall program performance measure than was reported for prior years. The FY 2005 Baseline and Targets were recalibrated in FY 2007 accordingly.

2. Progress is being made to establish long-term and annual performance measures for those aspects of the program for which data are not currently available, including retention rate and average investment??an indicator of efficiency in Defense Civilian Education and Training. The date of expected completion is September 2009. In FY 2007 long-term and annual performance measures were established for the retention rate for one additional program component and average investment for two additional program components. Now average investment is available for all 10 individual program components of Defense Civilian Education and Training. As a result, the composite retention rate and composite average investment reported for FY 2007 are more accurate and complete performance measures than were reported for prior years. The FY 2005 Baseline and Targets were recalibrated in FY 2007 accordingly.

3. Long-term and annual targets and timeframes are continually reviewed for any changes needed to accelerate program improvement. Because this review occurs semi-annually, there is no date of expected completion for this improvement action. This year these targets and timeframes were recalibrated from FY 2005 Baseline and Targets to FY 2007 to account for the more accurate and complete performance measures instituted this year. As improvements in performance measurement processes are completed, the composite performance measures will become more accurate and complete. Long-term and annual targets and timeframes will be reviewed for possible recalibration again after all planned improvements are completed in September 2009.

Measure: Completion Rate

Explanation:A desired long-term/annual program performance result is closing gaps in critical skills and competencies. The gaps are filled when participants successfully complete the program. This measure is the percentage of participants who successfully complete the program. It is a composite measure computed by weighting the completion rates of individual sub-programs by the funding associated with those sub-programs. Thus, sub-programs are not equally weighted in computing this composite measure; they are weighted by the size of the sub-program measured in terms of funding.

Measure: Retention Rate

Explanation:A desired long-term program performance result is maximizing return-on-investment in closing gaps in critical skills and competencies. Length of employee service after program completion is a measure of the payback period for this investment. This measure is the percentage of program completers who remain employed in the Department of Defense at least 1 year after completing the program. It is a composite measure computed by weighting the retention rates of individual sub-programs by the funding associated with those sub-programs. Thus, sub-programs are not equally weighted in computing this composite measure; they are weighted by the size of the sub-program measured in terms of funding.

Measure: Average Investment

Explanation:Average program dollars invested per unit of output is an efficiency measure used to assess return-on-investment. This measure is the average program dollars invested per full-time equivalent (FTE) work year for interns and per student for non-interns. It is a composite measure computed by weighting the average investments of individual sub-programs (for interns or non-interns) by the funding associated with those sub-programs. Thus, sub-programs are not equally weighted in computing this composite measure; they are weighted by the size of the sub-program measured in terms of funding.

Is the program purpose clear?

Explanation: The purpose of this program is to recruit, develop, and retain a more highly qualified, motivated, diverse, and cost-effective civilian workforce than that which would have been achieved without this program, and achieve that result in the most cost-effective manner.

Evidence: DoD Directive 1400.5, "DoD Policy for Civilian Personnel," January 12, 2005, Enclosure 1, Paragraph E1.1.3 states principles of DoD policy for civilian training and development (http://www.dtic.mil/whs/directives/corres/pdf/d14005_011205/d14005p.pdf). DoD Directive 1430.4, "Civilian Employee Training," November 16, 1994, Paragraph 3.1 states the overall policy for civilian training (http://www.dtic.mil/whs/directives/corres/pdf/d14304wch1_013085/d14304p.pdf). DoD Directive 1430.2, "Civilian Career Management," November 16, 1994, Paragraph 3 states the overall policy for development and operation of civilian career programs (http://www.dtic.mil/whs/directives/corres/pdf/d14302wch1_061381/d14302p.pdf). The Assistant Secretary of Defense for Force Management Policy established the Defense Career Intern Program (http://www.cpms.osd.mil/fas/staffing/pdf/dcip.pdf) to implement the Federal Career Intern Program established by Executive Order 13162 (http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=2000_register&docid=fr12jy00-126.pdf). DoD Directive 5000.52, "Defense Acquisition, Technology, and Logistics Workforce Education, Training, and Career Development Program," January 12, 2005 (http://www.dtic.mil/whs/directives/corres/pdf/d500052_011205/d500052p.pdf) established the DoD Acquisition Corps. DoD Directive 1430.16, "Defense Leadership and Management Program (DLAMP)" (http://www.dtic.mil/whs/directives/corres/pdf/d143016_041197/d143016p.pdf) established the DoD-wide management and leadership development program.

Does the program address a specific and existing problem, interest, or need?

Explanation: The Department of Defense (DoD) has a continuing need to recruit, develop, and retain civilian employees for positions that are hard to fill and keep filled due to competition for skills in the national labor market, retirements in the aging workforce, and other attrition, and for new positions created by transformation of the DoD Total Force (i.e., changes in the requirements for military, civilian, and contractor personnel).

Evidence: Section 1122 of the Fiscal Year 2006 National Defense Authorization Act (Public Law 109-163, January 6, 2006) requires DoD to develop and implement a strategic human capital plan to shape and improve its civilian employee workforce (http://thomas.loc.gov/cgi-bin/toGPObsspubliclaws/http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=109_cong_public_laws&docid=f:publ163.109.pdf). DoD must assess the critical skills and competencies needed by civilian employees to support current and future national security requirements, project trends in the civilian workforce based on expected losses due to retirement and other attrition, and develop and reshape the civilian workforce to address both current and projected gaps in critical skills and competencies. The President's Management Agenda (PMA) requires "Agencies will build, sustain, and effectively deploy the skilled, knowledgeable, diverse, and high-performing workforce needed to meet the current and emerging needs of government and its citizens." The quote is from PMA item "Strategic Management of Human Capital" (page 14). The DoD "Proud to Be III" (July 2006) PMA goals include planning to meet targets for closing competency gaps in mission-critical occupations; implementing succession strategies, such as structured executive development programs to produce a leadership talent pool, so that DoD can meet targets for closing leadership competency gaps; and reducing the under-representation of women and minorities and sustaining diversity in the civilian workforce, particularly in mission-critical occupations and leadership ranks. The DoD Civilian Human Capital Strategic Plan 2006-2010 established Goal 1, "World-Class Enterprise Leaders" and Goal 2, "Mission-Ready Workforce" and a business plan to achieve these goals. This program supports these goals. Example: The Army Civilian Training, Education, and Development System (ACTEDS) is a dynamic program that consistently assures that Army attracts, develops and retains a civilian workforce that is prepared to support and execute the Army's mission both in war time and in peace. It supports the eight Army Campaign Plan Objectives with particular emphasis on Goal 1: Support Global Operations; Goal 2: Transform the Force; Goal 3: Recruit and Retain Competent, Adaptive, and Confident Warfighters and Civilians; and Goal 6: Shape the Future Force. It is aligned with the PMA and the DoD Civilian Human Capital Strategic Plan. Army has a specific, validated methodology to accurately capture its requirements. Intern/Fellow needs are identified by occupation and command via a sophisticated forecasting tool that accurately identifies requirements for each of the Army Career Programs. Functional workforce proponents at Army Headquarters along with key officials of Army Commands assure that intern allocations are applied against the most critical Army needs. Once brought on board, the interns are developed under training plans that prescribe sequential and progressive on-the-job training, classroom sessions, and rotational assignments pertinent to that occupation. This approach assures that each intern achieves prescribed competencies, skills, and abilities needed to meet Army's requirements. Interns/Fellows sign a mobility agreement and upon graduation are placed where the Army needs them most. As they progress in their careers, they are eligible to compete for continuing education and training opportunities. These programs prepare the careerist to assume positions of greater responsibility and focus on developing leadership competencies and enhancing managerial capabilities. Army competitive professional development offerings prepare senior specialists, supervisors and managers to assume key Army leadership roles. Resource decisions associated with the Competitive Professional Development program are made by a committee of senior executives to assure that are applied towards Army's most compelling needs.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: This program uses already available education and training that meets its requirements, such as courses and programs offered by Federal, state, local, and private universities, colleges, and schools. When available sources are not appropriate or cost-effective, the Department of Defense (DoD) develops specialized sources in-house or by contract to meet its needs.

Evidence: DoD in-house providers and their contractors and other entities in the Federal, state, local, or private sectors supply the required education and training. For example, this program uses courses and programs offered by DoD schools, such as the National Defense University and the Defense Acquisition University. It also uses courses and programs offered by other Federal agencies, such as the Federal Executive Institute of the U.S. Office of Personnel Management and the Graduate School of the U.S. Department of Agriculture. It also uses courses and programs offered by state, local, and private universities, colleges, and schools, such as Syracuse University and the University of North Dakota. DoD Directive 5124.7, "Office of the Chancellor for Education and Professional Development," September 27, 1999 (http://www.dtic.mil/whs/directives/corres/pdf/d51247_092799/d51247p.pdf) set DoD policy that civilian education and professional development policies and requirements shall be implemented at the highest possible level of academic quality and cost-effectiveness. DoD officials who sponsor or have cognizance over DoD civilian education and professional development activities are charged with ensuring that appropriate standards of academic quality and cost-effectiveness are met.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: This program develops its annual and long-term requirements and budgets by analyzing civilian workforce gains, losses, and characteristics and mission requirements to identify the gaps in critical skills and competencies that must be mitigated. The Department of Defense prioritizes program plans and applies program resources to meet the most critical needs first. It uses already available education and training to keep program costs down. It compares program results with the results that would have been achieved without the program. It uses return-on-investment (ROI) and other studies to evaluate program effectiveness and efficiency, and to justify program improvements.

Evidence: Each year the Army analyzes employee gains and losses at grade levels 9 through 15 and projects employee retirements and attrition. It uses these analyses to develop intern hiring and budget requirements up to seven or more years in the future. For example, in Fiscal Year (FY) 2000 through FY 2003, losses exceeded gains at each of those grade levels. In FY 2004 through FY 2006, Army increased its budget for entry-level internships in its 26 career programs. As a result, gains began exceeding losses for positions at grades 9 /11 in FY 2004 and FY 2005. Army anticipates continuing improvements at grades 9 /11 in FY 2006 and beginning improvement at grade 12 in FY 2007 due to program investments. A recent Army ROI study found that 98% of interns remain with the Army after three years, versus 40% of non-interns, while 60% of interns remain with the Army after 25 years, versus 39% of non-interns. The loss of investment per 1,000 non-interns was estimated at $17.6 million per year. Moreover, 84% of interns are college graduates, versus 50% of non-interns. This study showed that the Army achieved a more highly qualified, motivated, diverse, and cost-effective workforce than that which it would have achieved without the program. The Air Force also uses workforce analysis and studies to optimize Civilian Competitive Development Program results. A recent Air Force study documented tangible or intangible positive ROI in program content, career progression and mobility, competency and skill proficiency, retention, and certain other aspects of the program, including substantial cost avoidance or savings. As a result of this study, the Air Force identified program strengths and weaknesses, as well as program improvements needed. The Navy also uses workforce analysis to develop its annual and long-term program requirements and budgets. It also prioritizes program budget execution and monitors program performance to achieve optimal results with its limited resources. The Defense Leadership and Management Program (DLAMP) also uses workforce analysis and studies to optimize its program results. A recent study showed that DLAMP contributed to a more diverse senior management and leadership workforce than that which existed previously.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The mid year program execution review is performed by senior leaders, managers, and comptrollers who are outside the program and thus are independent of it. This review determines whether program funds are obligated in a timely manner, program purposes remain valid, funds are being spent for the intended purpose, and program execution is being accurately reported. This review also determines what actions (if any) are required to properly align remaining resources with projected needs for the balance of the fiscal year. The annual program budget review focuses on justifying requirements and plans in future years. As a result of mid year and annual program reviews, senior leaders, managers, and comptrollers make programmatic and funding changes to improve program effectiveness and efficiency.

Evidence: In Fiscal Year (FY) 2005 senior Army leaders reviewed the Army program. By using funds from other sources, the Army increased the program funding to achieve a 60% increase in program effectiveness. In FY 2002 senior DoD leadership reviewed, refocused, and streamlined the Defense Leadership and Management Program to achieve a more effective and efficient program with a 40% reduction in program costs. DoD Directive 7045.14, "The Planning, Programming, and Budgeting System (PPBS)," November 21, 2003 (http://www.dtic.mil/whs/directives/corres/pdf/d704514wch1_052284/d704514p.pdf) and DoD Instruction 7045.7, "Implementation of the Planning, Programming, and Budgeting System (PPBS)," April 9, 1987 (http://www.dtic.mil/whs/directives/corres/pdf/i70457wch1_052384/i70457p.pdf) established policy and requirements for the annual and mid year program reviews.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Desired long-term program performance outcomes are closing gaps in critical skills and competencies and maximizing return-on-investment in the program. Primary measures of success are: (1) Completion Rate??participants who successfully complete the program; (2) Retention Rate??length of service after program completion; and (3) Average Investment??program dollars invested.

Evidence: The Department of Defense (DoD) Civilian Human Capital Strategic Plan 2006-2010 provides examples of outcome measures that may be used to assess progress toward achieving its goals, including Goal 1, "World-Class Enterprise Leaders" and Goal 2, "Mission-Ready Workforce" that relate directly to the purpose of this program. DoD is planning to systematically evaluate these proposed measures for utility as indicators of success. Illustrative outcome measures include: Acceptable Rate of Attrition (within a predefined range), Acceptable Rate of Internal Movement (within a predefined range), Acceptable Level of New Hire Turnover (within a predefined range), Acceptable Level of Retention (within a predefined range), Acceptable Level of Retention Beyond Retirement Eligibility (within a predefined range), and Competency Gaps Assessed and Closed. New measures under development include Correlation Between Training and Performance Appraisals, Promotions, and Awards, and Correlation Between Training and Retention.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Long-term program targets are: (1) increasing Completion Rate to >99% by Fiscal Year (FY) 2011; (2) increasing Retention Rate to >99% by FY 2011; and (3) maintaining Average Investment at a generally constant level after adjusting for inflation (e.g., no more than 2.5% increase per year).

Evidence: The Department of Defense (DoD) Civilian Human Capital Strategic Plan 2006-2010 provides examples of outcome measures that may be used to assess progress toward achieving its goals, including Goal 1, "World-Class Enterprise Leaders" and Goal 2, "Mission-Ready Workforce" that relate directly to the purpose of this program. DoD is planning to systematically evaluate these proposed measures for utility as indicators of program success. Illustrative outcome measures include: Acceptable Rate of Attrition (within a predefined range), Acceptable Rate of Internal Movement (within a predefined range), Acceptable Level of New Hire Turnover (within a predefined range), Acceptable Level of Retention (within a predefined range), Acceptable Level of Retention Beyond Retirement Eligibility (within a predefined range), and Competency Gaps Assessed and Closed. New measures under development include Correlation Between Training and Performance Appraisals, Promotions, and Awards, and Correlation Between Training and Retention. The Defense Human Resources Activity budget for the Defense Leadership and Management Program established the program goal of 200 graduates per year (page 57 of the budget justification for Fiscal Year 2007).

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Annual program targets are: (1) increasing Completion Rate by at least 1% per year; (2) increasing Retention Rate by at least 0.25% per year; and (3) maintaining Average Investment at a generally constant level after adjusting for inflation (e.g., no more than 2.5% increase per year).

Evidence: The Department of Defense (DoD) Civilian Human Capital Strategic Plan 2006-2010 provides examples of annual measures that may be used to assess progress toward achieving its goals, including Goal 1, "World-Class Enterprise Leaders" and Goal 2, "Mission-Ready Workforce" that relate directly to the purpose of this program. DoD is planning to systematically evaluate these proposed measures for utility as indicators of program success. Illustrative outcome measures include: Acceptable Rate of Attrition (within a predefined range), Acceptable Rate of Internal Movement (within a predefined range), Acceptable Level of New Hire Turnover (within a predefined range), Acceptable Level of Retention (within a predefined range), Acceptable Level of Retention Beyond Retirement Eligibility (within a predefined range), and Competency Gaps Assessed and Closed. New measures under development include Correlation Between Training and Performance Appraisals, Promotions, and Awards, and Correlation Between Training and Retention. The Defense Human Resources Activity budget for the Defense Leadership and Management Program established the program goal of 200 graduates per year (page 57 of the budget justification for Fiscal Year 2007).

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baseline measures have been established for Fiscal Year (FY) 2005: (1) Completion Rate--92.9%; (2) Retention Rate--97.7% of employees remain in Department of Defense at least 1 year after completing the program; and (3) Average Investment--$69.6K per full-time equivalent work year for interns, $4.3K per student for non-interns. Annual targets have been established for each of these measures through FY 2011.

Evidence: Examples: Army Career Intern Program ($107 million) achieved 99% Completion Rate, 98% Retention Rate for 1/+ years of service and 65% Retention Rate for 20/+ years of service after program completion, and Average Investment of $71,000 per full-time equivalent intern work year. Navy intern programs ($58 million) achieved 88% Completion Rate and Average Investment of $67,100 per full-time equivalent intern work year. The Air Force Renewal Program ($69 million) achieved 89% Completion Rate and 98% Retention Rate for 1/+ years of service after program completion. The Defense Leadership and Management Program ($23 million) achieved 73% Completion Rate (defined as 145 graduates out of the annual program goal of 200 graduates), 94% Retention Rate for 1/+ years of service and 83% Retention Rate for 5 or more years of service in the Department of Defense after graduation, and Average Investment of $13,600 per participant per year.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program manager, contracting officers, and their representatives incorporate specific measures into their performance agreements with program partners. Compliance with all applicable laws, regulations, policies, and performance agreements is necessary to receive program funding and achieve the program goals.

Evidence: Specific measures vary among performance agreements. For in-house providers, the agreements may include descriptions of the work to be performed, standards of performance, and estimated costs. An example is an agreement for senior level professional military education describing the curriculum to be provided, requirements for academic credit granted toward a degree, and student enrollments. For contracts, the statements of work typically include descriptions of the program objectives, tasks, and deliverables, as well as the schedule of delivery. An example is a contract for a program return-on-investment study. For tuition assistance, the agreements may include a description of the course, the accreditation of the academic institution providing the education, the dollar amount of tuition and fees, and any pay-back obligation of the student. Sample agreements include the Memoranda of Agreement between the Defense Leadership and Management Program and the United States Office of Personnel Management for training in executive competencies and the National Defense University for senior-level professional military education, as well as the agreements between the Army and Syracuse University for training in comptrollership and the University of North Dakota for training in military logistics. DoD Directive 5124.7, "Office of the Chancellor for Education and Professional Development," September 27, 1999 established policy for academic quality and cost-effectiveness of DoD civilian education and professional development activities, and charged DoD officials with ensuring that appropriate standards of academic quality and cost-effectiveness are met (http://www.dtic.mil/whs/directives/corres/pdf/d51247_092799/d51247p.pdf). On August 15, 2001, the Acting Deputy Assistant Secretary of Defense for Civilian Personnel Policy issued DoD policy on payment for expenses for civilian employees to obtain academic degrees (http://www.cpms.osd.mil/dlamp/education/docs/PDF/memo.pdf and http://www.cpms.osd.mil/dlamp/education/docs/DOC/policydegree1.doc). On June 17, 2002, the Deputy Assistant Secretary of Defense for Civilian Personnel Policy issued DoD policy on payment for expenses for civilian employees to obtain professional credentials (http://www.cpms.osd.mil/dlamp/education/docs/DOC/credentials.doc).

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Senior leaders, managers, and comptrollers who are outside the program and thus are independent of it evaluate program effectiveness and efficiency semi-annually, annually, and as needed. In addition, the program manager uses independent contractors as needed for return-on-investment (ROI) studies to evaluate program effectiveness and efficiency and justify program improvements. The Department of Defense (DoD) also uses the RAND Corporation and other professional organizations to evaluate the relevance and effectiveness of programs and recommend improvements needed. Another independent evaluation is the semiannual Status of Forces Survey of DoD civilian employees and their supervisors to provide feedback for evaluating existing programs and policies, assessing needs for new programs and policies, and monitoring progress. These evaluations all provide a basis for programmatic and funding changes to improve program effectiveness and efficiency.

Evidence: DoD Directive 7045.14, "The Planning, Programming, and Budgeting System (PPBS)," November 21, 2003 (http://www.dtic.mil/whs/directives/corres/pdf/d704514wch1_052284/d704514p.pdf) and DoD Instruction 7045.7, "Implementation of the Planning, Programming, and Budgeting System (PPBS)," April 9, 1987 (http://www.dtic.mil/whs/directives/corres/pdf/i70457wch1_052384/i70457p.pdf) established policy and requirements for the annual and mid year program reviews. The DoD Financial Management Regulation, DoD 7000.14-R (http://www.defenselink.mil/comptroller/fmr/index.html) was updated in March 2006. Volume 6A, Chapter 4, Annex 5 requires monthly submission of the DoD form "Appropriation Status by Fiscal Year Program and Subaccounts," Report Control Symbol Acct Rpt (M) 1002 (http://www.defenselink.mil/comptroller/fmr/06a/index.html) to monitor execution. The mid-year program execution review determines whether the program is obligating funds in a timely manner, program purposes remain valid, funds are being spent for the intended purpose, and program execution is being accurately reported. The mid-year review also determines what actions (if any) are required to properly align remaining resources with projected needs for the balance of the fiscal year. The annual review of the program budget is similar to the mid year review of program execution, except that it focuses on justifying program requirements and plans in future years. Relevant RAND Corporation studies include: "Intern Programs as a Human Resources Management Tool for the Department of Defense" (http://www.rand.org/pubs/monographs/2004/RAND_MG138.pdf), "Key Challenges in DoD Civilian Education and Workforce Planning" (http://www.rand.org/pubs/monograph_reports/MR1234/MR1234.ch4.pdf), "Trends Affecting DoD Civilian Education, Training, and Development" (http://www.rand.org/pubs/monograph_reports/MR1234/MR1234.ch3.pdf ), and "Characterizing the Future Defense Workforce" (http://www.rand.org/pubs/monograph_reports/MR1304/). DoD contracted with Arthur Anderson LLP to document the education and training opportunities in management, supervision, and organization offered to the DoD mid level (i.e., GS-9 to GS-12, or equivalent) civilian workforce. This study inventoried existing opportunities, identified critical gaps or needs relative to available opportunities, and recommended how DoD could address gaps that could affect its ability to prepare the next generation of managers and supervisors in the mid level civilian workforce. The semiannual DoD Status of Forces Survey is managed by the DoD Manpower Data Center (https://www.dmdc.osd.mil/pprofile/owa/pkg_profile.print_file_names).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The relationship between performance and budget is unclear and the Department of Defense (DoD) is unable to determine what level of performance increase a given level of additional funding would cause. However, DoD is working towards better performance budgeting. Steps DoD is taking include improving the planning, programming, budgeting, and execution system, which requires budgeting based on resources needed to achieve goals. Budget models are determined using historical program and financial data in combination with known or perceived changes to program elements (e.g., number of users, location of user populations, and future directives). Both annual and out year requests are tied to information provided by the forecasting models and identified needs of the programs. DoD Supplementary Fiscal Year (FY) 2007-FY 2011 Program and Budget Submission Guidance states that the Deputy Secretary of Defense has directed each Service to use performance-based measures for its budget formulation. The annual DoD Program and Budget Review Exhibit OP-5 and other exhibits for this program include a Financial Summary of program resource requirements and a Performance Criteria and Evaluation Summary of program results and goals. Future exhibits will strengthen the integration of budget and performance by including the annual program measures of Completion Rate, Retention Rate, and Average Investment for comparison with the FY 2005 baseline measures and the annual program targets.

Evidence: The OP-5 for Fiscal Year 2007 Budget Estimates is at pages 290-297 of the Army budget justification (http://www.asafm.army.mil/budget/fybm/FY07/oma/vol1.pdf), pages 1-7 of section 3C4L of the Navy budget justification (http://public.secnav.navy.mil/fy07.nsf/($reload)/AD057F4194B57FB78525711C007F1F85/$FILE/OMN+Book+Vol1.pdf ), and pages 615-623 of the Air Force budget justification (https://www.saffm.hq.af.mil/FMB/pb/2007/afoandm_3400/AF3400_FY07_PB_OM_Vol%201.pdf ). The Defense Human Resources Activity (DHRA) budget exhibit for the Defense Leadership and Management Program starts on page 147 of the DHRA budget justification (http://www.dod.mil/comptroller/defbudget/fy2007/budget_justification/pdfs/operation/O_and_M(co)_Volume_I_PB_2007.pdf).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: This program is covered along with other civilian human resources programs in the Department of Defense (DoD) Civilian Human Capital Strategic Plan. This plan is being revised in 2006 to integrate recent new requirements of law, the President's Management Agenda, the DoD "Proud to Be" goals, and the recommendations in the Quadrennial Defense Review Report and the DoD Performance and Accountability Report. The revised plan addresses both current and projected gaps in critical skills and competencies for the civilian workforce and focuses on assessing program performance against specific metrics and standards. It includes two goals that relate directly to this program: Goal 1, "World-Class Enterprise Leaders;" and Goal 2, "Mission-Ready Workforce."

Evidence: Section 1122 of the Fiscal Year 2006 National Defense Authorization Act (Public Law 109-163, signed into law on January 6, 2006) requires DoD to develop and implement a strategic human capital plan to shape and improve its civilian employee workforce (http://thomas.loc.gov/cgi-bin/toGPObsspubliclaws/http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=109_cong_public_laws&docid=f:publ163.109.pdf). DoD must assess the critical skills and competencies needed by civilian employees to support current and future national security requirements, project trends in the civilian workforce based on expected losses due to retirement and other attrition, and develop and reshape the civilian workforce to address both current and projected gaps in critical skills and competencies. The President's Management Agenda (PMA) requires "Agencies will build, sustain, and effectively deploy the skilled, knowledgeable, diverse, and high-performing workforce needed to meet the current and emerging needs of government and its citizens." (The quotation is from PMA item "Strategic Management of Human Capital" on page 14.) The DoD "Proud to Be III" (July 2006) PMA goals include planning to meet targets for closing competency gaps in mission-critical occupations; implementing succession strategies, such as structured executive development programs to produce a leadership talent pool, so that DoD can meet targets for closing leadership competency gaps; and reducing the under-representation of women and minorities and sustaining diversity in the civilian workforce, particularly in mission-critical occupations and leadership ranks. The 2006 Quadrennial Defense Review Report reinforced the need to maintain a mission-ready workforce and ensure world-class enterprise leadership, both now and in the future (pages 75-82, http://www.qr.hq.af.mil/pdf/2006%20QDR%20Report.pdf). The DoD Performance and Accountability Report published on November 15, 2005 recommended that program output measures be replaced with outcome measures to provide better information about the overall effect of program activities, and that program performance be analyzed against specific metrics and standards (pages 88-90, http://www.dod.mil/comptroller/par/index.html).

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: This program uses data collected in the Department of Defense (DoD) Civilian Personnel Data System. This system provides current and historical data for civilian workforce management and trend analysis. It includes data on employee types, positions, gains, losses, qualifications, job performance, retention, and diversity. It is a primary source of data for developing program performance measures, and for developing program requirements and prioritizing program resource allocations. This program also uses information provided by program partners per negotiated performance agreements. It uses this information to evaluate partner effectiveness and efficiency, and to assure delivery of required goods and services.

Evidence: The DoD Civilian Personnel Management Service maintains the DoD Civilian Personnel Data System (http://www.cpms.osd.mil/regmod/index.html) and publishes a monthly report of DoD Civilian Workforce Statistics (http://www.cpms.osd.mil/regmod/demographics/). This program uses contractors and other types of partners. For example, the Defense Leadership and Management Program (DLAMP) uses Troy State University, the National Defense University, and Senior Service Schools to provide courses. DLAMP also partnered with the United States Office of Personnel Management and the Human Resources Research Organization for a return-on-investment study and program evaluation of DLAMP. In each case, DLAMP developed a partnership agreement or a contract to ensure partner performance and delivery of satisfactory courses or reports. Sample agreements include the Memoranda of Agreement between the Defense Leadership and Management Program and the United States Office of Personnel Management for training in executive competencies and the National Defense University for senior-level professional military education, and the agreements between the Army and Syracuse University for training in comptrollership and the University of North Dakota for training in military logistics.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Senior leaders, managers, and comptrollers hold the program manager accountable for developing program requirements and budgets, and for meeting those requirements within available resources. Performance expectations and standards include meeting established annual and long-term program performance targets. In turn, the program manager holds program staff and partners accountable under negotiated performance plans, partnership agreements, or contracts that establish cost, schedule, and performance expectations and standards, as appropriate. The program manager evaluates performance as needed, semi-annually, annually, or as specified in performance agreements and contracts, including the consideration of student end-of-course evaluations as appropriate. The program manager also takes appropriate action to remedy performance deficiencies and reward effective and efficient performance.

Evidence: Within Army, a committee of seven Senior Executive Service members reviews program requirements, mission needs, and historical execution data and allocates available program funds. The requirements forecasting model, major command requirements, and career program managers all play a part in deciding how internship authorizations are allocated each year. Accountability is based on the placement of the interns after they graduate and how training funds are executed. For competitive professional development and management and leadership development, accountability is based on whether mission needs are met and how training funds are executed. Within Navy, senior Navy executives review program requirements, mission needs, and historical execution data and allocate available funds. Accountability is based on whether the statutory and mandatory education requirements are met for the acquisition workforce, the placement of trainees and interns after they graduate, and how training funds are executed. Within Air Force, the Directorate of Airmen Development and Sustainment develops and manages the Civilian Training, Education, and Development Plan. The program is based on validated requirements and priorities identified by commanders of the Major Commands, career program managers, and functional managers, as well as those specified by the United States Office of Personnel Management, the Department of Defense (DoD), and the Department of the Air Force. Accountability is based on established program performance criteria. For the Defense Leadership and Management Program (DLAMP), the program manager develops program requirements and plans based on workforce analyses and coordination with DoD Components. The Defense Leadership and Management Council establishes guidelines for program management, recommends resource levels, oversees effective use of resources based on continuous assessment of funding requirements, assesses the degree to which DLAMP and related programs are mutually complementary, and performs other program coordination and integration functions. The Position Description and Annual Performance Plan of the DLAMP manager illustrate how this program manager is held accountable for performance. Army, Navy, Air Force, and DoD use as-needed, semiannual, and annual staff reviews to identify and remedy performance deficiencies and reward staff performance. Program managers review partner cost, schedule, and performance as specified in performance agreements or contracts to identify and remedy performance deficiencies and ensure delivery of required products and services. The DoD Planning, Programming, and Budgeting System holds program managers accountable via monthly, mid year, and annual program reviews. Programmatic and funding changes may be made as required. DoD Directive 7045.14, "The Planning, Programming, and Budgeting System (PPBS)," November 21, 2003 established PPBS policy (http://www.dtic.mil/whs/directives/corres/pdf/d704514wch1_052284/d704514p.pdf). DoD Instruction 7045.7, "Implementation of the Planning, Programming, and Budgeting System (PPBS)," April 9, 1987 established PPBS procedures (http://www.dtic.mil/whs/directives/corres/pdf/i70457wch1_052384/i70457p.pdf). Army Regulation 690-950, "Career Management" (http://www.army.mil/usapa/epubs/pdf/r690_950.pdf), Secretary of the Navy Instruction 12273.1, "Assessment and Evaluation of Civilian Human Resources Management (HRM)" (http://neds.daps.dla.mil/directives/12273_1.pdf), Air Force Policy Directive 36-4, "Air Force Civilian Training, Education, and Development" (http://www.e-publishing.af.mil/pubfiles/af/36/afpd36-4/afpd36-4.pdf), and DoD Directive 1430.16, "Defense Leadership and Management Program (DLAMP)" (http://www.dtic.mil/whs/directives/corres/pdf/d143016_041197/d143016p.pdf) establish policy and responsibilities relating to program accountability.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: This program is funded with Operations and Maintenance appropriations, which expire annually. The Department of Defense (DoD) allocates these funds, monitors obligations of the funds, and compares obligations against time-phased plans to ensure effective resource use during program execution. In addition to monthly execution reviews, a mid year review is conducted by the DoD Components, the DoD, and the United States Office of Management and Budget. This review determines whether the program is obligating funds in a timely manner, program purposes remain valid, funds are being spent for the intended purpose, and program execution is being accurately reported. This review also determines what actions (if any) are required to properly align remaining resources with projected needs for the balance of the fiscal year. Report accuracy is ensured by mandatory compliance with the DoD Financial Management Regulation. It requires monthly submission of the DoD form "Appropriation Status by Fiscal Year Program and Subaccounts," Report Control Symbol Acct Rpt (M) 1002, to provide a detailed summary of program execution status. The DoD Components are working toward achieving a clean audit opinion for Fiscal Year 2007.

Evidence: DoD Directive 7045.14, "The Planning, Programming, and Budgeting System (PPBS)," November 21, 2003 established PPBS policy (http://www.dtic.mil/whs/directives/corres/pdf/d704514wch1_052284/d704514p.pdf). DoD Instruction 7045.7, "Implementation of the Planning, Programming, and Budgeting System (PPBS)," April 9, 1987 established PPBS procedures (http://www.dtic.mil/whs/directives/corres/pdf/i70457wch1_052384/i70457p.pdf). The DoD Financial Management Regulation, DoD 7000.14-R (http://www.defenselink.mil/comptroller/fmr/index.html) was updated in March 2006. Volume 6A, Chapter 4, Annex 5 requires the DoD form "Appropriation Status by Fiscal Year Program and Subaccounts," Report Control Symbol Acct Rpt (M) 1002 (http://www.defenselink.mil/comptroller/fmr/06a/index.html). The DoD Financial Improvement and Audit Readiness Plan was published on November 8, 2005 (http://www.defenselink.mil/comptroller/FIAR/documents/FIAR_Plan_Dec_2005Complete.pdf) to improve financial management so that DoD can obtain a clean audit opinion.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The overall program manager relies on sub-program managers in the Department of Defense (DoD) Components to review program performance continuously and conduct return-on-investment (ROI) studies as needed to evaluate and improve program effectiveness and efficiency. The sub-program managers monitor program attrition and take actions to reduce it, thereby increasing the percentage of program completions and also reducing program investments needed to fill gaps in critical skills and competencies. To assure program efficiency, the sub-program managers use already available sources that meet DoD requirements for education, training, and professional development. When appropriate sources do not exist, they may develop specialized sources in-house or by contract. They follow the applicable procurement laws, regulations, policies, and procedures, including cost comparison studies and competitive sourcing, to assure program efficiency. Where appropriate, they use distance learning or other innovative technology solutions to improve program effectiveness or efficiency.

Evidence: Return-on-investment (ROI) studies conducted by the Army Career Intern Program, the Air Force Civilian Competitive Development Program, and the Defense Leadership and Management Program (DLAMP) demonstrated that these programs achieved a more highly qualified, motivated, and diverse workforce than that which would have been achieved without these programs. These ROI studies also provided the basis for improvements in program effectiveness and efficiency. DLAMP partnered with the United States Office of Personnel Management to conduct a competitive source selection for its ROI study. DLAMP and Navy and Air Force internship programs have identified a need to reduce program attrition and are reviewing ways to accomplish that goal and thereby increase program efficiency. Distance learning and other innovative information technology solutions may provide more effective and efficient alternatives to the traditional classroom mode or other modes of course delivery. Where cost-effective, such solutions are being incorporated into the curricula of DLAMP and other DoD programs. Applicable law includes Title 5, Chapter 41 of the United States Code (http://uscode.house.gov/uscode-cgi/fastweb.exe?getdoc+uscview+t05t08+360+21++%28%29%20%20AND%20%28%285%29%20ADJ%20USC%29%3ACITE%20%20AND%20%28CHAPTER%20ADJ%20%2841%29%29%3AEXPCITE%20%20%20%20%20%20%20), and Title 10, Subtitle A, Part III of the United States Code (http://www.access.gpo.gov/uscode/title10/subtitlea_partiii_.html). Applicable regulations include Title 5, Code of Federal Regulations, Part 410 (http://www.access.gpo.gov/nara/cfr/waisidx_01/5cfr410_01.html), the Federal Acquisition Regulation (FAR) (http://www.acqnet.gov/far), and the DoD FAR Supplement (http://www.acq.osd.mil/dpap/dars/dfars/index.htm).

Does the program collaborate and coordinate effectively with related programs?

Explanation: This program collaborates and coordinates with Federal Government and industry programs to share lessons learned and promote synergy, and with the Department of Defense (DoD) occupation-specific career development programs to ensure that employees who complete this program meet requirements for education and training in their DoD occupational specialties.

Evidence: The Defense Leadership and Management Program (DLAMP) collaborates and coordinates with the United States Office of Personnel Management (OPM), the Senior Executive Service candidate development programs in other Federal agencies, the Federal Executive Institute, the Brookings Institution, and executive development programs in the private sector to share lessons learned and promote synergy. DLAMP also participates in the forums managed by OPM, such as the Learning and Development Advisory Council which is chartered to seek efficiencies in employee training and development. DLAMP also uses research and benchmark studies provided by the Corporate Leadership Council on executive development, succession management, and related topics. This program collaborates and coordinates with the Defense Acquisition, Technology, and Logistics Workforce Education, Training, and Career Development Program. This DoD functional workforce management program interprets statutory requirements, develops DoD requirements, and monitors program compliance with DoD policy for education and training of the acquisition, technology, and logistics functional workforces. DoD Directive 1430.16, "Defense Leadership and Management Program (DLAMP)" (http://www.dtic.mil/whs/directives/corres/pdf/d143016_041197/d143016p.pdf) established the DoD-wide leadership and management development program, including the Defense Leadership and Management Council to oversee it. DoD Directive 5000.52, "Defense Acquisition, Technology, and Logistics Workforce Education, Training, and Career Development Program," January 12, 2005 (http://www.dtic.mil/whs/directives/corres/pdf/d500052_011205/d500052p.pdf) established the Defense Acquisition Corps and the workforce management structure for program implementation.

Does the program use strong financial management practices?

Explanation: This program operates in compliance with the Department of Defense (DoD) policy, procedures, and practices for financial management. Heads of DoD Components submit to the Secretary of Defense an annual Statement of Assurance on the effectiveness of internal controls for programs, administrative activities, and operations, including financial reporting. To support this requirement, the program manager relies on sub-program managers in the DoD Components to assess the effectiveness of management controls applicable to their responsibilities. If material deficiencies are discovered, these managers must report those deficiencies with scheduled milestones leading to resolution of the deficiencies. This program is free of material internal control weaknesses. Program managers also continuously monitor program execution and report monthly on any discrepancies between planned and actual expenditures.

Evidence: The DoD Performance and Accountability Report for Fiscal Year (FY) 2005 (pages 35-54) does not list any current material weaknesses for this program, based on Statements of Assurance for the Army, Navy, Air Force, and Defense Leadership and Management Program). Earlier, the DoD Inspector General (IG) investigated allegations of inappropriate financial management of the Defense Leadership and Management Program in FY 2000 through FY 2002. The DoD IG final report did not substantiate the allegations, and thus it affirmed the financial management practices used in managing this program (http://www.dodig.mil/audit/reports/fy03/03-094.pdf). The United States Office of Management and Budget raised the DoD FY 2005 fourth quarter President's Management Agenda progress score from "red" to "yellow" because of the DoD Financial Improvement and Audit Readiness Plan. This plan is the DoD roadmap to improve internal controls, correct processes, and obtain an unqualified audit opinion. The DoD Financial Management Regulation, DoD 7000.14-R (http://www.defenselink.mil/comptroller/fmr/index.html) was updated in March 2006, and the DoD Financial Improvement and Audit Readiness Plan was published on November 8, 2005 (http://www.defenselink.mil/comptroller/FIAR/documents/FIAR_Plan_Dec_2005Complete.pdf). DoD Instruction 5010.40, "Managers' Internal Control (MIC) Program Procedures," January 4, 2006 (http://www.dtic.mil/whs/directives/corres/pdf/i501040_010406/i501040p.pdf) implements Section 3512, Title 31, United States Code (also referred to as Public Law 97-255 and the Financial Managers' Financial Integrity Act (FMFIA) of 1982 (http://www4.law.cornell.edu/uscode/html/uscode31/usc_sec_31_00003512----000-.html). The DoD Comptroller annually issues guidance to the DoD Components for the preparation of the required Statement of Assurance (http://www.defenselink.mil/comptroller/fmfia/index.html). For example, the May 31, 2005 Management Control Evaluation Certification Statement for the Army Civilian Training and Education System found declared "No material weaknesses are detected."

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Senior leaders, managers, and comptrollers review program performance semiannually and annually to identify program management deficiencies. They also perform other program reviews as needed. When deficiencies are found, they make the programmatic and funding changes needed to correct them. For example, several initiatives are now underway to improve the data in the DoD Civilian Personnel Data System that supports this program. The program manager relies annually on sub-program managers in the Department of Defense (DoD) Components to use the DoD Managers' Internal Control Program to discover and address any potential fraud, waste, and abuse or other program management deficiencies. When deficiencies are found, sub-program managers must report those deficiencies with scheduled milestones leading to resolution of the deficiencies. The program manager and sub-program managers also continuously review program performance and make the appropriate programmatic and funding changes within their respective discretionary authorities.

Evidence: In Fiscal Year (FY) 2005 senior Army leadership reviewed this program. By using funds from other sources, Army increased program funding to achieve a 60% increase in program effectiveness. In FY 2002 senior DoD leadership reviewed, refocused, and streamlined the Defense Leadership and Management Program (DLAMP) to achieve a more effective and efficient program with a 40% reduction in program costs. The DoD Civilian Personnel Data System supports this program. System management deficiencies are being addressed. This system is being updated to accommodate requirements of the National Security Personnel System. The previous limit of 20 training records per employee has been eliminated, and the Air Force and the Navy are working to reduce data deficiencies. The results will be higher program effectiveness and efficiency due to more complete and accurate workforce management data, and higher system efficiency due to minimizing the costs of frequent, ad hoc data calls. According to the DoD Performance and Accountability Report for Fiscal Year 2005 (pages 35-54), no material weaknesses exist in this program. The DoD Inspector General (IG) investigated allegations of inappropriate financial management of DLAMP in FY 2000 through FY 2002. The DoD IG final report did not substantiate the allegations, and thus it affirmed the financial management practices used in managing this program (http://www.dodig.mil/audit/reports/fy03/03-094.pdf).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: SMALL EXTENT - Overall, Completion Rate increased 4.8% from 88.1% in Fiscal Year (FY) 2004 to 92.9% in FY 2005 toward the long-term performance goal of >99% by FY 2011. Overall, Average Investment increased more than the 2.5% allowance of the annual performance goal. N/A (New Measure) - Retention Rate trend data will not be available until the end of FY 2006.

Evidence: Completion Rate for Army intern, competitive professional development, and management and leadership development programs exceeded the long-term performance goal of >99% in FY 2003, FY 2004, and FY 2005. Completion Rate for Navy intern programs increased from 87% in FY 2003 and FY 2004 to 88% in FY 2005. Completion Rate for Air Force intern programs increased from 87% in FY 2004 to 89% in FY 2005. Completion Rate for Navy acquisition workforce tuition assistance and continuous learning programs exceeded the long-term performance goal of >99% in FY 2003, FY 2004, and FY 2005. Completion Rate for Air Force civilian development programs increased from 81% in FY 2004 to 83% in FY 2005. Completion Rate for Air Force civilian training programs exceeded the long-term performance goal of >99% in FY 2004 and FY 2005. Completion Rate for the Defense Leadership and Management Program (DLAMP) increased from 9% in FY 2003 to 13% in FY 2004 and 73% in FY 2005 of annual program goals of 200 graduates. Completion Rate was lower in FY 2003 and FY 2004 because DLAMP had not yet fully implemented the refocused program. FY 2005 marked full implementation of the refocused program, and therefore, the beginning of eligibility for completion of this long-term (2-5 year) program. Retention Rate for Army and Air Force interns was 98% in FY 2005. Retention Rate for DLAMP graduates decreased from 81% in FY 2003 to 73% in FY 2004, but increased to 94% in FY 2005. The long-term performance goal is >99% Retention Rate for at least one year of service after program completion. The Army Career Intern Program return-on-investment (ROI) study showed that 98% of interns remain with Army after three years, versus 40% of non-interns, while 60% of interns remain with Army after 25 years, versus 39% of non-interns. Loss of investment per 1,000 non-interns was estimated at $17.6 million per year. Moreover, 84% of Army interns are college graduates, versus 50% of non-interns; 36% of Army interns are minorities, versus 30% of non-interns and 51% of Army interns are women, versus 36% of non-interns. A recent ROI study of the Air Force Civilian Competitive Development Program documented tangible or intangible positive ROI in the areas of program content, career progression and mobility, competency proficiency, retention, and certain other aspects of the program, including substantial cost avoidance or actual savings. As a result of the ROI study, the Air Force identified strengths and weaknesses in this program that provided the basis for program improvements. DLAMP is comparable to Senior Executive Service (SES) candidate development programs offered by other Federal agencies. DLAMP develops a diverse pool of senior management and leadership talent ready for executive responsibilities. DLAMP participants reflect the demographic characteristics of the Department of Defense (DoD) applicant pool (36% women and 18% minorities in DLAMP versus 26% women and 17% minorities in the DoD population at comparable grade levels). DLAMP graduates also have higher representation of women and minorities than the DoD population at comparable grade levels (33% women and 18% minorities among DLAMP graduates versus 26% women and 17% minorities in the DoD population at comparable grade levels). Moreover, DLAMP participants promoted to SES or equivalent positions have higher representation of women and minorities than the current SES population (35% women and 15% minorities among DLAMP participants promoted to senior positions versus 20% women and 7% minorities in the current DoD SES population).

Does the program (including program partners) achieve its annual performance goals?

Explanation: SMALL EXTENT - Overall, Completion Rate increased 4.8% from Fiscal Year (FY) 2004 to FY 2005, exceeding the annual performance goal of at least 1% per year increase. Overall, Average Investment increased more than the 2.5% allowance of the annual performance goal. N/A (New Measure) - Retention Rate trend data will not be available until the end of FY 2006.

Evidence: Completion Rate for Army intern, competitive professional development, and management and leadership development programs exceeded the long-term performance goal of >99% in FY 2003, FY 2004, and FY 2005. Completion Rate for Navy intern programs increased from 87% in FY 2003 and FY 2004 to 88% in FY 2005. Completion Rate for Air Force intern programs increased from 87% in FY 2004 to 89% in FY 2005. Completion Rate for Navy acquisition workforce tuition assistance and continuous learning programs exceeded the long-term performance goal of >99% in FY 2003, FY 2004, and FY 2005. Completion Rate for Air Force civilian development programs increased from 81% in FY 2004 to 83% in FY 2005. Completion Rate for Air Force civilian training programs exceeded the long-term performance goal of >99% in FY 2004 and FY 2005. Completion Rate for the Defense Leadership and Management Program (DLAMP) increased from 9% in FY 2003 to 13% in FY 2004 and 73% in FY 2005 of annual program goals of 200 graduates. Completion Rate was lower in FY 2003 and FY 2004 because DLAMP had not yet fully implemented the refocused program. FY 2005 marked full implementation of the refocused program, and therefore, the beginning of eligibility for completion of this long-term (2-5 year) program. Retention Rate for Army and Air Force interns was 98% in FY 2005. Retention Rate for DLAMP graduates decreased from 81% in FY 2003 to 73% in FY 2004, but increased to 94% in FY 2005. The long-term performance goal is >99% Retention Rate for at least one year of service after program completion. The Army Career Intern Program return-on-investment (ROI) study showed that 98% of interns remain with Army after three years, versus 40% of non-interns, while 60% of interns remain with Army after 25 years, versus 39% of non-interns. Loss of investment per 1,000 non-interns was estimated at $17.6 million per year. Moreover, 84% of Army interns are college graduates, versus 50% of non-interns; 36% of Army interns are minorities, versus 30% of non-interns and 51% of Army interns are women, versus 36% of non-interns. A recent ROI study of the Air Force Civilian Competitive Development Program documented tangible or intangible positive ROI in the areas of program content, career progression and mobility, competency proficiency, retention, and certain other aspects of the program, including substantial cost avoidance or actual savings. As a result of the ROI study, the Air Force identified strengths and weaknesses in this program that provided the basis for program improvements. DLAMP is comparable to Senior Executive Service (SES) candidate development programs offered by other Federal agencies. DLAMP develops a diverse pool of senior management and leadership talent ready for executive responsibilities. DLAMP participants reflect the demographic characteristics of the Department of Defense (DoD) applicant pool (36% women and 18% minorities in DLAMP versus 26% women and 17% minorities in the DoD population at comparable grade levels). DLAMP graduates also have higher representation of women and minorities than the DoD population at comparable grade levels (33% women and 18% minorities among DLAMP graduates versus 26% women and 17% minorities in the DoD population at comparable grade levels). Moreover, DLAMP participants promoted to SES or equivalent positions have higher representation of women and minorities than the current SES population (35% women and 15% minorities among DLAMP participants promoted to senior positions versus 20% women and 7% minorities in the current DoD SES population).

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: SMALL EXTENT - Overall, Completion Rate increased 4.8% from 88.1% in Fiscal Year (FY) 2004 to 92.9% in FY 2005, resulting in cost avoidance due to less program attrition. Overall, Average Investment increased more than the 2.5% allowance of the annual performance goal. N/A (New Measure) - Retention Rate trend data will not be available until the end of FY 2006. Recent, long-term return-on-investment (ROI) studies have demonstrated positive ROI. This program recruits, develops, and retains a more highly qualified, motivated, diverse, and cost-effective civilian workforce than that which would have been achieved without this program.

Evidence: Completion Rate for Army intern, competitive professional development, and management and leadership development programs exceeded the long-term performance goal of >99% in FY 2003, FY 2004, and FY 2005. Completion Rate for Navy intern programs increased from 87% in FY 2003 and FY 2004 to 88% in FY 2005. Completion Rate for Air Force intern programs increased from 87% in FY 2004 to 89% in FY 2005. Completion Rate for Navy acquisition workforce tuition assistance and continuous learning programs exceeded the long-term performance goal of >99% in FY 2003, FY 2004, and FY 2005. Completion Rate for Air Force civilian development programs increased from 81% in FY 2004 to 83% in FY 2005. Completion Rate for Air Force civilian training programs exceeded the long-term performance goal of >99% in FY 2004 and FY 2005. Completion Rate for the Defense Leadership and Management Program (DLAMP) increased from 9% in FY 2003 to 13% in FY 2004 and 73% in FY 2005 of annual program goals of 200 graduates. Completion Rate was lower in FY 2003 and FY 2004 because DLAMP had not yet fully implemented the refocused program. FY 2005 marked full implementation of the refocused program, and therefore, the beginning of eligibility for completion of this long-term (2-5 year) program. Retention Rate for Army and Air Force interns was 98% in FY 2005. Retention Rate for DLAMP graduates decreased from 81% in FY 2003 to 73% in FY 2004, but increased to 94% in FY 2005. The long-term performance goal is >99% Retention Rate for at least one year of service after program completion. The Army Career Intern Program ROI study showed that 98% of interns remain with Army after three years, versus 40% of non-interns, while 60% of interns remain with Army after 25 years, versus 39% of non-interns. Loss of investment per 1,000 non-interns was estimated at $17.6 million per year. Moreover, 84% of Army interns are college graduates, versus 50% of non-interns; 36% of Army interns are minorities, versus 30% of non-interns and 51% of Army interns are women, versus 36% of non-interns. A recent ROI study of the Air Force Civilian Competitive Development Program documented tangible or intangible positive ROI in the areas of program content, career progression and mobility, competency proficiency, retention, and certain other aspects of the program, including substantial cost avoidance or actual savings. As a result of the ROI study, the Air Force identified strengths and weaknesses in this program that provided the basis for program improvements. DLAMP is comparable to Senior Executive Service (SES) candidate development programs offered by other Federal agencies. DLAMP develops a diverse pool of senior management and leadership talent ready for executive responsibilities. DLAMP participants reflect the demographic characteristics of the Department of Defense (DoD) applicant pool (36% women and 18% minorities in DLAMP versus 26% women and 17% minorities in the DoD population at comparable grade levels). DLAMP graduates also have higher representation of women and minorities than the DoD population at comparable grade levels (33% women and 18% minorities among DLAMP graduates versus 26% women and 17% minorities in the DoD population at comparable grade levels). Moreover, DLAMP participants promoted to SES or equivalent positions have higher representation of women and minorities than the current SES population (35% women and 15% minorities among DLAMP participants promoted to senior positions versus 20% women and 7% minorities in the current DoD SES population).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: This program compares favorably with the overall industry in two ways. First, this program and each of its sub-programs has, or is in the processing of developing, a program management scorecard to enable results-based program evaluation. Second, several of the sub-programs have undertaken a return-on-investment (ROI) evaluation. The Army Career Intern Program compares favorably with both the Presidential Management Intern (PMI) Program and early career professional development programs in the private sector. The Air Force Civilian Competitive Development Program (CCDP) has completed a comprehensive ROI evaluation that compares favorably with similar Department of Defense (DoD) programs. The Defense Leadership and Management Program (DLAMP) compares favorably with the Senior Executive Service (SES) candidate development programs offered by the United States Office of Personnel Management (OPM) and other Federal agencies.

Evidence: The American Society for Training and Development (ASTD) noted in its 2005 State of the Industry Report that most ASTD BEST Award organizations either had or were in the process of developing management dashboards and scorecards to monitor the enterprise function. The ASTD noted in its 2003 State of the Industry Report that only about 10% of organizations were attempting any results-based evaluation, such as ROI. These ASTD State of the Industry Reports are available online (http://www.astd.org/astd/research/research_reports). The PMI Program is comparable to DoD internship programs. Its purpose is to recruit and select individuals with management potential who have recently earned or are about to receive a graduate-level degree. PMIs perform a two-year internship in the excepted service, entering at the GS-9 grade level and normally being promoted to GS-11 after successful completion of the first year. PMIs who perform successfully can be appointed in the competitive service at the end of the two-year program, when they typically are eligible for promotion to GS-12. A 2001 study by the United States Merit Systems Protection Board (MSPB) found that more PMIs advance into management ranks (30%) than do comparable non-PMI employees (18%), PMIs reflect the demographic diversity of the applicant pool (60% women and 22% minorities receiving graduate-level degrees), and PMI turnover rates are about the same as other comparable Federal employees (overall yearly average rate of 7% leave during the first five years of employment). This MSPB study is available online (http://www.mspb.gov/studies/rpt%2008-01%20pmi_program/pmi_program.pdf). Similarly, the Army Career Intern Program ROI study showed that 98% of interns remain with Army after three years, versus 40% of non-interns, while 60% of interns remain with Army after 25 years, versus 39% of non-interns. The loss of investment per 1,000 non-interns was estimated at $17.6 million per year. Moreover, 84% of Army interns are college graduates, versus 50% of non-interns; 36% of Army interns are minorities, versus 30% of non-interns and 51% of Army interns are women, versus 36% of non-interns. A recent RAND Corporation report stated "We found the evaluation of ECPD [early career professional development] programs to be more advanced in DoD than in the private sector. Specifically, the Army Career Intern Program has conducted extensive evaluations of the careers of program participants and has compared them with the careers of noninterns hired into similar positions." (Page 77, "Intern Programs as a Human Resources Management Tool for the Department of Defense" (http://www.rand.org/pubs/monographs/2004/RAND_MG138.pdf). A recent Air Force study documented tangible or intangible positive ROI in program content, career progression and mobility, competency and skill proficiency, retention, and certain other aspects of the CCDP. DLAMP is comparable to the SES candidate development programs offered by OPM and several other Federal agencies, in that DLAMP develops a diverse pool of senior management and leadership talent ready for executive responsibilities. DLAMP participants reflect the demographic characteristics of the DoD applicant pool (36% women and 18% minorities in DLAMP versus 26% women and 17% minorities in the DoD population at comparable grade levels). DLAMP graduates also have higher representation of women and minorities than the DoD population at comparable grade levels (33% women and 18% minorities among DLAMP graduates versus 26% women and 17% minorities in the DoD population at comparable grade levels). Moreover, DLAMP participants promoted to Senior Executive Service (SES) or equivalent positions have higher representation of women and minorities than the current SES population (35% women and 15% minorities among DLAMP participants promoted to senior positions versus 20% women and 7% minorities in the current DoD SES population).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent and comprehensive evaluations of program effectiveness and efficiency are performed as needed for each sub-program of this program. Recent return-on-investment (ROI) studies demonstrated that this program has achieved a more highly qualified, motivated, diverse, and cost-effective civilian workforce than that which would have been achieved without this program. The program manager and senior leaders use the RAND Corporation or other professional organizations to identify best practices in business, government, and academia; evaluate program relevance, effectiveness, and efficiency; and recommend program improvements. Another independent evaluation is the recurring Status of Forces Survey to provide feedback from DoD civilian employees and supervisors for program improvement.

Evidence: Independent Department of Defense (DoD) contractors, the RAND Corporation, or other professional organizations conducted ROI studies or other comprehensive evaluations of the Army Career Intern Program, the Air Force Civilian Competitive Development Program, and the Defense Leadership and Management Program (DLAMP). Those studies demonstrated that these programs were effective in achieving a more highly qualified, motivated, and diverse workforce than that which would have been achieved without these programs. The Army Career Intern Program ROI study results were summarized in an Army program briefing, dated March 23, 2006. The Air Force Civilian Competitive Career Development Program ROI study results were documented in a series of contract task reports by Human Technology, Inc. in 2004-2005. The ROI evaluation of DLAMP is currently underway by the Human Resources Research Organization. Other recent evaluations of DLAMP include "The Defense Science Board Task Force on Human Resources Strategy" (http://www.acq.osd.mil/dsb/reports/humanresources.pdf), "The Defense Leadership Management Program: Taking Career Development Seriously" (http://whitepapers.techrepublic.com.com/abstract.aspx?docid=115759&promo=400222&kw=%255C'career%255C'), a financial management audit by the DoD Inspector General (http://www.dodig.mil/audit/reports/fy03/03-094.pdf), and a study by the IBM Endowment for the Business of Government. DoD contracted with Arthur Anderson LLP to document the education and training opportunities in management, supervision, and organization offered to the DoD mid level (i.e., GS-9 to GS-12, or equivalent) civilian workforce. This study inventoried existing opportunities, identified critical gaps or needs relative to available opportunities, and recommended how DoD could address gaps that could affect its ability to prepare the next generation of managers and supervisors in the mid level civilian workforce. (See "The Department of Defense (DoD) Education and Training Survey: Survey Findings and Recommendations," March 13, 2001.) Relevant RAND Corporation studies include: "Intern Programs as a Human Resources Management Tool for the Department of Defense" (http://www.rand.org/pubs/monographs/2004/RAND_MG138.pdf), "Key Challenges in DoD Civilian Education and Workforce Planning" (http://www.rand.org/pubs/monograph_reports/MR1234/MR1234.ch4.pdf), "Trends Affecting DoD Civilian Education, Training, and Development" (http://www.rand.org/pubs/monograph_reports/MR1234/MR1234.ch3.pdf ), "Characterizing the Future Defense Workforce" (http://www.rand.org/pubs/monograph_reports/MR1304/), "Workforce Planning in Complex Organizations" (http://www.rand.org/pubs/research_briefs/RB7570/index1.html), "An Executive Perspective on Workforce Planning" (http://www.rand.org/pubs/monograph_reports/2005/MR1684.2.pdf), "An Operational Process for Workforce Planning" (http://www.rand.org/pubs/monograph_reports/2005/MR1684.1.pdf), "An Optimization Approach to Workforce Planning in the Information Technology Field" (http://www.rand.org/pubs/monograph_reports/2005/MR1484.pdf), "Civilian Education and Training in the Department of Defense: How Can We Gauge Its Value?" (http://www.rand.org/pubs/research_briefs/RB7542/index1.html), and "Ensuring the Quality and Productivity of Education and Professional Development Activities: A Review of Approaches and Lessons for DoD" (http://www.rand.org/pubs/monograph_reports/MR1257). The DoD Status of Forces Survey is managed by the DoD Manpower Data Center (https://www.dmdc.osd.mil/pprofile/owa/pkg_profile.print_file_names).