This is the accessible text file for GAO report number GAO-08-561 entitled 'Veterans' Benefits: Increased Focus on Evaluation and Accountability Would Enhance Training and Performance Management for Claims Processors' which was released on May 27, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Chairman, Committee on Veterans' Affairs, House of Representatives: United States Government Accountability Office: GAO: May 2008: Veterans' Benefits: Increased Focus on Evaluation and Accountability Would Enhance Training and Performance Management for Claims Processors: GAO-08-561: GAO Highlights: Highlights of GAO-08-561, a report to the Chairman, Committee on Veterans' Affairs, House of Representatives. Why GAO Did This Study: Faced with an increase in disability claims, the Veterans Benefits Administration (VBA) is hiring a large number of new claims processing staff. We were asked to determine: (1) What training is provided to new and experienced claims processors and how uniform is this training? (2) To what extent has VBA planned this training strategically, and how well is the training designed, implemented, and evaluated? and (3) To what extent is the performance management system for claims processors consistent with generally accepted practices? To answer the questions, GAO reviewed documents including VBA policies and training curricula; interviewed VBA central office officials; visited 4 of VBA’s 57 regional offices, which were selected to achieve diversity in geographic location, number of staff, and officewide accuracy in claims processing; and compared VBA’s training and performance management to generally accepted practices identified by GAO. What GAO Found: VBA has a standardized training curriculum for new staff and a training requirement for all staff, but does not hold staff accountable for meeting this requirement. The curriculum for new staff includes what is referred to as centralized training and training at their home offices. All claims processors must complete 80 hours of training annually, which may cover a mix of topics identified centrally and by regional offices. Individual staff members face no consequences for failing to meet the training requirement, however, and VBA has not tracked training completion by individuals. It is implementing a new system that should provide this capacity. Although VBA has taken steps to plan its training strategically, the agency does not adequately evaluate training and may be falling short in training design and implementation. VBA has a training board that assesses its overall training needs. However, the agency does not consistently collect feedback on regional office training, and both new and experienced staff GAO interviewed raised issues with their training. Some new staff raised concerns about the consistency of training provided by different instructors and about the usefulness of an on-line learning tool. Some experienced staff believe that 80 hours of training annually is not necessary, some training was not relevant for them, and workload pressures impede training. The performance management system for claims processors generally conforms to GAO-identified key practices, but the formula for assigning overall ratings may prevent managers from fully acknowledging and rewarding staff for higher levels of performance. The system aligns individual and organizational performance measures and requires that staff be given feedback throughout the year. However, VBA officials raised concerns about the formula used to assign overall ratings. Almost all staff in the offices GAO visited were placed in only two of five overall rating categories, although managers said greater differentiation would more accurately reflect actual performance differences. The Department of Veterans Affairs (VA) has not examined the ratings distribution, but acknowledges a potential issue with its formula and is considering changes. Figure: Fiscal Year 2007 Appraisals for Four Offices Were Concentrated in Two Categories: [See PDF for image] This figure is a stacked vertical bar graph depicting the following data: Office: Atlanta; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 81%; Excellent: 10%; Outstanding: 9%. Office: Baltimore; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 66%; Excellent: 10%; Outstanding: 24%. Office: Milwaukee; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 70%; Excellent: 2%; Outstanding: 28%. Office: Portland; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 72%; Excellent: 2%; Outstanding: 26%. Source: VBA regional offices. [End of figure] What GAO Recommends: GAO is recommending that VBA collect feedback on training provided by regional offices and use this feedback to further improve training, and hold staff accountable for meeting their training requirement. GAO also recommends that the VA assess and, if necessary, adjust its process for placing staff in overall performance categories. In its comments, VA agreed with GAO’s conclusions and concurred with the recommendations. To view the full product, including the scope and methodology, click on [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-561]. For more information, contact Daniel Bertoni (202) 512-7215 bertonid@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: VBA Has a Uniform Training Curriculum for New Claims Processors and an Annual Training Requirement for All Claims Processors, but Staff Are Not Held Accountable for Meeting This Requirement: VBA Is Taking Steps to Strategically Plan Its Training for Staff, but Does Not Adequately Evaluate Training and May Be Falling Short in Design and Implementation: Performance Management System for Claims Processors Generally Conforms to Accepted Practices, but May Not Clearly Differentiate between Performance Levels: Conclusions: Recommendations for Executive Action: Agency Comments: Appendix I: Objectives, Scope, and Methodology: Appendix II: Fiscal Year 2007 Core Technical Training Requirements for VSRs and RVSRs: Appendix III: Excerpts from Fiscal Year 2007 Training Plans for Four Regional Offices: Appendix IV: Agency Comments: Appendix V: GAO Contact and Staff Acknowledgments: Related GAO Products: Tables: Table 1: VBA's Disability Compensation and Pension Service's Claims Processing Teams: Table 2: Individual Performance Elements for VSRs and RVSRs: Table 3: Performance Elements for VSRs and RVSRs and Corresponding Organizational Performance Measures for VBA: Table 4: Selected Key Questions to Consider in Assessing Agency's Training Program: Table 5: Selected Accepted Practices for Effective Performance Management Systems: Table 6: Excerpt from Atlanta Regional Office Training Plan: Table 7: Excerpt from Baltimore Regional Office Training Plan: Table 8: Excerpt from Milwaukee Regional Office Training Plan: Table 9: Excerpt from Portland Regional Office Training Plan: Figures: Figure 1: Phases of Training for New VSRs and RVSRs: Figure 2: Excerpt from the Instructor's Guide for a Centralized Training Lesson on Reference Materials: Figure 3: Most Fiscal Year 2007 Training Hours Completed by Claims Processors in the Offices We Visited Were in the Form of Classroom Instruction: Figure 4: Sample of VBA's Centralized Training Evaluation Form: Figure 5: VA Overall Performance Appraisal Formula: Figure 6: Fiscal Year 2007 Overall Performance Ratings for Claims Processors in Four Regional Offices Were Concentrated in the Outstanding and Fully Successful Categories: Figure 7: Regional Offices Selected for Site Visits: Figure 8: Four Components of the Training and Development Process: Abbreviations: RVSR: Rating Veterans Service Representative: STAR: Systematic Technical Accuracy Review: TPSS: Training and Performance Support System: VA: Department of Veterans Affairs: VBA: Veterans Benefits Administration: VSR: Veterans Service Representative: [End of section] United States Government Accountability Office: Washington, DC 20548: May 27, 2008: The Honorable Bob Filner: Chairman: Committee on Veterans' Affairs: House of Representatives: Dear Mr. Chairman: The Veterans Benefits Administration (VBA) is facing an increased volume of claims for disability benefits related to the current conflicts in Afghanistan and Iraq as well as the aging of veterans from past conflicts. Between fiscal years 2000 and 2006, the number of disability-related claims filed annually with VBA increased by almost 40 percent. As a result, VBA continues to experience challenges in processing veterans' disability claims. As of fiscal year 2007, VBA had approximately 392,000 disability claims pending benefit decisions, and the average time these claims were pending was 132 days. According to VBA, the current conflicts have also produced more claims related to post-traumatic stress disorder and traumatic brain injury, conditions few VBA staff have had much experience evaluating. To process the increased volume of claims, in fiscal year 2007 the agency began hiring a large number of new Veterans Service Representatives (VSR), who collect evidence related to veterans' claims, and Rating Veterans Service Representatives (RVSR), who evaluate claims and determine benefit eligibility. It plans to add 3,100 new claims-processing staff by the end of fiscal year 2008. Given the increased volume of claims, the increased focus on certain types of disabilities, and the large number of new hires, training and performance management systems for VSRs and RVSRs now play an especially critical role in enabling VBA to meet its organizational claims processing goals for accuracy and productivity. Training that is properly designed and implemented is vital both to help new staff learn their jobs and experienced staff to update their knowledge and learn about emerging issues. An effective performance management system would also help VBA manage its staff on a day-to-day basis to achieve its organizational goals. To provide Congress with information on the training and performance management of claims processors, we were asked to determine: (1) What training is provided to new and experienced claims processors and how uniform is this training? (2) To what extent has VBA developed a strategic approach to planning training for claims processors and how well is their training designed, implemented, and evaluated? And (3) To what extent is the performance management system for claims processors consistent with generally accepted performance management practices in the public sector? To address these objectives, we collected documents and data from VBA central office and interviewed central office staff. In addition, GAO experts on training reviewed VBA documents related to training curriculum, lesson plans, and course evaluations. We conducted site visits to 4 of VBA's 57 regional offices--Atlanta, Baltimore, Milwaukee, and Portland, Oregon. These offices were selected to achieve diversity in geographical location, number of staff, and officewide accuracy in claims processing. While we examined VBA-wide policies and requirements, we primarily assessed how the training and performance management systems are implemented at four sites. Therefore, our results may not be representative of how these systems are implemented across all regional offices. We assessed VBA's training and performance management practices by comparing them to certain generally accepted practices for federal agencies in these areas that have been identified by GAO.[Footnote 1] We conducted this performance audit from September 2007 through May 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. (See app. I for more detailed information on our objectives, scope, and methodology.) Results in Brief: VBA has a standard training curriculum for new claims processors and an 80-hour annual training requirement for all claims processors, but staff are not held accountable for meeting this requirement. VBA's three-stage training program for new staff is intended to deliver training in a consistent manner. First, VBA policy states that new staff are required to complete some orientation training, which is provided in their home offices. Second, they are required to attend a 3- week standardized training session, referred to as centralized training, that provides a basic introduction to their job responsibilities. Third, new staff are required to spend several more months in training at their home offices, which is supposed to include on-the-job training, instructor-led training classes that follow a required curriculum, and use of an on-line learning tool called the Training and Performance Support System. VBA policy states that all claims processors are required to complete a minimum of 80 hours of training annually, and regional offices have some discretion over what training they provide to meet this requirement. At least 60 hours must be selected from a list of core topics identified by VBA central office. Regional offices may choose the topics for the remaining 20 hours based on local needs, such as to prevent errors identified in processing claims. Each regional office develops an annual training plan listing the courses needed, and VBA central office periodically reviews these plans and provides feedback to regional offices. Although VBA has a training requirement for VSRs and RVSRs, it does not have a policy outlining consequences for individual staff who do not complete their required training. Further, VBA does not maintain data on the training completed by individuals, but agency officials said they are currently implementing a new, on-line learning management system that should enable them to do so in the future. VBA is taking steps to strategically plan its training, but does not adequately evaluate its training and may be falling short in some areas of training design and implementation. VBA appears to have followed several accepted practices in planning its training, including the establishment of a training board that assesses VBA's overall training needs and makes recommendations to the Undersecretary for Benefits. Also, VBA makes some effort to evaluate its centralized training for new staff, soliciting feedback from students with forms that are well- constructed and well-balanced. However, VBA does not require regional offices to collect feedback on any of the training they provide to new and experienced staff. In fact, claims processors we interviewed raised some issues with the training they received. For example, some new staff reported that different instructors in centralized training sessions sometimes taught different ways of performing the same procedure, and that one of VBA's on-line learning tools--the Training and Performance Support System--is too theoretical and often out-of- date. More experienced staff had mixed opinions as to whether 80 hours of training annually is appropriate for all staff. Also, many experienced staff indicated that training topics are redundant from year to year, and some told us that courses available to them are not always relevant for their position or experience level because they are often adapted from courses for new staff. Some staff said they struggle to meet the annual 80-hour training requirement because of workload pressures. The Department of Veterans Affairs' performance management system for VSRs and RVSRs generally conforms to accepted practices, including aligning individual and organizational performance measures, but the system may not clearly differentiate among staff's overall performance levels. Several elements of VSRs' and RVSRs' performance are evaluated, and these elements are generally aligned with VBA's organizational performance measures. For example, VSRs and RVSRs are evaluated on their accuracy in claims processing, and one of VBA's organizational performance measures is accuracy in claims processing. VBA's performance management system is also consistent with other accepted practices, such as providing performance feedback throughout the year and emphasizing collaboration. However, the system may not clearly and accurately differentiate among the overall performance levels of VSRs and RVSRs. A VA-wide formula is used to translate an employee's ratings on all individual elements into one of five overall rating categories. Several VBA central and regional office managers raised concerns with this formula, saying that it is difficult for staff to be placed in certain overall performance categories, even if staff's performance truly does fall within one of those categories. In fact, when we reviewed the results of VSR and RVSR appraisals at the regional offices we visited, almost all staff were placed in either the outstanding (highest) or fully successful (middle) categories. To the extent that the performance appraisals do not make meaningful distinctions in performance, staff may lack the constructive feedback they need to improve, and managers may lack the information they need to reward top performers and address performance issues. Although VA acknowledged this issue and indicated that it is considering changes to the system, no formal actions have been taken to date. We are recommending that VBA central office collect feedback on training provided by the regional offices, to determine whether (1) 80 hours is the appropriate amount of annual training for all staff, (2) regional offices are providing training that is relevant for all staff, and (3) whether any changes are needed to improve the Training and Performance Support System. We are also recommending that VBA central office hold individual staff accountable for meeting their training requirement and that VA assess and, if necessary, adjust its performance rating system for staff to make it a more meaningful management tool. In its comments, VA agreed with our conclusions and concurred with our recommendations. Background: In fiscal year 2007, the Department of Veterans Affairs (VA) paid about $37.5 billion in disability compensation and pension benefits to more than 3.6 million veterans and their families. Through its disability compensation program, the VBA pays monthly benefits to veterans with service-connected disabilities (injuries or diseases incurred or aggravated while on active military duty). Monthly benefit amounts vary according to the severity of the disability. Through its pension benefit program, VBA pays monthly benefits to wartime veterans with low incomes who are either elderly or permanently and totally disabled for reasons not service-connected. In addition, VBA pays dependency and indemnity compensation to some deceased veterans' spouses, children, and parents and to survivors of servicemembers who died while on active duty. When a veteran submits a benefits claim to any of VBA's 57 regional offices, a Veterans Service Representative (VSR) is responsible for obtaining the relevant evidence to evaluate the claim. For disability compensation benefits, such evidence includes veterans' military service records, medical examinations, and treatment records from VA medical facilities and private providers. Once a claim is developed (i.e., has all the necessary evidence), a Rating Veterans Service Representative (RVSR) evaluates the claim, determines whether the claimant is eligible for benefits, and assigns a disability rating based on degree of impairment. The rating determines the amount of benefits the veteran will receive. For the pension program, claims processing staff review the veteran's military, financial, and other records to determine eligibility. Eligible veterans receive monthly pension benefit payments based on the difference between their countable income, as determined by VA, and the maximum pension amounts as updated annually by statute.[Footnote 2] In fiscal year 2007, VBA employed over 4,100 VSRs and about 1,800 RVSRs to administer the disability compensation and pension programs' caseload of almost 3.8 million claims. In 2001 the VA Claims Processing Task Force noted that VSRs were responsible for understanding almost 11,000 separate benefit delivery tasks, such as tasks in claims establishment, claims development, public contacts, and appeals. To improve VBA's workload controls, accuracy rates, and timeliness, the Task Force recommended that VA divide these tasks among a number of claims processing teams with defined functions. To that end, in fiscal year 2002, VBA developed the Claims Processing Improvement model that created six claims processing teams, based on phases of the claims process. (See table 1.) Table 1: VBA’s Disability Compensation and Pension Service’s Claims Processing Teams: Team: Triage Team; Summary of claims processing duties: Establishes the regional office's tracking procedures for all mail as well as processes claims that only require a brief review to determine eligibility. Team: Pre-Determination Team; Summary of claims processing duties: Develops evidence for disability ratings and prepares administrative decisions. Team: Rating Team; Summary of claims processing duties: Makes decisions on claims that require consideration of medical evidence. Team: Post-Determination Team; Summary of claims processing duties: Develops evidence for non-rating issues, processes benefit awards, and notifies veterans of rating decisions. Team: Public Contact Team; Summary of claims processing duties: Conducts personal interviews and handles telephone inquiries, including calls from veterans. Team: Appeals Team; Summary of claims processing duties: Handles requests for reconsideration of claims where veterans have formally disagreed with claim decisions. Source: VBA. Note: The Rating Board is made up of RVSRs, the Post-Determination and Public Contact teams are made up of VSRs, and the Pre-Determination, Triage, and Appeals teams are made up of both RVSRs and VSRs. [End of table] According to one VA official, new claims processing staff generally begin as VSRs and typically have a probationary period of about one year.[Footnote 3] After their probationary period ends, staff can either continue to qualify to become senior VSRs or apply for RVSR positions.[Footnote 4] VSRs are also given the option to rotate to other VSR claim teams to gain a broader understanding of the claims process. VBA Has a Uniform Training Curriculum for New Claims Processors and an Annual Training Requirement for All Claims Processors, but Staff Are Not Held Accountable for Meeting This Requirement: VBA has established a standardized curriculum for training new VSRs and RVSRs on how to process claims, and it has an 80-hour annual training requirement for both new and experienced staff; however, it does not hold individual staff accountable for meeting this requirement. VBA has designed a uniform curriculum for training new VSRs and RVSRs that is implemented in three phases--initial orientation training, a 3-week training session referred to as centralized training, and comprehensive on-the-job and classroom training after centralizing training. It also requires all staff to meet an annual 80-hour training requirement. To ensure that staff meet this requirement, each regional office must develop an annual training plan, which can contain a mix of training topics identified by VBA central office and by the regional office. However, individual staff members are not held accountable for meeting their training requirement. Training for New Staff Is Conducted in Three Stages Using a Uniform Curriculum: VBA has a highly structured, three-phased program for all new claims processors designed to deliver standardized training, regardless of training location or individual instructors. (See fig. 1.) For example, each topic included in this training program contains a detailed lesson plan with review exercises, student handouts, and copies of slides used during the instructor's presentation. Each phase in this program is designed to both introduce new material and reinforce material from the previous phase, according to a VBA official. Figure 1: Phases of Training for New VSRs and RVSRs: [See PDF for image] This figure is an illustration of the phases of training for new VSRs and RVSRs, as follows: Phase I: Prerequisite Training; * Objective: Lays foundation for future training; * Location: Home office; * Duration: 2-3 weeks. Phase II: Centralized Training; * Objective: Provides overview of VSR and RVSR claims processing tasks; * Location: A regional office or Veterans Benefits Academy, Baltimore, MD; * Duration: 3 weeks. Phase III: Additional Training; * Objective: Provides hands-on experience and builds on previous training; * Location: Home office; * Duration: 6-12 months. Source: GAO analysis. [End of figure] According to VBA policy, the first phase of training for new VSRs and RVSRs is prerequisite training. New VSRs and RVSRs begin prerequisite training at their home regional office as soon as they begin working. Prerequisite training lays the foundation for future training by introducing new VSRs to topics such as the software applications used to process and track claims, medical terminology, the system for maintaining and filing a case folder, and the process for requesting medical records. Although VBA specifies the topics that must be covered during prerequisite training, regional offices can choose the format for the training and the time frame. New VSRs and RVSRs typically spend 2 to 3 weeks completing prerequisite training in their home office before they begin the second program phase, centralized training. During what is referred to as centralized training, new VSRs and RVSRs spend 3 weeks in intensive classroom training. Participants from multiple regional offices are typically brought together in centralized training sessions, which may occur at their home regional office, another regional office, or the Veterans Benefits Academy in Baltimore, Maryland. According to VBA officials in three of the four offices we visited, bringing together VSRs and RVSRS from different regional offices helps to promote networking opportunities, while VBA officials from two of these offices also stated that it provides a nationwide perspective on VBA. Centralized training provides an overview of the technical aspects of the VSR and RVSR positions. Training instructors should follow the prescribed schedule and curriculum dictating when and how material is taught. For example, for a particular topic, the instructor's guide explains the length of the lesson, the instructional method, and the materials required; lays out the information that must be covered; and provides exercises to review the material. (See fig. 2 for a sample of an instructor's guide from the centralized training curriculum.) Centralized training classes have at least three instructors, but the actual number can vary depending on the size of the group. VBA's goal is to maintain a minimum ratio of instructors to students. Figure 2: Excerpt from the Instructor’s Guide for a Centralized Training Lesson on Reference Materials: [See PDF for image] This figure contains the following excerpts: Excerpt 1: Teaching guide for instructors, page 1: Reference Materials Lesson Plan: Identifying and Accessing VA Laws, Regulations, and Procedural Directives: Prerequisite Training: Prior to this training, the trainee must have completed the Tour of the C&P Website lesson. Purpose of Lesson: Introduce trainees to VSR reference materials (including VA laws, regulation, manuals and directives), as well as the tools for accessing those references (including WARMS and the Internet) so that by the end of the lesson they will be able to: * Competently discuss the importance of developing a good working knowledge of the manuals and CFRs; * Correctly list the main topics found in M21-1, Parts 1 through 7, and the Manual Rewrite; * Properly cite a reference under 38 CFR, M21-1 and M21-1MR; * Demonstrate WARMS system skills, including opening the application, selecting an index, performing a subject search, and retrieving reference documents; * Demonstrate reference retrieval skills using both WARMS and the Publications page on the C&P Intranet to successfully complete the practical exercises in the student handouts. Time required: 8 hours. Instructional Method: Lectures, participatory discussion, instructor demonstration, and group practical exercise. Materials/Training Aids: * Computers with Intranet access for all students; * Reference Materials Trainee Handout; * Reference Materials PowerPoint presentation. Instructor: Print trainee handout (from the website) prior to class. Excerpt 2: Page 9: Except of the Reference Materials Lesson Plan. Source: VBA. [End of figure] The first week of centralized training for VSRs focuses on key concepts, such as security, privacy and records management; terminology; and job tools, such as the policy manual and software applications. The final 2 weeks of training focus on the different roles and responsibilities of VSRs on the Pre-determination and Post- determination teams in processing claims. To practice processing different types of claims and processing claims from start to finish, VSRs work on either real claims or hypothetical claims specifically designed for training. Centralized training for new RVSRs--many of whom have been promoted from the VSR position--focuses on topics such as systems of the human body, how to review medical records, and how to interpret a medical exam. According to staff in one site we visited, RVSRs new to VBA also take VSR centralized training or its equivalent to learn the overall procedures for processing claims. To accommodate the influx of new staff it must train, in fiscal year 2007 VBA substantially increased the frequency of centralized training and is increasing student capacity at the Veterans Benefits Academy. During fiscal year 2007, VBA held 67 centralized training sessions for 1,458 new VSRs and RVSRs. Centralized training sessions were conducted at 26 different regional offices during fiscal year 2007, in addition to the Veterans Benefits Academy. By comparison, during fiscal year 2006, VBA held 27 centralized training sessions for 678 new claims processors. To implement centralized training, VBA relies on qualified regional office staff who have received training on how to be an instructor. According to VBA officials, centralized training instructors may be Senior VSRs, RVSRs, supervisors, or other staff identified by regional office managers as having the capability and the right personality to be effective instructors. Potential instructors have certain training requirements. First, they must complete the week-long Instructor Development Course, which covers the ways different adults learn, the process for developing lesson plans, and the use of different training methods and media. During this course, participants are videotaped and given feedback on their presentation style. In addition, each time instructors teach a centralized training session, they are supposed to take the 2.5 day Challenge Curriculum Course, designed to update instructors on changes to the curriculum and general training issues. Between October 2006 and February 2008, about 250 VSRs and RVSRs from regional offices completed the Instructor Development Course, and VBA officials reported that, given the influx of new VSRs and RVSRs, they are increasing the number of times this course is offered in order to train more instructors. Instructors can teach centralized training sessions in their home office, another regional office, or the Veterans Benefits Academy.[Footnote 5] When new VSRs and RVSRs return to their home office after centralized training, they are required to begin their third phase of training, which is supposed to include on-the-job, classroom, and computer-based training, all conducted by and at their regional office. In the regional offices we visited, managers indicated that new VSRs and RVSRs typically take about 6 to 12 months after they return from centralized training to complete all the training requirements for new staff. During this final phase, new claims processing staff cover more advanced topics, building on what they learned in centralized training. Under the supervision of experienced claims processors, they work on increasingly complex types of real claims. On-the-job training is supplemented in the offices we visited by regular classroom training that follows a required curriculum of courses developed by VBA's Compensation and Pension Service, specifically for new VSRs and RVSRs. For example, new VSRs might complete a class in processing burial claims and then spend time actually processing such claims. The amount of time spent working on each type of claim varies from a couple of days to a few weeks, depending on the complexity of the claim. On-the- job training is also supposed to be supplemented with modules from the Training and Performance Support System (TPSS), an interactive on-line system that can be used by staff individually or in a group.[Footnote 6] TPSS modules provide detailed lessons, practice cases, and tests for VSRs and RVSRs. Modules for new VSRs cover topics such as burial benefits and medical terminology; RVSR modules cover topics such as the musculoskeletal system, general medical terminology, and introduction to post-traumatic stress disorder. New and Experienced Staff Have an Annual Training Requirement, and Regional Offices Develop Training Plans That Cover a Mix of Topics Identified Centrally and Locally: A policy established by VBA's Compensation and Pension Service requires both new and experienced VSRs and RVSRs to complete a minimum of 80 hours of technical training annually, double the number VBA requires of its employees in other technical positions.[Footnote 7] VBA officials said this higher training requirement for VSRs and RVSRs is justified because their jobs are particularly complex and they must work with constantly changing policies and procedures. The 80-hour training requirement has two parts. At least 60 hours must come from a list of core technical training topics identified by the central office of the Compensation and Pension Service. For example, core topics for VSRs in fiscal year 2007 included establishing veteran status and asbestos claims development; topics for RVSRs included due process provisions and eye-vision issues. VBA specifies more core topics than are necessary to meet the 60-hour requirement, so regional offices can choose those topics most relevant to their needs. They can also choose the training method used to address each topic, such as classroom or TPSS training. (See app. II for the list of core technical training topics for fiscal year 2007.) Regional offices determine the training topics that are used to meet the remaining 20 hours, based on local needs and input. Regional offices may select topics from the list of core technical training topics or identify other topics on their own. The four regional offices we visited varied in the extent to which they utilized their discretion to choose topics outside the core technical training topics in fiscal year 2007. Two sites selected the required 60 hours of training from the core requirements and identified their own topics for the remaining 20 hours. In the other two sites, almost all the training provided to staff in fiscal year 2007 was based on topics from the list of core requirements. An official in one regional office, for example, said that his office used its full 20 hours to provide training on new and emerging issues that are not covered by the core technical training topics, as well as training to address error prone areas. An official in another regional office said the core requirements satisfied staff training needs in fiscal year 2007, possibly because this regional office had a large proportion of new staff and the core topics are focused on the needs of new staff. Regional offices must develop training plans each year that indicate which courses will actually be provided to staff to enable them to meet the 80-hour training requirement. The training plan is a list of courses that the regional office plans to offer throughout the year, as well as the expected length and number and types of participants in each course. In the regional offices we visited, when managers develop their training plans, they solicit input from supervisors of VSRs and RVSRs and typically also consider national or local error trend data. Regional offices must submit their plans to the VBA central office at the beginning of each fiscal year for review and feedback. Central office officials review the plans to determine whether (1) the regional office will deliver at least 60 hours of training on the required core topics, (2) the additional topics identified by the regional office are appropriate, and (3) staff in similar positions within an office receive the same level and type of training. According to central office officials, they provide feedback to the regional offices on their current plans as well as guidance on what topics to include in the next year's training plans. Regional offices can adjust their training plans throughout the year to address shifting priorities and unexpected training needs. For example, a regional office may add or remove courses from the plan in response to changing trends in errors or policy changes resulting from legal decisions. (See app. III for excerpts from the fiscal year 2007 training plans from the regional offices we visited.) While regional offices have discretion over the methods they use to provide training, the four offices we visited relied primarily on classroom training in fiscal year 2007. In each of these offices, at least 80 percent of the total fiscal year 2007 training hours completed by all claims processors was in the form of classroom instruction (see fig. 3). Officials in two of the regional offices we visited said they used lesson plans provided by the Compensation and Pension Service and adapted these plans to the needs of their staff; one regional office developed its own courses. An official in one office said they sometimes invite guest speakers, and an official in another regional office said that classroom training is sometimes delivered as part of team meetings. The offices we visited generally made little use of other training methods. Only one office used TPSS for its training more than 1 percent of the time. Two offices used self-instruction--such as reading memos from VBA central office--for about 10 percent of their training, and no office used videos for more than 1 percent of their training. The central office usually communicates immediate policy and regulatory changes through memos called Fast Letters, which may be discussed in team meetings or may just be read by staff individually. Figure 3: Most Fiscal Year 2007 Training Hours Completed by Claims Processors in the Offices We Visited Were in the Form of Classroom Instruction: [See PDF for image} This figure is a stacked vertical bar graph depicting the following data: Office: Atlanta; Classroom training: 99%; Self-instruction: 0; TPSS: 0; Video: 1%. Office: Baltimore; Classroom training: 80%; Self-instruction: 1%; TPSS: 19%; Video: 0. Office: Milwaukee; Classroom training: 91%; Self-instruction: 8%; TPSS: 0; Video: 1%. Office: Portland; Classroom training: 85%; Self-instruction: 14%; TPSS: 1%; Video: 0. Source: VBA central office and regional offices. [End of figure] Staff Are Not Held Accountable for Meeting Their Training Requirement: Because the agency has no policy outlining consequences for individual staff who do not complete their 80 hours of training per year, individual staff are not held accountable for meeting their annual training requirement, and at present, VBA central office lacks the ability to track training completed by individual staff members. According to VBA officials, however, the agency is in the process of implementing an automated system that should allow it to track the training each staff member completes. Officials reported that this system is expected to be implemented during fiscal year 2008. VBA officials reported that this system will be able to record the number of training hours and the courses completed for each individual, staff position, and regional office. One official said the central office and regional office supervisors will have the ability to monitor training completed by individual staff members, but that central office will likely not monitor the training completed by each individual staff member, even though it may monitor the training records for a sample of staff members. Furthermore, despite the absence of a VBA-wide tracking system, managers in two of the regional offices we visited reported using locally developed tracking methods to determine the number of training hours their staff had completed. While individuals are not held accountable, VBA reported taking some steps to ensure that staff complete the required number of training hours. VBA central office periodically reviews the aggregated number of training hours completed at each regional office to determine whether the office is on track to meet the training requirement.[Footnote 8] According to a VBA official, managers in offices where the staff is not on track to complete 80 hours of training during the year can be reprimanded by a higher-level manager, and if their staff do not meet the aggregate training hours at the end of the fiscal year, managers could face negative consequences in their performance assessments. VBA Is Taking Steps to Strategically Plan Its Training for Staff, but Does Not Adequately Evaluate Training and May Be Falling Short in Design and Implementation: VBA is taking steps to strategically plan its training for VSRs and RVSRs including the establishment of a training board to assess VBA's training needs. VBA has also made some effort to evaluate its training for new staff, but does not require regional offices to collect feedback from staff on any of the training they provide. Although some regional offices collect some training feedback, it is not shared with VBA central office. Both new and experienced staff we interviewed did, in fact, report some problems with their training. A number of new staff raised issues with how consistently their training curriculum was implemented. Experienced staff differed in their assessments of the VBA's annual training requirement, with some indicating they struggle to meet this requirement because of workload pressures or that training topics are sometimes redundant or not relevant to their position. VBA Is Taking Steps to Strategically Plan Its Training: VBA is taking steps to strategically plan its training for claims processors, in accordance with generally accepted practices identified by GAO. (See app. I for a detailed description of these generally accepted practices.) Aligning Training with the Agency's Mission and Goals: VBA has made an effort to align training with the agency's mission and goals. According to VBA documents, in fiscal year 2004 an Employee Training and Learning Board (board) was established to ensure that training decisions within the VBA are coordinated; support the agency's strategic and business plans, goals and objectives; and are in accordance with the policy and vision of VBA.[Footnote 9] Some of the board's responsibilities include establishing training priorities and reviewing regional office and annual training plans. Identifying the Skills and Competencies Needed by the Workforce: VBA has identified the skills and competencies needed by VBA's claims processing workforce. VBA developed a decision tree and task analysis of the claims process, which GAO experts in the field of training told us made it possible to understand and map both the claims process and the decisions associated with it that supported the development of VBA's training curriculum. Determining the Appropriate Level of Investment in Training and Prioritizing Funding: VBA is taking steps to determine the appropriate level of investment in training and prioritize funding. According to VBA documents, some of the board's responsibilities include developing annual training budget recommendations and identifying and recommending training initiatives to the Under Secretary of Benefits. VBA officials also reported developing several documents that made a business case for different aspects of VBA's training, such as VA's annual budget and the task analysis of the VSR and RVSR job positions. Considering Government Reforms and Initiatives: According to one VBA official, the agency identifies regulatory, statutory, and administrative changes as well as any legal or judicial decisions that affect how VBA does business and issues guidance letters, or Fast Letters, which can be sent out several times a year, to notify regional offices of these changes. Also, as a result of Congress authorizing an increase in its number of full-time employees and VBA's succession planning efforts, VBA has increased the number of centralized training sessions for new staff and has also increased the number of Instructor Development Courses offered to potential centralized training instructors. As a result, VBA is taking steps to consider government reforms and initiatives to improve its management and performance when planning its training. VBA Collects Feedback on Centralized Training, but Regional Offices Do not Always Collect Feedback on the Training They Provide: According to accepted practices, federal agencies should also evaluate their training programs and demonstrate how these efforts help employees, rather than just focusing on activities or processes (such as number of training participants or hours of training). VBA has made some efforts to evaluate its training for claims processors. During the 3-week centralized training session for new staff, VBA solicits daily feedback from participants using forms that experts in the training field consider well-constructed and well-balanced. According to one GAO expert, the forms generally employ the correct principles to determine the effectiveness of the training and ascertain whether the instructor effectively presented the material (see fig. 4). VBA officials told us that they have used this feedback to improve centralized training for new staff. Management at one regional office cited the decision to separate training curricula for VSRs on Pre-determination teams and VSRs on Post-determination teams as an example of a change based on this feedback. Figure 4: Sample of VBA’s Centralized Training Evaluation Form: [See PDF for image] This figure is a sample of VBA’s Centralized Training Evaluation Form. as follows: Baltimore Classroom 1 Centralized Training Topic Evaluation - 2008 Session 4: * Please select your training topic from the list of options: 1. Rate the overall quality of the content: Excellent: Good: Fair: Poor: 2. Rate the overall effectiveness of the practical exercises: Excellent: Good: Fair: Poor: 3. Rate the overall effectiveness of the instructor(s): Excellent: Good: Fair: Poor: 4. How was the pace of the instruction? About right: Too slow: Too fast: Much too slow: Much too fast: 5. What would have made this training more useful? Please explain: 6. Please include any additional narrative comments about the training/instructor(s): Submit: Source: VBA. [End of figure] Although VBA evaluates centralized training, it does not require regional offices to obtain feedback from participants on any of the training they provide to new and experienced staff. In a previous GAO report, VA staff told us that new training materials they develop are evaluated before being implemented.[Footnote 10] However, none of the regional offices we visited consistently collect feedback on the training they conduct. Supervisors from three of the regional offices we visited told us that they collect feedback on some of the training their office conducts, but this feedback largely concerns the performance of the instructor. Participants are generally not asked for feedback on course content. Moreover, regional offices we visited that do, to some degree, collect feedback do not share this information with VBA. VBA's Training Curriculum for New Staff Appears Generally Well Designed, but Some Staff Raised Issues Concerning Its Implementation: According to GAO experts in the training field, VBA's training curriculum for new staff appears well designed. VBA's curriculum for new staff conforms to adult learning principles, carefully defining all pertinent terms and concepts, and providing abundant and realistic examples of claims work. GAO experts also determined that VBA's training for those who teach the curriculum for new staff was well designed and would enable experienced claims processors to become competent trainers because they are coached on teaching theory and have multiple opportunities to practice their teaching skills and receive feedback. Many of the new staff at all four sites we visited reported that centralized training provided them with a good foundation of knowledge and prepared them for additional training conducted by their regional office. Also, regional office managers from three offices we visited told us that centralized training affords new staff the opportunity to network with other new staff at different regional offices, which imbues a sense of how their positions fit in the organization. However, some staff reported that VBA's implementation of their centralized training was not always consistent. A number of staff at three regional offices reported that during their centralized training the instructors sometimes taught different ways of performing the same procedures or disagreed on claim procedures. Regional office officials told us that while centralized training instructors attempt to teach consistently through the use of standardized training materials, certain procedures can be done differently in different regional offices while adhering to VBA policy. For example, regional offices may differ on what to include in veteran notification letters. VBA officials also told us that centralized training conducted at the regional offices may not be as consistent as centralized training conducted at the Veterans Benefits Academy. According to these officials, unlike the regional offices, the Veterans Benefits Academy has on-site training experts to guide and ensure that instructors are teaching the curriculum consistently. New staff also gave mixed assessments about how training was conducted at their home office after they returned from centralized training. While some staff at all of the regional offices we visited told us that the additional training better prepared them to perform their jobs, with on-the-job training identified as a useful learning tool, others told us that the training could not always be completed in a timely manner due to regional office priorities. Some management and staff at two of the regional offices we visited reported that, because of workload pressures, some of their RVSRs had to interrupt their training to perform VSR duties. Also, a few new staff indicated that VBA's TPSS was somewhat difficult to use.[Footnote 11] Although TPSS was developed to provide consistent technical training designed to improve the accuracy of claims ratings, a number of staff at all of the regional offices we visited reported that TPSS was too theoretical. For example, some staff said it provided too much information and no practical exercises in applying the knowledge. Some staff also noted that certain material in TPSS was out-of-date with policy changes such as how to order medical examinations. Some staff at three of the regional offices also reported that TPSS was not always useful in training staff, in part, because TPSS does not use real cases. Three of the regional offices reported using TPSS for less than 1 percent of their training and VSRs at one regional office were unaware of what TPSS was. Experienced Staff Expressed Mixed Views of the Design and Implementation of Their Training: At all of the regional offices we visited, staff we spoke with generally noted that training enables them to keep up-to-date on changes in laws and regulations as well as provides opportunities for obtaining refresher training on claims procedures they perform infrequently. However, regional office staff we spoke with differed in their assessment of the 80-hour requirement. Some regional office staff said the number of training hours required was appropriate, while others suggested that VBA adopt a graduated approach, with the most experienced staff being required to complete fewer hours than new staff. VBA officials told us that, in 2007, the Compensation and Pension Service reviewed their annual training requirements and determined the 80-hour annual training requirement was appropriate. However, the officials we spoke with could not identify the criteria that were used to make these determinations. Furthermore, VBA management does not systematically collect feedback from staff evaluating the usefulness of the training they must receive to meet this requirement. Consequently, when determining the appropriateness of the 80-hour requirement, VBA has not taken into account the views of staff to gauge the effect the requirement has on them. Experienced staff had mixed views on training provided by the regional office. Staff at three regional offices said the core technical training topics set by the Compensation and Pension Service are really designed for newer staff and do not change much from year to year, and therefore experienced staff end up repeating courses. Also, a number of staff at all of the regional offices we visited told us some regional office training was not relevant for those with more experience. Conversely, other regional office staff note that although training topics may be the same from year to year, a person can learn something new each time the course is covered. Some VBA officials and regional office managers also noted that some repetition of courses is good for several reasons. Staff may not see a particular issue very often in their day-to-day work and can benefit from refreshers. Also, regional office managers at one office told us that the core technical training topics could be modified to reflect changes in policy so that courses are less repetitive for experienced staff. Many experienced staff also reported having difficulty meeting the 80- hour annual training requirement due to workload pressures. Many of the experienced staff we spoke with, at each of the regional offices we visited, told us that there is a constant struggle between office production goals and training goals. For example, office production goals can affect the availability of the regional office's instructors. A number of staff from one regional office noted that instructors were unable to spend time teaching because of their heavy workloads and because instructors' training preparation hours do not count toward the 80-hour training requirement. Staff at another regional office told us that, due to workload pressures, staff may rush through training and may not get as much out of it as they should. Performance Management System for Claims Processors Generally Conforms to Accepted Practices, but May Not Clearly Differentiate between Performance Levels: VA's performance management system for claims processors is consistent with several accepted practices for effective performance management systems in the public sector, but may not clearly differentiate between staff's overall performance levels. VA's performance management system aligns individual performance elements with broader organizational performance measures, provides performance feedback to staff throughout the year, and emphasizes collaboration. However, the system may not clearly differentiate VSRs' and RVSRs' varying levels of performance. While the system has five summary rating categories for VSRs and RVSRs, several VBA managers told us that, because of a problem with the formula used to convert ratings on individual performance elements into an overall performance category, it is more difficult for staff to be placed in certain categories than others: Performance Management System for Claims Processors Is Generally Consistent with Accepted Practices: The elements used to evaluate individual VSRs' and RVSRs' performance appear to be generally aligned with VBA's organizational performance measures, something prior GAO work has identified as a well-recognized practice for effective performance management systems (see app. I). Aligning individual and organizational performance measures helps staff see the connection between their daily work activities and their organization's goals and the importance of their roles and responsibilities in helping to achieve these goals. VSRs must be evaluated on four critical elements: quality, productivity, workload management, and customer service. RVSRs are evaluated on quality, productivity, and customer service. In addition, VBA central office requires regional offices to evaluate their staff on at least one non- critical element. The central office has provided a non-critical element called cooperation and organizational support, and although regional offices are not required to use this particular element, all four offices we visited did so (see table 2). For each element, there are three defined levels of performance: exceptional, fully successful, or less than fully successful.[Footnote 12] Table 2 refers only to the fully successful level of performance for each element. Table 2: Individual Performance Elements for VSRs and RVSRs: Performance element: Critical: Quality; How performance is evaluated: A random selection of 5 cases or phone calls per month is reviewed for accuracy based on certain criteria, for example whether all necessary evidence was requested, proper notifications were sent to applicants, and accurate information was provided in phone calls. Any case or phone call with one or more errors is counted as one inaccurate case or call; Standard for minimum fully successful performance (journey-level VSR): 85% accuracy; Standard for minimum fully successful performance (journey-level RVSR): 85% accuracy. Performance element: Critical: Productivity; How performance is evaluated: Number of weighted actions (VSRs) or weighted cases (RVSRs) completed per day. VSRs receive different weights for different actions, such as 0.125 for conducting a telephone interview or 1.50 for developing the evidence for a claim with a special issue such as radiation. RVSRs receive different levels of credit for processing cases with different numbers of issues to be evaluated; Standard for minimum fully successful performance (journey-level VSR): 8 weighted actions per day[A]; Standard for minimum fully successful performance (journey-level RVSR): 3.5 weighted cases per day[B]. Performance element: Critical: Customer service; How performance is evaluated: Number of valid complaints about employee's behavior from external customers or internal colleagues; Standard for minimum fully successful performance (journey-level VSR): No more than 3 valid complaints or incidents. Performance element: Critical: Workload management; How performance is evaluated: Completion of designated tasks in a timely manner, such as obtaining the results of a medical exam within a specified period of time; Standard for minimum fully successful performance (journey-level VSR): Tasks are completed in timely manner 85 percent of the time; Standard for minimum fully successful performance (journey-level RVSR): Not applicable. Performance element: Non-critical[C]: Cooperation and organizational support; How performance is evaluated: Understanding of agency goals, interaction with colleagues, contribution to agency goals; Standard for minimum fully successful performance (journey-level VSR): Interacts with colleagues professionally. Follows directions and adheres to guidance conscientiously. Adjusts easily to different working styles and perspectives. Source: GAO analysis of VBA information. Note: This table includes the levels set for journey-level VSRs and RVSRs, who are considered experienced and fully trained in their positions. For some elements VBA sets different performance standards for entry-level and experienced claims processors. For example, VSRs are typically promoted to the journey-level position after about 2 years. VBA has separate, lower performance standards in the accuracy, productivity, and workload management elements for VSRs who are not yet at the journey level. Also, regional offices have the option of setting fully successful levels for their staff that are higher than the national minimum, but not lower. This table indicates instances when the regional offices we visited have set thresholds that are higher than the national minimum. [A] Milwaukee has set a fully successful level of 10 weighted actions per day. [B] Baltimore, Milwaukee, and Portland have set fully successful levels of, respectively, 4, 5, and 3.8 weighted cases per day. [C] Regional offices are required to use at least one non-critical element. VBA central office provided regional offices with the cooperation and organizational support element, but regional offices are not required to use this element in particular. [End of table] Three critical elements in particular--quality, workload management, and productivity--are aligned with VBA's organizational performance measures (see table 3). According to VA's strategic plan, one key organizational performance measure for VBA is overall accuracy in rating disability claims. This organizational measure is aligned with the quality element for VSRs and RVSRs, which is assessed by measuring the accuracy of their claims-processing work. An individual performance element designed to motivate staff to process claims accurately should, in turn, help VBA meet its overall accuracy goal. Two other key performance measures for VBA are the average number of days that open disability claims have been pending and the average number of days it takes to process disability claims. VSRs are evaluated on their workload management, a measure of whether they complete designated claims-related tasks within specific deadlines. Individual staff performance in this element is linked to the agency's ability to manage its claims workload and process claims within goal time frames. Finally, a performance measure that VBA uses to evaluate the claims- processing divisions within its regional offices--and that, according to VBA, relates to the organization's overall mission--is production, or the number of compensation and pension claims processed by each office in a given time period. Individual VSRs and RVSRs are evaluated on their productivity, i.e., the number of claims-related tasks they complete per day. Higher productivity by individual staff should result in more claims being processed by each regional office and by VBA overall. Table 3: Performance Elements for VSRs and RVSRs and Corresponding Organizational Performance Measures for VBA: Performance element for VSRs and RVSRs: Quality; Corresponding VBA performance measure(s): Accuracy rate for ratings of compensation claims. Performance element for VSRs and RVSRs: Productivity; Corresponding VBA performance measure(s): Number of compensation and pension claims completed by the claims-processing division within a regional office in a given time period. Performance element for VSRs and RVSRs: Workload management[A]; Corresponding VBA performance measure(s): Average days pending for compensation and pension claims; (average number of days since claim was received by VBA, for all open claims); Average days to process compensation and pension claims (average number of days from receipt of claim to final decision). Source: VBA and GAO analysis. [A] Workload management element applies only to VSRs, not RVSRs. [End of table] The performance management system for VSRs and RVSRs also appears to be consistent with several other accepted practices for performance management systems in the public sector: Providing and Routinely Using Performance Information to Track Organizational Priorities: Providing objective performance information to individuals helps show progress in achieving organizational goals and allows individuals to manage their performance during the year by identifying performance gaps and improvement opportunities. Regional offices are supposed to use the critical and non-critical performance elements to evaluate and provide feedback to their staff. Supervisors are required to provide at least one progress review to their VSRs and RVSRs each year, indicating how their performance on each element compares to the defined standards for fully successful performance. In the offices we visited, supervisors typically provide some feedback to staff on a monthly basis. For example, VSRs in the Atlanta regional office receive a memo on their performance each month showing their production in terms of average weighted actions per day, their accuracy percentage based on a review of a sample of cases, and how their performance compared to the minimum requirements for production and accuracy. If staff members fall below the fully successful level in a critical element at any time during the year, a performance improvement plan must be implemented to help the staff member improve. Connecting Performance Expectations to Crosscutting Goals: Performance elements related to collaboration or teamwork can help reinforce behaviors and actions that support crosscutting goals and provide a consistent message to all employees about how they are expected to achieve results. VSR and RVSR performance related to customer service is evaluated partly based on whether any valid complaints have been received about a staff member's interaction with their colleagues. And performance related to the cooperation and organizational support element is based on whether staff members' interaction with their colleagues is professional and constructive. Using Competencies to Provide a Fuller Assessment of Performance: Competencies, which define the skills and supporting behaviors that individuals are expected to exhibit to carry out their work effectively, can provide a fuller assessment of an individual's performance. In addition to elements that are evaluated in purely quantitative terms, VBA uses a cooperation and organizational support element for VSRs and RVSRs that requires supervisors to assess whether their staff are exhibiting a number of behaviors related to performing well as a claims processor. Involving Employees and Stakeholders to Gain Ownership of the Performance Management System: Actively involving employees and stakeholders in developing the performance management system and providing ongoing training on the system helps increase their understanding and ownership of the organizational goals and objectives. For example, VA worked with the union representing claims processors to develop an agreement about its basic policies regarding performance management. Also, VBA indicated that it planned to pilot revisions to how productivity is measured for VSRs in a few regional offices, partly so VSRs would have a chance to provide feedback on the changes. VA's System May Not Clearly Differentiate between Performance Levels: Clear differentiation between staff performance levels is also an accepted practice for effective performance management systems. Systems that do not result in meaningful distinctions between different levels of performance fail to give (1) employees the constructive feedback they need to improve, and (2) managers the information they need to reward top performers and address performance issues. GAO has previously reported that, in order to provide meaningful distinctions in performance for experienced staff, agencies should use performance rating scales with at least three levels, and scales with four or five levels are preferable because they allow for even greater differentiation between performance levels.[Footnote 13] If staff members are concentrated in just one or two of multiple performance levels, however, the system may not be making meaningful distinctions in performance. VA's performance appraisal system has the potential to clearly differentiate between staff performance levels. Each fiscal year, regional offices give their staff a rating on each critical and non- critical performance element using a three-point scale--exceptional, fully successful, or less than fully successful. Based on a VA-wide formula, the combination of ratings across these elements is converted into one of VA's five overall performance levels: outstanding, excellent, fully successful, minimally satisfactory, and unsatisfactory (see fig. 5). Regional offices may award financial bonuses to staff on the basis of their end-of-year performance category.[Footnote 14] Prior to fiscal year 2006, VA used two performance levels--successful and unacceptable--to characterize each staff member's overall performance. To better differentiate between the overall performance levels of staff, VA abandoned this pass-fail system in that year, choosing instead to use a five-level scale. Figure 5: VA Overall Performance Appraisal Formula: [See PDF for image] This figure is an illustration of the VA Overall Performance Appraisal Formula. The following information is depicted: Ratings on Critical Elements: All exceptionals; and; Ratings on Non-Critical elements: All exceptionals; yields: Overall Performance Category: Outstanding. Ratings on Critical Elements: All exceptionals; and; Ratings on Non-Critical elements: One or more fully successful(s), any others exceptionals; yields: Overall Performance Category: Excellent. Ratings on Critical Elements: One or more fully successful(s), all others exceptionals; and; Ratings on Non-Critical elements: All fully successfuls or exceptionals; yields: Overall Performance Category: Fully Successful. Ratings on Critical Elements: One or more less than fully successful(s); yields: Overall Performance Category: Unsatisfactory. Source: GAO analysis of VBA information. [End of figure] However, there is evidence to suggest that the performance management system for VSRs and RVSRs may not clearly or accurately differentiate among staff's performance. VBA central office officials and managers in two of the four regional offices we visited raised concerns with VA's formula for translating ratings on individual performance elements into an overall performance rating.[Footnote 15] These officials said that under this formula it is more difficult for staff to be placed in certain overall performance categories than others, even if staff's performance truly does fall within one of those categories. Indeed, at least 90 percent of all claims processors in the regional offices we visited were placed in either the outstanding or the fully successful category in fiscal year 2007. (Fig. 6 shows the distribution of overall performance ratings for claims processors in each office.) [Footnote 16] Figure 6: Fiscal Year 2007 Overall Performance Ratings for Claims Processors in Four Regional Offices Were Concentrated in the Outstanding and Fully Successful Categories: [See PDF for image] This figure is a stacked vertical bar graph depicting the following data: Office: Atlanta; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 81%; Excellent: 10%; Outstanding: 9%. Office: Baltimore; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 66%; Excellent: 10%; Outstanding: 24%. Office: Milwaukee; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 70%; Excellent: 2%; Outstanding: 28%. Office: Portland; Unsatisfactory: 0; Minimally satisfactory: 0; Fully successful: 72%; Excellent: 2%; Outstanding: 26%. Source: VBA regional offices. Note: These data cover VSRs, RVSRs, and some other claims processing staff. [End of figure] Central and regional office managers noted that, in particular, it is difficult for staff to receive an overall rating of excellent. Managers in one office said there are staff whose performance is better than fully successful but not quite outstanding, but under the formula it is difficult for these staff to be placed in the excellent category as the managers feel they should be. An excellent rating requires exceptional ratings in all the critical elements and a fully successful rating in at least one non-critical element. However, according to staff we interviewed, virtually all staff who are exceptional in the critical elements are also exceptional in all non-critical element(s), so they appropriately end up in the outstanding category. On the other hand, the overall rating for staff who receive a fully successful rating on just one of the critical elements--even if they are rated exceptional in all the other elements--drops down to fully successful. Managers in one regional office commented that the system would produce more accurate overall performance ratings if staff were given an overall rating of excellent when they had, for example, exceptional ratings on three of five overall elements and fully successful ratings on the other two. An official in VA's Office of Human Resources Management acknowledged that there may be an issue with the agency's formula. Although neither VBA nor VA central office officials have examined the distribution of VSRs and RVSRs across the five overall performance ratings, VA indicated it is considering changes to the system designed to allow for greater differentiation in performance ratings. For example, one possible change would be to use a five-point scale for rating individual elements--probably mirroring the five overall performance rating categories of outstanding, excellent, fully successful, minimally satisfactory, and unsatisfactory--rather than the current three-point scale. Under the proposed change, a staff member who was generally performing at the excellent but not outstanding level could get excellent ratings in all the elements and receive an overall rating of excellent. This change must still be negotiated with several stakeholder groups, according to the VA official we interviewed. Conclusions: In many ways, VBA has developed a training program for its new staff that is consistent with accepted training practices in the federal government. However, because VBA does not centrally evaluate or collect feedback on training provided by its regional offices, it lacks the information needed to determine if training provided at regional offices is useful and what improvements, if any, may be needed. Ultimately, this information would help VBA determine if 80 hours of training annually is the right amount, particularly for its experienced staff, and whether experienced staff members are receiving training that is relevant for their positions. Identifying the right amount of training is crucial for the agency as it tries to address its claims backlog. An overly burdensome training requirement needlessly may take staff away from claims processing, while too little training could contribute to processing inaccuracies. Also, without collecting feedback on regional office training, VBA may not be aware of issues with the implementation of its TPSS, the on-line training tool designed to ensure consistency across offices in technical training. Setting aside the issue of how many hours of training should be required, VBA does not hold its staff accountable for fulfilling their training requirement. As a result, VBA is missing an opportunity to clearly convey to staff the importance of managing their time to meet training requirements as well as production and accuracy goals. With the implementation of its new learning management system, VBA should soon have the ability to track training completed by individual staff members, making it possible to hold them accountable for meeting the training requirement. As with its training program for VSRs and RVSRs, the VA is not examining the performance management system for claims processors as closely as it should. VBA is generally using the right elements to evaluate its claims processors' performance, and the performance appraisals have the potential to give managers information they can use to recognize and reward higher levels of performance. However, evidence suggests the formula used to place VSRs and RVSRs into overall performance categories may not clearly and accurately differentiate among staff's performance levels. Absent additional examination of the distribution of claims processors among overall performance categories, VA lacks a clear picture of whether its system is working as intended and whether any adjustments are needed. Recommendations for Executive Action: The Secretary of Veterans Affairs should direct VBA to: * Collect and review feedback from staff on the training conducted at the regional offices to determine: * if the 80-hour annual training requirement is appropriate for all VSRs and RVSRs; * the extent to which regional offices provide training that is relevant to VSRs' and RVSRs' work, given varying levels of staff experience; and: * whether regional offices find the TPSS a useful learning tool and, if not, what adjustments are needed to make it more useful; and: * Use information from its new learning management system to hold individual VSRs and RVSRs accountable for completing whatever annual training requirement it determines is appropriate. The Secretary of Veterans Affairs should also examine the distribution of claims processing staff across overall performance categories to determine if its performance appraisal system clearly differentiates between overall performance levels, and if necessary adjust its system to ensure that it makes clear distinctions. Agency Comments: We provided a draft of this report to the Secretary of Veterans Affairs for review and comment. In VA's written comments (see app. IV), the agency agreed with our conclusions and concurred with our recommendations. For example, VBA plans to consult with regional office staff to evaluate its annual 80-hour training requirement and will examine if staff performance ratings clearly differentiate between overall performance levels. VA also provided technical comments that were incorporated as appropriate. We are sending copies of this report to the Secretary of Veterans Affairs, relevant congressional committees, and others who are interested. We will also provide copies to others on request. The report is also available at no charge on GAO's Web site at [hyperlink http://www.gao.gov]. Please contact me on (202) 512-7215 if you or your staff have any questions about this report. Contact points for the Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors are listed in appendix V. Sincerely, Singed by: Daniel Bertoni: Director, Education, Workforce, and Income Security Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: We were asked to determine: (1) What training is provided to new and experienced claims processors and how uniform is this training? (2) To what extent has the Veterans Benefits Administration (VBA) developed a strategic approach to planning training for claims processors and how well is their training designed, implemented, and evaluated? And (3) To what extent is the performance management system for claims processors consistent with generally accepted performance management practices in the public sector? To answer these questions, we reviewed documents and data from the central office of the Department of Veterans Affairs' Veterans Benefits Administration (VBA) and interviewed VBA central office officials. We conducted site visits to and collected data from four VBA regional offices, and visited the Veterans Benefits Academy. We also interviewed officials from the American Federation of Government Employees, the labor union that represents Veterans Service Representatives (VSR) and Rating Veterans Service Representatives (RVSR). We compared VBA's training and performance management systems to accepted human capital principles and criteria compiled by GAO. We conducted this performance audit from September 2007 through May 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Regional Office Site Visits: We conducted site visits to 4 of VBA's 57 regional offices--Atlanta; Baltimore; Milwaukee; and Portland, Oregon. We judgmentally selected these offices to achieve some diversity in geographic location, number of staff, and claims processing accuracy rates, and what we report about these sites may not necessarily be representative of any other regional offices or all regional offices (see fig. 7). [Footnote 17] Figure 7: Regional Offices Selected for Site Visits: [See PDF for image] This figure is a map of the United States depicting the regional offices selected for site visits. The following information is represented: Eastern Area: Location: Baltimore, Maryland; Number of staff[A]: 122.5; rank[B]: 37th; Accuracy rate[C]: 88%; rank[B]: 32nd. Southern Area: Location: Atlanta, Georgia; Number of staff[A]: 590.5; rank[B]: 3rd; Accuracy rate[C]: 84%; rank[B]: 48th. Central Area: Location: Milwaukee, Wisconsin; Number of staff[A]: 362.9; rank[B]: 12th; Accuracy rate[C]: 92%; rank[B]: 13th. Western Area: Location: Portland, Oregon; Number of staff[A]: 156.1; rank[B]: 31st; Accuracy rate[C]: 86%; rank[B]: 44th. Sources: VBA; Map Resources (map). [A] Full-time equivalents as of September 2007. [B] Rank among all 57 regional offices. [C] Claims-processing accuracy rate for the period of August 1, 2006 to July 31, 2007. [End of figure] During our site visits, we interviewed regional office managers, supervisors of VSRs and RVSRs, VSRs, and RVSRs about the training and performance management practices in their offices. The VSRs and RVSRs we interviewed at the four regional offices had varying levels of experience at VBA. Regional office managers selected the staff we interviewed. We also observed a demonstration of VBA's on-line learning tool, the Training and Performance Support System (TPSS), and collected data from the regional offices on, for example, the training they provided during fiscal year 2007.[Footnote 18] In conjunction with our visit to the Baltimore regional office, we also visited VBA's Veterans Benefits Academy, where we observed classes for VSRs and RVSRs and interviewed the director of the Academy. Assessment of VBA's Training for Claims Processors: To determine whether VBA's training program is consistent with accepted training practices in the public sector, we relied partly on a guide developed by GAO that lays out principles that federal agencies should follow to ensure their training is effective.[Footnote 19] This guide was developed in collaboration with government officials and experts in the private sector, academia, and nonprofit organizations; and in conjunction with a review of laws, regulations and literature on training and development issues, including previous GAO reports. The guide lays out the four broad components of the training and development process (see fig. 8). Figure 8: Four Components of the Training and Development Process: [See PDF for image] This figure is an illustration of the four components of the training and development process. The first three components are sequential, but all three provide information the the final component. The following information is depicted: Planning/Front-end Analysis: * Develop a strategic approach that establishes priorities and leverages investments in training and development to achieve agency results. Design/Development: * Identify specific training and development initiatives that, in conjunction with other strategies, improve individual and agency performance. Implementation: * Ensure effective and efficient delivery of training and development opportunities in an environment that supports learning and change. Evaluation: * Demonstrate how training and development efforts contribute to improved performance and results. Source: GAO. Note: The evaluation component may include the use of participant feedback to ensure continuous improvement, as well as an assessment of the impact of training on organizational performance. We have reported that higher-level evaluations that attempt to measure the return on investment in a training program may not always be appropriate, given the complexity and costs associated with efforts to directly link training programs to improved individual and organizational performance. [End of figure] The guide also provides key questions for federal agencies to consider in assessing their performance in each component. (See table 4 for a sample of these questions.) Table 4: Selected Key Questions to Consider in Assessing Agency’s Training Program: Planning/Front End Analysis: * Does the agency have training goals and related performance measures that are consistent with its overall mission, goals, and culture? * How does the agency identify the appropriate investment to provide for training and development efforts and prioritize funding so that the most important training needs are addressed first? Design and Development; * What criteria does the agency use in determining whether to design training and development programs in-house or obtain these services from a contractor or other external source? * Does the agency use the most appropriate mix of centralized and decentralized approaches for its training and development programs? Implementation; * What steps do agency leaders take to communicate the importance of training and developing employees, and their expectations for training and development programs to achieve results? * How does the agency select employees to participate in training and development efforts? Evaluation; * To what extent does the agency systematically plan for and evaluate the effectiveness of its training and development efforts? * How does the agency incorporate evaluation feedback into the planning, design, and implementation of its training and development efforts? Source: GAO. [End of table] In addition, GAO training experts reviewed VBA materials, including training curricula, lesson plans, and course evaluation forms, to determine if these materials are consistent with accepted training practices. Assessment of VBA's Performance Management System for Claims Processors: In assessing the performance management system for VSRs and RVSRs, we relied primarily on a set of accepted practices of effective public sector performance management systems that has been compiled by GAO.[Footnote 20] To identify these accepted practices, GAO reviewed its prior reports on performance management that drew on the experiences of public sector organizations both in the United States and abroad. For the purpose of this review, we focused on the six accepted practices most relevant for VBA's claims-processing workforce (see table 5). Table 5: Selected Accepted Practices for Effective Performance Management Systems: Practice: Aligning individual performance expectations with organizational goals; Description: Explicitly aligning individuals' daily activities with broader results helps individuals see the connection between their work and organizational goals and encourages individuals to focus on their roles and responsibilities to help achieve those broader goals. Practice: Connecting performance expectations to crosscutting goals; Description: Fostering collaboration, interaction, and teamwork across organizational boundaries to achieve results strengthens accountability for these results. Practice: Providing and routinely using performance information to track organizational priorities; Description: Providing objective performance information to both managers and staff to show progress in achieving organizational results and other priorities helps them manage during the year, identify performance gaps, and pinpoint improvement opportunities. Practice: Using competencies to provide a fuller assessment of performance; Description: Using competencies, which define the skills and supporting behaviors that individuals need to effectively contribute to organizational results, can provide a fuller picture of an individual's performance. Practice: Making meaningful distinctions in performance; Description: Providing individuals with candid and constructive feedback helps them maximize their contribution, and providing management with objective and fact-based information that clearly differentiates between different levels of performance enables it to reward top performers and deal with poor performers. Practice: Involving employees and stakeholders to gain ownership of performance management systems; Description: Actively involving employees and stakeholders in developing the performance management system and providing ongoing training on the system helps increase their understanding and ownership of the organizational goals and objectives. Source: GAO. [End of table] [End of section] Appendix II: Fiscal Year 2007 Core Technical Training Requirements for VSRs and RVSRs: Position: Decision Review Officers (DRO); GS13/Rating Veterans Service Representatives (RVSR); GS7-12; (Seasoned); Required: 80 hours; Any DRO or RVSR who conducts a training session will also be given credit for those training hours as part of their training requirement; Course title or topic: Effective Dates; Diabetes Mellitus; Leishmaniasis; Original Compensation Ratings; Original Pension Ratings; Original DIC Ratings; Rating re-opened claims; Claims for Increase; New and Material Evidence; Re-opened DIC ratings; Routine Future Exams; 3.105(e) reductions; Paragraph 28/29/30 ratings; Due Process Provisions; Clear and unmistakable errors (3.105(a)); Ancillary Benefits; Accrued Ratings; Musculoskeletal issues; Eye-Vision Issues; Infectious Diseases; Ear-Hearing; Respiratory Disorders; Cardiovascular Issues; Digestive Issues; Genitourinary System; Gynecology; Hemic/Lymphatic; Endocrine (other than DM); Neurological; Mental Disorder (other than PTSD); PTSD; Special Monthly Compensation (SMC); The Appeals Process; Responsibilities of a DRO; Hearings; Informal Conferences; Resolution of Claims; Certifying a case to BVA; Processing Remands; Preparing a Statement of the Case (SOC); Preparing a Supplemental Statement of the Case (SSOC); Role of the Rating Specialist; Benefit of the Doubt; Weighing Evidence; 60 Hours of the required 80 Hours will be selected from the suggested topics above. The remaining 20 hours will be selected at the Stations discretion based upon their own individual quality review; (Training provided from the above topics can be focused on a particular aspect of the topic; i.e. Cold Injuries and Rating Hypertension from Cardiovascular issues could be separate classes); Participation in Agency Advancement Programs (i.e., LEAD, LVA) does not substitute for Required training requirements; Training source: C&P Training Website [hyperlink, http://cptraining.vba.va.gov/C&P_Training/RVSR/RVSR_Tng_Curr.htm]; Fast Letters' Training Letters; Court Decisions; TPSS (Can be used as refresher training for seasoned employees by module); EPSS; Manuals; Regulations; Additional Issue Specific Lesson Plans are under development; (Lesson plans can be taken from the Centralized Training Curriculum found on the C&P Intranet Training Site. If used as provided they do not require C&P review and approval); These plans can and often should be modified to focus in on a particular narrow issue of training need. Modified lesson plans are to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training; Any Challenge-oriented original lesson plan developed by Station personnel is to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training.) C&P Service Broadcasts that may be provided during the course of the FY may be substituted in place of any training scheduled on an hour by hour basis. Position: Veteran Service Representative (VSR); GS 7-12 (Seasoned); Required: 80 hours; Any Super Senior VSR, Senior VSR, or VSR, who conducts a training session will also be given credit for those training hours including preparation time as part of their training requirement; Course title or topic: Reference Materials: Manual Training & WARMS; C&P Website; Claims Folder Maintenance; Records Management; POA/Service Orgs.; Original Compensation Claims; Re-opened Compensation Claims; VA Form 21-526; Establishing Veteran Status; Claims Recognition; Duty to Assist; Requesting VA Exams; Issue Specific Claims Development; Asbestos Claims Development; Herbicide Claims Development; POW Claims Development; Radiation Claims Development; PTSD Claims Development; Undiagnosed Illness Claims Development; Dependency Issues; Contested Claims; Deemed Valid and Common Law Marriages; Continuous Cohabitation; Pension; SHARE; COVERS; MAP D; MAP A; Administrative Decisions; Character of Discharge; Line of Duty-Willful Misconduct; Matching Programs; Workload Management; DEA Training; Intro to Ratings; Paragraph 29 & 30 Ratings; Ratings & BDN; BDN 301 Interface; PCGL Award Letters; Dependents and the BDN; Compensation Offsets; Drill Pay Waivers; Pension Awards Processing & BDN; Hospital Reductions; Burial Benefits; Death Pension; Accrued Benefits; Accrued Awards & the BDN; Apportionments; Special Monthly Pension; Helpless Child; Incompetency/Fiduciary Arrangements; Claims Processing; Auto Allowance and Adaptive Equipment; Special Adapted Housing; Special Home Adaptation Grants; Incarcerated Veterans; Processing Write Outs; FOIA/Privacy Act; Telephone & Interview Techniques; Telephone Development; IRIS; Introduction to VACOLS; Education Benefits; Insurance Benefits; National Cemetery; VR&E Benefits; Loan Guaranty Benefits; General Benefits - FAQs; Suicidal Caller Guidance; Non-Receipt of BDN Payments; Mail Handling; Income & Net Worth Determinations; Bootcamp test and review of VSR; Readiness Guide (2 HRS Required); Reference Material Training and Navigation (1 HR Required); Appeals and Ancillary Benefits; Ready to Rate Development; Customer Service; FNOD Info and PMC Process; Intro to Appeals Process; DRO Selection Letter; Income Adjustment Materials; Income Adjustments; 60 Hours of the required 80 Hours will be selected from the suggested topics above. The remaining 20 hours will be selected at the Stations discretion based upon their own individual quality review; Training source: C&P Training Website [hyperlink, http://cptraining.vba.va.gov/C&P_Training/VSR/VSR_Curriculum.htm] or, [hyperlink, http://cptraining.vba.va.gov/C&P_Training/VSR/VSR_Curriculum.htm#pctt]; Fast Letters; Training Letters; Court Decisions; TPSS (Can be used as refresher training for seasoned employees by module); EPSS; Manuals; Regulations; Federal Benefits for Veterans and Dependents; Business Line Internet Sites; Conference Calls (VACO/C&P Service); Star Reporter; Additional Issue Specific Lesson Plans are under development; (Lesson plans can be taken from the Centralized Training Curriculum found on the C&P Intranet Training Site. If used as provided they do not require C&P review and approval); These plans can and often should be modified to focus in on a particular narrow issue of training need. Modified lesson plans are to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training; Any Challenge-oriented original lesson plan developed by Station personnel is to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training); C&P Service Broadcasts that may be provided during the course of the FY may be substituted in place of any training scheduled on an hour by hour basis. Position: Veterans Services Representative (VSR); GS 7-12 (New); Required: Entire Curriculum; (Follow C&P Prescribed Curriculum for new VSRs, as posted on intranet); Course title or topic: Curriculum is posted on C&P Training Intranet Site; Claims Processing Prerequisites: Human Resources and Orientation; Computer Security and LAN; Procedures; Core Values; Core Competencies and Your Job; Voice of the Veteran video; VA in Motion video; VSR Handbook; VA Terminology; SHARE (BDN & CEST); COVERS; PIES; Return with Honor Video; MAPD; AMIE/CAPRI; Medical TPSS (Medical Terminology); Reader Focused Writing Tools; Pre-Determination Team Training; Overview of VA Mission; Reference Materials: Manual Training & WARMS; C&P Website; Claims Folder Maintenance; Records Management; POA/Service Organizations; Compensation; Original Compensation Claims; Non-Original Compensation Claims; VA Form 21-526, App. For Compensation or Pension; Establishing Veteran Status; Claims Recognition; Duty to Assist; Selecting the Correct Worksheet for VA Exams; Issue Specific Claim Development; Asbestos Claim Development; Herbicide Claim Development; POW Claim Development; Radiation Claim Development; PTSD Claim Development; Undiagnosed Illness Claim Development; Dependency; Contested Claims; Deemed Valid and Common-law Marriage; Continuous Cohabitation; Pension; Intro. To Disability Pension; Overview of SHARE (SSA); Administrative Decision Process; Character of Discharge; Line of Duty - Willful Misconduct Claims Development; Workload Management Utilizing WIPP; DEA Training (req. added 4/06); Post-Determination Team Training: Intro to Ratings; Paragraph 29 & 30 Ratings; Ratings & the BDN; BDN 301 Interface Video; PCGL Award Letters; PCGL; Dependents & the BDN; Compensation Offsets; Drill Pay Waivers; Star Reporter; Pension Awards Processing & the BDN; Hospital Reductions; Burial Benefits; Disallowance Processing; DIC Benefits; Death Pension; Accrued Benefits; Accrued Awards & the BDN; Apportionment; Special Monthly Pension; Helpless Child; Incompetency/Fiduciary Arrangements; Claims Processing; Automobile Allowance and Adaptive Equipment; Specially Adapted Housing and Special Home Adaptation Grants; Incarceration; Processing Computer Write Outs; DEA Training (req. added 4/06); Public Contact Team Training; FOIA/Privacy Act; Communication Skills; Telephone Development; Inquiry Routing and Information System (IRIS); Intro to VACOLS; Other VBA Business Lines; Customer Service Insurance Education (2 hrs); Triage Team Training: FNOD Information & PMC Processing; Appeals Team Training: Intro to Appeals Process VACOLS; Pension Maintenance Centers; Income Adjustment Materials: Income Adjustments; Training source: [hyperlink, http://cptraining.vba.va.gov/C&P_Training/vsr/VSR_Curriculum.htm#cpp]. [hyperlink, http://cptraining.vba.va.gov/C&P_Training/vsr/VSR_Curriculum.htm#pred]. [hyperlink, http://cptraining.vba.va.gov/C&P_Training/vsr/VSR_Curriculum.htm#postd]. [hyperlink, http://cptraining.vba.va.gov/C&P_Training/vsr/VSR_Curriculum.htm#pctt]; VSR Core Curriculum Video; VSR Curriculum; VSR Core Curriculum Fast Ltr. 04-12; VSR Core Curriculum Threshold Videos Video; [hyperlink, http://cptraining.vba.va.gov/C&P_Training/vsr/VSR_Curriculum.htm#ttt]; [hyperlink, http://cptraining.vba.va.gov/C&P_Training/vsr/VSR_Curriculum.htm#att]; [hyperlink, http://cptraining.vba.va.gov/C&P_Training/vsr/VSR_Curriculum.htm#iam]. Source: VBA. [End of table] [End of section] Appendix III: Excerpts from Fiscal Year 2007 Training Plans for Four Regional Offices: Each training plan we reviewed contained the same informational categories, some of which were what courses were offered by the regional office, whether or not the course was conducted, and how many employees completed the training. Although the fiscal year 2007 training plans we reviewed include data on whether and when the course was actually completed, the initial training plans submitted at the beginning of the fiscal year of course do not have this information. The lists provided below include the first 25 courses listed on each plan alphabetically, a small sample of the courses that the regional offices reported they completed for the fiscal year. Table 6: Excerpt from Atlanta Regional Office Training Plan: Course name: Accrued Benefits; Number of employees completed: 15; Total hours of training completed: 150. Course name: Accrued Ratings (2 sessions conducted); Number of employees completed: 47; Total hours of training completed: 80. Course name: Administrative Decisions; Number of employees completed: 15; Total hours of training completed: 60. Course name: Ancillary Benefits; Number of employees completed: 14; Total hours of training completed: 14. Course name: Appeals and Ancillary Benefits (2 sessions conducted); Number of employees completed: 26; Total hours of training completed: 41. Course name: Apportionments (2 sessions conducted); Number of employees completed: 29; Total hours of training completed: 194. Course name: Asbestos Claims Development; Number of employees completed: 9; Total hours of training completed: 9. Course name: Auto Allowance/Special Adapted Housing/Special Home Adaptation Grant; Number of employees completed: 15; Total hours of training completed: 30. Course name: Benefits Delivery Network 301 Interface (2 sessions conducted); Number of employees completed: 48; Total hours of training completed: 48. Course name: Beneficiary Identification Records Locator Subsystem Update; Number of employees completed: 17; Total hours of training completed: 17. Course name: Blast Injuries (2 sessions conducted); Number of employees completed: 20; Total hours of training completed: 20. Course name: Burial Benefits (2 sessions conducted); Number of employees completed: 36; Total hours of training completed: 100. Course name: Board of Veterans Appeals Examinations; Number of employees completed: 46; Total hours of training completed: 69. Course name: Compensation & Pension Website (2 sessions conducted); Number of employees completed: 108; Total hours of training completed: 270. Course name: Change of Address/Power of Attorney Processing/No Record Mail; Number of employees completed: 17; Total hours of training completed: 34. Course name: Cardiovascular Issues; Number of employees completed: 38; Total hours of training completed: 76. Course name: Certifying a Case to Board of Veterans Appeals; Number of employees completed: 12; Total hours of training completed: 12. Course name: Character of Discharge; Number of employees completed: 78; Total hours of training completed: 78. Course name: Claims Folder Maintenance (2 sessions conducted); Number of employees completed: 17; Total hours of training completed: 28. Course name: Claims for Direct Service Connection/Aggravation/ Presumptive Service Connection; Number of employees completed: 34; Total hours of training completed: 34. Course name: Claims for Increase; Number of employees completed: 29; Total hours of training completed: 58. Course name: Claims Processing; Number of employees completed: 139; Total hours of training completed: 69.5. Course name: Claims Recognition; Number of employees completed: 84; Total hours of training completed: 336. Course name: Compensation Offsets (3 sessions conducted); Number of employees completed: 167; Total hours of training completed: 352.5. Course name: Computer Security and LAN Procedures; Number of employees completed: 6; Total hours of training completed: 6. Source: VBA. Note: Atlanta's training plan reported the regional office conducted a total of 133 courses for fiscal year 2007. [End of table] Table 7: Excerpt from Baltimore Regional Office Training Plan: Course name: Accrued Benefits; Number of employees completed: 5; Total hours of training completed: 10. Course name: Automated Medical Information Exchange/Compensation and Pension Record Interchange; Number of employees completed: 6; Total hours of training completed: 48. Course name: Appeals and Ancillary Benefits; Number of employees completed: 3; Total hours of training completed: 3. Course name: Asbestos Claims Development; Number of employees completed: 3; Total hours of training completed: 3. Course name: Access Standardized Performance Elements Nationwide; Number of employees completed: 2; Total hours of training completed: 2. Course name: Auto Allowance and Adaptive Equipment (2 sessions conducted); Number of employees completed: 16; Total hours of training completed: 8. Course name: Benefits Delivery at Discharge Development; Number of employees completed: 14; Total hours of training completed: 21. Course name: Benefits Delivery Network 301 Interface (2 sessions conducted); Number of employees completed: 5; Total hours of training completed: 7. Course name: Benefit of the Doubt; Number of employees completed: 3; Total hours of training completed: 12. Course name: Burial Benefits (2 sessions conducted); Number of employees completed: 7; Total hours of training completed: 14. Course name: Compensation & Pension Website (3 sessions conducted); Number of employees completed: 15; Total hours of training completed: 36.5. Course name: Certifying a Case to Board of Veterans Appeals; Number of employees completed: 3; Total hours of training completed: 12. Course name: Character of Discharge; Number of employees completed: 15; Total hours of training completed: 7.5. Course name: Claims Folder Maintenance; Number of employees completed: 7; Total hours of training completed: 14. Course name: Claims Recognition; Number of employees completed: 5; Total hours of training completed: 20. Course name: Communication-Nonverbal Cues; Number of employees completed: 3; Total hours of training completed: 1.5. Course name: Computer Security and LAN Procedures; Number of employees completed: 6; Total hours of training completed: 12. Course name: Conducting a Field Exam; Number of employees completed: 3; Total hours of training completed: 1.5. Course name: Continuous Cohabitation (2 sessions conducted); Number of employees completed: 20; Total hours of training completed: 20. Course name: Core Values; Number of employees completed: 5; Total hours of training completed: 5. Course name: Control of Veterans Records System (3 sessions conducted); Number of employees completed: 10; Total hours of training completed: 12.5. Course name: Customer Service (5 sessions conducted); Number of employees completed: 40; Total hours of training completed: 416. Course name: Dealing with Difficult Payee Situations; Number of employees completed: 3; Total hours of training completed: 3. Course name: Deemed Valid and Common Law Marriages (2 sessions conducted); Number of employees completed: 20; Total hours of training completed: 12.5. Course name: Dependency Issues (3 sessions conducted); Number of employees completed: 22; Total hours of training completed: 26.5. Source: VBA. Note: Baltimore's training plan reported the regional office conducted a total of 191 courses for fiscal year 2007. [End of table] Table 8: Excerpt from Milwaukee Regional Office Training Plan: Course name: 8824e; Number of employees completed: 1; Total hours of training completed: 1. Course name: Administrative Decisions; Number of employees completed: 14; Total hours of training completed: 91. Course name: Advanced Data Manipulation in Excel (VA Learning Online); Number of employees completed: 1; Total hours of training completed: 4. Course name: All - Litigation Hold Memo; Number of employees completed: 130; Total hours of training completed: 32.5. Course name: All-Encryption Training; Number of employees completed: 1; Total hours of training completed: 0.5. Course name: Ancillary Benefits; Number of employees completed: 21; Total hours of training completed: 42. Course name: Auto Allowance and Adaptive Equipment; Number of employees completed: 28; Total hours of training completed: 28. Course name: Blast Injuries (Video); Number of employees completed: 33; Total hours of training completed: 33. Course name: Board of Veterans Appeals review; Number of employees completed: 7; Total hours of training completed: 14. Course name: Compensation & Pension Website; Number of employees completed: 41; Total hours of training completed: 102.5. Course name: Claims Assistant - Burials; Number of employees completed: 4; Total hours of training completed: 4. Course name: Claims Assistant/Program Support Clerk - Power of Attorney; Number of employees completed: 24; Total hours of training completed: 24. Course name: Claims Assistant/Program Support Clerk - Share and Cest; Number of employees completed: 21; Total hours of training completed: 178.5. Course name: Claims Assistant/Program Support Clerk - Veterans Appeals Control and Locator System; Number of employees completed: 25; Total hours of training completed: 25. Course name: Cardiovascular Issues; Number of employees completed: 30; Total hours of training completed: 180. Course name: Challenge 07-02 Centralized Training; Number of employees completed: 6; Total hours of training completed: 720. Course name: Challenge 07-02 Post Centralized Training; Number of employees completed: 6; Total hours of training completed: 1440. Course name: Challenge 07-02 Pre-Req.; Number of employees completed: 6; Total hours of training completed: 720. Course name: Claims Folder Maintenance; Number of employees completed: 41; Total hours of training completed: 82. Course name: Claims Recognition; Number of employees completed: 26; Total hours of training completed: 26. Course name: Character of Discharge Determinations, Line of Duty Determinations, and Administrative Decisions; Number of employees completed: 11; Total hours of training completed: 24.75. Course name: Compensation Offsets; Number of employees completed: 27; Total hours of training completed: 94.5. Course name: Core Values; Number of employees completed: 2; Total hours of training completed: 3.5. Course name: Control of Veterans Records System (2 sessions conducted); Number of employees completed: 2; Total hours of training completed: 3. Course name: Compensation and Pension Examination Project; Number of employees completed: 1; Total hours of training completed: 18. Source: VBA. Note: Milwaukee's training plan reported the regional office conducted a total of 323 courses for fiscal year 2007. [End of table] Table 9: Excerpt from Portland Regional Office Training Plan: Course name: 020 Development; Number of employees completed: 3; Total hours of training completed: 16.5. Course name: 2007 Veterans Service Center Management Workshop; Number of employees completed: 1; Total hours of training completed: 26. Course name: 3.105(e) Reductions; Number of employees completed: 15; Total hours of training completed: 15. Course name: 38 CFR 3.14 & Pyramiding; Number of employees completed: 2; Total hours of training completed: 0.5. Course name: 5-Tier Performance Evaluations; Number of employees completed: 7; Total hours of training completed: 5.25. Course name: 8824 Preparation; Number of employees completed: 1; Total hours of training completed: 5. Course name: Absence & Leave Circular Training; Number of employees completed: 13; Total hours of training completed: 13. Course name: Account Analysis; Number of employees completed: 3; Total hours of training completed: 6. Course name: Account Audits; Number of employees completed: 3; Total hours of training completed: 6. Course name: Accrued Awards & the Benefits Delivery Network; Number of employees completed: 2; Total hours of training completed: 2. Course name: Accrued Ratings; Number of employees completed: 16; Total hours of training completed: 4. Course name: Add Dependents/Verifying Service; Number of employees completed: 18; Total hours of training completed: 9. Course name: Admin Decisions/Rebuilt/Special Monthly Compensation; Number of employees completed: 4; Total hours of training completed: 18. Course name: Administrative Decisions; Number of employees completed: 5; Total hours of training completed: 2.5. Course name: Agent Orange development; Number of employees completed: 4; Total hours of training completed: 4. Course name: Amputation Rule; Number of employees completed: 2; Total hours of training completed: 0.5. Course name: Ancillary Benefits; Number of employees completed: 28; Total hours of training completed: 28. Course name: Appeal Procedures--Refresher; Number of employees completed: 3; Total hours of training completed: 5.25. Course name: Appeals; Number of employees completed: 33; Total hours of training completed: 33. Course name: Appeals and Ancillary Benefits (3 sessions conducted); Number of employees completed: 34; Total hours of training completed: 13. Course name: Appeals--Training and Performance Support System modules; Number of employees completed: 1; Total hours of training completed: 16. Course name: Application/eligibility; Number of employees completed: 1; Total hours of training completed: 3.5. Course name: Apportionments (2 sessions conducted); Number of employees completed: 4; Total hours of training completed: 14.5. Course name: Asbestos Claims Development; Number of employees completed: 23; Total hours of training completed: 23. Course name: Access Standardized Performance Elements Nationwide; Number of employees completed: 6; Total hours of training completed: 6. Source: VBA. Note: Portland's training plan reported the regional office conducted a total of 509 courses for fiscal year 2007. [End of table] [End of section] Appendix IV: Comments from the Department of Veterans Affairs: The Secretary Of Veterans Affairs: Washington: May 9, 2008: Mr. Daniel Bertoni: Director: Education, Workforce, and Income Security Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Bertoni: The Department of Veterans Affairs (VA) has reviewed the Government Accountability Office's (GAO) draft report, Veterans' Benefits: Increased Focus on Evaluation and Accountability Would Enhance Training and Performance Management for Claims Processors (GAO-08-561) and agrees with GAO's conclusions and concurs with GAO's recommendations. Robust employee training and effective performance management are critical to VA's goal of improving overall timeliness and accuracy of veterans claims processing. VA will closely monitor and evaluate the success of our efforts to enhance claims processor performance. The enclosure addresses each of GAO's recommendations in detail and provides technical comments for clarity and accuracy. VA appreciates the opportunity to comment on your draft report. Sincerely yours, Signed by: James B. Peake, M.D. Enclosure: Department Of Veterans Affairs (VA) Comments To Government Accountability Office (GAO) Draft Report: Veterans' Benefits: Increased Focus on Evaluation and Accountability Would Enhance Training and Performance Management for Claims Processors (GAO-08-561): GAO recommends that the Secretary of Veterans Affairs should direct VBA to: * Collect and review feedback from staff on the training conducted at the regional offices to determine: - if the 80-hour annual training requirement is appropriate for all VSRs and RVSRs. Concur. The Veterans Benefits Administration (VBA) has an active program for training evaluation driven by the Administration's priorities. Recent evaluations in support of improving veterans service representative (VSR) and rating veterans service representative (RVSR) training have included evaluation of VBA's Training and Performance Support System (TPSS) (2006-2007 and 2007-2008) and Challenge training for VSRs and RVSRs (2007-2008). The 80-hour training requirement for VSRs and RVSRs was established in FY 2006 to improve the knowledge and skills of claims processors with the ultimate goal of improving accuracy and consistency. The 80-hour training requirement has been evaluated annually by the Central Office staff and maintained at its current level to improve the organization's ability to serve veterans. VBA will consult with its regional offices and evaluate the 80-hour training requirement for new and seasoned VSRs and RVSRs prior to issuance of the training plan call letter for FY 2009. - the extent to which regional offices provide training that is relevant to VSRs' and RVSRs' work, given varying levels of staff experiences. Concur. With the introduction of more than 3,100 new VSRs and RVSRs into regional offices during FY 2007 and FY 2008, VBA's priority in training has been focused on building the knowledge and skill level of new employees through consistent curriculum and delivery methods. VBA will continue its initiatives to provide relevant field training that will hinge on increasing use of the VA Learning Management System (VA LMS) to identify training achievements and gaps at the individual level. In 2006, VBA established a long-term initiative for advanced planning of annual training within regional offices. In its initial phase (2006-2008), regional offices developed and submitted annual training plans that were reviewed by Central Office officials for thoroughness and relevance. Training plans listed the classes that each regional office would conduct for each position (e.g., VSR/RVSR), including the projected date and designated instructors selected from each regional office's more experienced personnel. The initiative's second phase began with the fielding of VA LMS this year, which will allow for recording and tracking of training on an individual employee level. While many regional offices keep records on individual employees' training, VA LMS will become the VBA-wide method of collecting and reporting this information. In FY 2009, regional offices will establish quarterly curricula for VSRs and RVSRs in VA LMS. Regional offices have the option of prescribing different curricula for varying experience levels. Central Office staff will review the curricula for thoroughness and relevance at the administration level, but direct supervisors in the field will determine relevance at the individual level, given varying levels of staff experiences. With VA LMS, supervisors will have easy access to the training records of their subordinates, and, therefore, be able to determine the training needs of each employee to tailor the training curriculum when appropriate. - whether regional offices find the TPSS module a useful learning tool and, if not, what adjustments are needed to make it more useful. Concur. This action has been completed. VBA will continue to collect and evaluate field opinions of TPSS during evaluations, as it has during evaluations of TPSS (2006-2007 and 2007-2008) and Challenge training for VSRs and RVSRs (2007-2008). While GAO's opportunities to collect feedback were confined to interview of a limited number of employees from four regional offices, VBA's three evaluations examined data collected from 37 regional office visits, 470 interviews, and 2,718 survey responses. The 2006-2007 TPSS evaluation found that TPSS provided value to VBA, and identified areas for improved implementation within regional offices. Actions have been assigned for the findings of the TPSS evaluation of 2006-2007, and will be assigned based upon the findings of the other two evaluations when the reports are finalized. * Use information from its new learning management system to hold individual VSRs and RVSRs accountable for completing whatever annual training requirement it determines is appropriate. Concur. The Department has selected a learning management system that places oversight responsibility on supervisors for ensuring training requirements are met. To support the use of VA LMS in the field, VBA has established full-time or part-time training managers in all regional offices to support the directors' training information needs, and is providing classes that enable use of VA LMS to record individual training assignments and completions. The establishment of job-position curricula will provide visibility on requirements and accomplishments to supervisors at all levels, and to Central Office staff. VBA is working closely with the Department's VA LMS program staff to develop standardized reports to provide training progress reports to regional office and Central Office directors to increase visibility on training achievements and deficiencies. * GAO recommends the Secretary of Veterans Affairs should also examine the distribution of claims processing staff across overall performance categories to determine if its performance appraisal system clearly differentiates between overall performance levels, and if necessary adjust its system to ensure that it makes clear distinctions. Concur: VBA will examine how the ratings of claims processing staff are distributed over the performance categories to determine if its performance appraisal system clearly differentiates between overall performance levels. If necessary, VBA will develop recommendations for adjustment of VA's performance appraisal system. [End of section] Appendix V: GAO Contact and Staff Acknowledgments: GAO Contact: Daniel Bertoni (202) 512-7215 bertonid@gao.gov. Staff Acknowledgments: In addition to the contact named above, Clarita Mrena, Assistant Director; Lorin Obler, Analyst-in-Charge; Carolyn S. Blocker; and David Forgosh made major contributions to this report; Margaret Braley, Peter Del Toro, Chris Dionis, Janice Latimer, and Carol Willett provided guidance; Walter Vance assisted with study design; Charles Willson helped draft the report; and Roger Thomas provided legal advice. [End of section] Related GAO Products: Veterans' Benefits: Improved Management Would Enhance VA's Pension Program. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-112]. Washington, D.C.: February 14, 2008. Veterans' Disability Benefits: Claims Processing Challenges Persist, while VA Continues to Take Steps to Address Them. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-473T]. Washington, D.C.: February 14, 2008. Disabled Veterans' Employment: Additional Planning, Monitoring, and Data Collection Efforts Would Improve Assistance. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1020]. Washington, D.C.: September 12, 2007. Veterans' Benefits: Improvements Needed in the Reporting and Use of Data on the Accuracy of Disability Claims Decisions. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-1045]. Washington, D.C.: September 30, 2003. Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-03-893G]. Washington, D.C.: July 2003. Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-488]. Washington D.C.: March 14, 2003. Major Management Challenges and Program Risks: Department of Veterans Affairs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-110]. Washington, D.C.: January 1, 2003. Veterans' Benefits: Claims Processing Timeliness Performance Measures Could Be Improved. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03- 282]. Washington, D.C.: December 19, 2002. Veterans' Benefits: Quality Assurance for Disability Claims and Appeals Processing Can Be Further Improved. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-02-806]. Washington, D.C.: August 16, 2002. Veterans' Benefits: Training for Claims Processors Needs Evaluation. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-601]. Washington, D.C.: May 31, 2001. Veterans Benefits Claims: Further Improvements Needed in Claims- Processing Accuracy. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO/HEHS-99-35]. Washington, D.C.: March 1, 1999. [End of section] Footnotes: [1] These practices are laid out primarily in two GAO reports: Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-04-546G] (Washington, D.C.: March 2004) and Results Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-488] (Washington, D.C.: Mar. 14, 2003). [2] 38 U.S.C. § 5312(a). [3] While new claims processors are on probation, 100 percent of the claims work they perform is quality reviewed by a supervisor. After their probationary period, only a small sample of their claims is quality reviewed. [4] Typically, RVSRs are promoted VSRs, although in some instances, VA hires RVSRs from outside of VA who have medical or legal experience. [5] Staff who teach classes other than centralized training are not required to take the week-long Instructor Development Course, although they may do so if openings exist. They can also take an 8-hour condensed course for regional instructors. [6] In 2001, we reported that VBA had spent or obligated about $18.6 million of the estimated total TPSS program cost of $32 million. See GAO, Veterans' Benefits: Training for Claims Processors Needs Evaluation, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-601] (Washington, D.C.: May 31, 2001). [7] VBA defines an experienced VSR or RVSR as one who has been in that position for 1 year or more. [8] To determine if VSRs and RVSRs in a regional office are generally meeting their annual training requirement, the aggregate number of training hours completed in a given year by all staff in that office is divided by the number of staff in that office. [9] According to VBA officials, the board is made up of a mix of regional office and central office staff from different VBA business lines including Employee Development and Training, Human Resources, the Compensation and Pension Service, and the Insurance Service. [10] GAO, Veterans' Benefits: Improved Management Would Enhance VA's Pension Program, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08- 112] (Washington, D.C.: Feb. 14, 2008). [11] In 2001, GAO reported that VBA's TPSS may not fully achieve its objectives of providing standardized training to new employees, reducing the training period required for new employees, or improving claims-processing accuracy and consistency. In the report, we recommended actions the agency should consider in providing timely standardized training and providing indicators of the impact of TPSS on accuracy and consistency. In its technical comments on this report, VA indicated it accomplished the first recommendation. See [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-601]. [12] The central office has set a minimum performance level for each element that defines the fully successful level of performance. Regional offices may set higher fully successful levels for their staff, and three of the offices we visited had set a higher level for at least one element. Regional offices also have discretion to set the level for exceptional performance in each element for their staff. [13] See GAO, Human Capital: Preliminary Observations on the Administration's Draft Proposed "Working for America Act," [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-142T] (Washington, D.C.: Oct. 5, 2005). [14] In three of the four offices we visited, staff members placed in the outstanding and excellent categories receive bonuses, and in one of these offices some staff in the fully successful category also receive bonuses. [15] Officials in the other two offices we visited reported no problems with the performance appraisal formula. Officials in one of these offices told us the current five-level system provides more flexibility than the previous pass/fail system. [16] We asked VA for fiscal year 2007 performance appraisal data for VSRs and RVSRs nationally to determine whether the distribution of staff across overall performance categories is similar at the national level. While VA indicated that it collects performance appraisal data for regional office staff, the agency was unable to provide us with appraisal data specifically for VSRs and RVSRs, as these positions are part of a broader job series. [17] To determine each office's accuracy performance in fiscal year 2007, we used data obtained from VBA's Systematic Technical Accuracy Review (STAR) system. In an earlier GAO report, Veterans' Benefits: Further Changes in VBA's Field Office Structure Could Help Improve Disability Claims Processing, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-06-149] (Washington, D.C.: Dec. 9, 2005), we identified problems that affected the use of the STAR data to make distinctions in accuracy among regional offices. However, for the purposes of site selection for our current review, we judged the STAR data to be sufficiently reliable. We made this determination based on a sensitivity analysis we did on earlier year data that considered sampled cases that were not sent in for STAR review. After this analysis we found that even with the existing limitations in the STAR data, Milwaukee and Baltimore had higher accuracy scores and Atlanta and Portland had lower accuracy scores. Even though the sensitivity analysis was done on earlier year data, the ranking of the four offices was similar in fiscal year 2007, showing that the offices we deemed to have higher accuracy scores in an earlier year still had higher accuracy scores in fiscal year 2007 and the same remained true for the offices with lower accuracy scores. [18] One question we asked the regional offices was whether each course on their fiscal year 2007 training plan addressed a core technical training topic. For three of the offices, the data we received did not cover all training hours provided during the fiscal year, but each office provided data on at least 99 percent of its training hours. [19] GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-546G] (Washington, D.C.: March 2004). [20] GAO, Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-488] (Washington, D.C.: Mar. 14, 2003). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: