Detailed Information on the
Bureau of Labor Statistics Assessment

Conducting an independent evaluation on how the agency is improving its effectiveness in meeting the needs of its data users.

Developing additional efficiency and cost-effectiveness measures to demonstrate ongoing program improvement.

Establishing more ambitious targets for its long term and annual performance measures to drive continued improvement.

Showing more clearly to the general user aspects of program performance such as outcome-based, quantitative measures of data accuracy.

Developing efficiency measures to cover more of the program.

Maintaining program-monitoring and operational successes.

Measure: The percent of output, timeliness, accuracy, and long-term improvement targets achieved for labor force statistics.

Explanation:The Labor Force Statistics programs provide comprehensive and timely information on the labor force, employment, unemployment, and related labor market characteristics at the national level; industrial and occupational employment at the State and local levels; and labor force and unemployment at State and local levels. In addition, these programs develop projections of the labor force, economic growth, industrial output, and employment by industry and occupation for 10 years into the future for the Nation as a whole. This performance indicator is composed of the underlying performance measures of output, timeliness, accuracy, and long-term improvement for the Labor Force Statistics programs. BLS demonstrates continuous improvement by setting more ambitious targets on its overall indicator, as well as the individual measures that underlie the indicator. For example, in 2008, the Current Employment Statistics program released experimental new all employee hours and earnings series at the State and area levels. In 2008 and again in 2009, BLS will increase the number of State and area estimates released by the Local Area and Unemployment Statistics program, and establishments surveyed by the Quarterly Census of Employment and Wages program. In 2009, the Current Population Survey will begin publishing employment information on people with disabilities.

Measure: The percent of output, timeliness, accuracy, and long-term improvement targets achieved for prices and living conditions.

Explanation:The Prices and Cost of Living programs produce the Consumer Price Index (CPI), the Producer Prices and Price Indexes (PPI), and the U.S. Import and Export Price Indexes. These programs provide data for many geographic areas within the United States, and estimates of consumers' incomes and expenditures that are used in analysis of price behavior and consumer spending patterns. These programs also provide these data for interpretation of price movements in relation to other major economic changes, and the formulation and evaluation of economic policy. This performance indicator is composed of the underlying performance measures of output, timeliness, accuracy, and long-term improvement for the Prices and Cost of Living programs. BLS demonstrates continuous improvement by setting more ambitious targets on its overall indicator, as well as the individual measures that underlie the indicator. For example, in 2008, PPI published new price indexes for non-residential construction of new manufacturing and industrial buildings, and will publish new indexes for four specialty trade contractors. In 2009, PPI will complete work on an improvement initiative to upgrade its data estimation system, and continue work to upgrade its data repricing system, which will ensure a stable operating environment.

Measure: The percent of output, timeliness, accuracy, and long-term improvement targets achieved for compensation and working conditions.

Explanation:The Compensation and Working Conditions programs produce data on employee compensation, including information on wages, salaries, and employer-provided benefits, by occupation, for major labor market areas and industries. These programs also produce information on work stoppages. In addition, these programs compile annual information on the incidence and number of work-related injuries, illnesses, and fatalities. This performance indicator is composed of the underlying performance measures of output, timeliness, accuracy, and long-term improvement for the Compensation and Working Conditions programs. BLS demonstrates continuous improvement by setting more ambitious targets on its overall indicator, as well as the individual measures that underlie the indicator. For example, in 2008, BLS published data from a redesigned, more efficient Survey of Occupational Injuries and Illnesses (SOII) sample, which led to an increase in the number of days away from work cases collected and coded in the SOII. In addition, the Employee Benefits Survey will publish a special annual release on the incidence and key provisions for State and local governments.

Measure: The percent of output, timeliness, accuracy, and long-term improvement targets achieved for productivity and technology.

Explanation:The Productivity and Technology programs measure and analyze productivity trends for major sectors of the economy and individual industries in order to examine the factors underlying changes in productivity. The programs also develop international comparisons of productivity, hourly compensation, unit labor costs, and employment and unemployment. This performance indicator is composed of the underlying performance measures of output, timeliness, accuracy, and long-term improvement for the Productivity and Technology programs. BLS demonstrates continuous improvement by setting more ambitious targets on its overall indicator, as well as the individual measures that underlie the indicator. For example, in 2008, the International Comparisons program added three new countries to its manufacturing series. In addition, the Industry Productivity program will expand the percent of industries covered by labor productivity measures.

Measure: Customer satisfaction with BLS data and assistance.

Explanation:BLS measures customer satisfaction with its Occupational Outlook Handbook (OOH) website, a nationally recognized source of career information that is designed to provide valuable assistance to individuals making decisions about their future work lives. OOH is one of the most frequently used BLS websites. OOH is assessed independently by the American Customer Satisfaction Index (ACSI), which enables BLS to see how it is doing not just over time, but also in comparison to other Federal agencies and the private sector. The ACSI score allows BLS to measure the impact of enhancements it makes to its OOH website. Underlying data from the ACSI also help to identify potential areas for further improvement, such as in navigation, functionality, and site performance.

Measure: Cost per transaction of Internet Data Collection Facility.

Explanation:BLS collects survey data from respondents using many different methods, such as fax, telephone, mail, and personal visits. Wherever possible, BLS offers multiple reporting options to make it easier and more convenient for respondents to report, which reduces their data collection burden and helps to maintain high response rates. The Internet Data Collection Facility (IDCF) is a single, secure architecture that multiple BLS surveys use to collect information from respondents online. Although the IDCF is an important tool for offering a wider range of reporting options to respondents, it also represents a more economical means of data collection. The IDCF efficiency measure is calculated by dividing the cost of maintaining and operating the facility by the number of transactions. Because BLS can add respondents to the IDCF without appreciably adding to its cost, the more respondents who report via the IDCF, the lower the unit cost.

Is the program purpose clear?

Explanation: Clear focus on labor-related data gathering and analysis. BLS must produce impartial, objective, and accurate data on employment and unemployment, price change, compensation, safety and health, productivity, and economic growth.

Evidence: Authorizing law: 29 USC Secs. 1-9. GPRA document: www.bls.gov/bls/blsplan00.htm. FY 2005 BLS budget submission.

Does the program address a specific and existing problem, interest, or need?

Explanation: Much of BLS data represents a national-level, pure public good -- essential to develop macro-economic policy -- which will not be provided in the market because revenue for dissemination cannot defray the cost of data collection.

Evidence: In Statistical Policy Directive #3, OMB designates seven BLS series as Principal Federal Economic Indicators, subject to the directive's government-wide requirements. General lack of competition in providing BLS-like data. Paul A. Samuelson, "The Foundations of Economic Analysis."

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: BLS is virtually the only organization collecting labor-related data nationwide. Rare competing data sources -- resulting data has little redundancy. The newly enacted data sharing law will allow BLS and Census to share business data which should further reduce redundancy. The Bureau of Economic Analysis (BEA) does very little of its own data collection. With rare exceptions, BLS data are reliable, consistent, produced efficiently, and (so far as is known), accurate, although definitions and coverage may lag behind economic change.

Evidence: The OMB designation of seven BLS series as Principal Economic Indicators, subject to government-wide requirements. Lack of competing data sources. Professional economic and statistical literature, e.g., National Research Council, "Principles & Practices for a Federal Statistical Agency," 2nd ed., 2001.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: No evidence that any other mechanism for data collection would be more effective.

Evidence: National Research Council, "Principles & Practices for a Federal Statistical Agency," 2nd ed., 2001.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: A mixed situation that meets the standard for "yes." BLS has made good progress in focusing its efforts toward improving its price data and increasing service-sector coverage. Still (and granted the need for due continuity in statistical time series), the agency has sometimes been slow to serve the new users interested in data on new and growing industries, occupations, and geographic areas. That is because it can be slow to reallocate resources from old to new industries, etc. and also from one survey to another. For example, a key table of the monthly employment gives the goods sector (e.g., mining, manufacturing) nine times the per-employee disaggregation of the service sector. Also, a March 2000 report to the Workforce Information Council recommended comprehensively rethinking the five layoff-related data sources, but there is little sign thus far of such agency action.

Evidence: National Research Council, "At What Price? Conceptualizing and Measuring Cost-of-Living and Price Indexes," 2002. Evaluation of employment data by the American Statistical Association, 1993. Disaggregation of employment data: www.bls.gov/news.release/pdf/empsit.pdf for 7/03, Table B-3. "Layoff-Related Data Sets," a working group report to the Workforce Information Council (www.workforceinfocouncil.org).

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Working with DOL, BLS has made considerable progress since last year in limiting the number of long-term performance measures and adding an efficiency measure for one survey. Adding long-term and annual measures, BLS has reduced the length of the list from 56 pages to one, and from about 360 measures to eight. Moreoever, despite outcome measures ordinarily being difficult for statistical agencies, BLS includes customer satisfaction.

Evidence: Draft of performance table for FY 05 Budget and other GPRA materials. Comparison with those of other statistical agencies. National Research Council, "Principles and Practices for a Federal Statistical Agency," 2nd ed., 2001. [Federal] Interagency Council on Statistical Policy (ICSP), "Guidelines on Reporting the Performance of Statistical Agencies," 2000. GAO, "Statistical Agencies: Adherence to Guidelines" 1995. The preceding three reports provide principles for many of the performance concepts that BLS uses.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: A mixed situation that does not yet meet the standard for "yes." Often, targets reflect the design of surveys and information systems, rather than program performance. As a result some targets and measures do not necessarily reflect continual improvement. However, some of the projects referred to are ambitious -- even bold.

Evidence: See above. FY 04 Congressional Budget Justification.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: As mentioned, BLS has drastically limited its count of performance measures, as one step in achieving greater transparency.

Evidence: See 2.1 .

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: BLS has made considerable improvement since last year toward setting ambitious annual targets.

Evidence: See 2.1 .

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Contracts with State agencies specify deliverables and data quality. Program partners are monitored.

Evidence: For example, LMI Memo S-02-4, "2003 Labor Market Information Cooperative Agreement," May 14, 2002.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Many of the evaluations -- often initiated by BLS -- are totally independent and of extremely high quality. Many evaluations are sparked by research discoveries of data problems (often, through BLS research) and/or by changed economic circumstances highlighting data deficiencies -- these represent evaluations based on the needs of the program. The evaluations have frequently led to program improvements.

Evidence: Boskin Commission on the consumer price index (M.J. Boskin et al., "Final Report on the Advisory Commission to Study the Consumer Price Index," GPO, 1996) and followups, such as BLS-commissioned study by the National Research Council ("At What Price? Conceptualizing and Measuring Cost-of-Living and Price Indexes," 2002). Evaluation of employment data by the American Statistical Association, 1993.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Despite improvement, does not yet meet the standards for "yes." Although the FY 2005 budget submission does relate a few numerical performance measures to the overall resource level, the presentation is neither complete nor transparent. Most of the submission uses the qualitative and non-transparent performance level of "produce [a] full set of information." Still, BLS has served as the model for DOL's submission of an integrated performance budget that accounts for expenditures at the survey-program level. Despite the emphasis on a "full set of information" as a performance measure, the FY 2005 BLS budget submission does show how one survey's sample size, output publications, and response rate would be hurt by a budget total at what is called the allowance level. Two years ago, BLS updated its allocation of direct and indirect costs to its major surveys to show the linkage between funding and total costs, although not unit costs.

Evidence: FY 2005 and prior BLS budget submissions.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The agency has worked closely with DOL's Center for Program Planning and expects further program in improving transparency. e.g., by reporting what general users consider indicators of statistical accuracy, through more emphasis on non-sampling errors, such as response rates. BLS plans to include an efficiency measure in its 2005 budget.

Evidence: FY 2004 and 2005 BLS budget submissions. GPRA document: www.bls.gov/bls/blsplan00.htm.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Regular monitoring of survey partners such as State agencies is used in real time to modify procedures. Results of independent, quality, outside evaluations have been used to modify BLS practices.

Evidence: Boskin Commission on the consumer price index (M.J. Boskin et al., "Final Report on the Advisory Commission to Study the Consumer Price Index," GPO, 1996) and followups, such as BLS-commissioned study by the National Research Council ("At What Price? Conceptualizing and Measuring Cost-of-Living and Price Indexes," 2002). Evaluation of employment data by the American Statistical Association, 1993. BLS' internal Quarterly Review and Analysis Reports.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: Quarterly progress reports by executives and meetings with the Commissioner. Systematic reviews of cooperative agreements with program partners. The agreements delineate deliverables, specified quality assurances, and performance standards. Also, DOL ties performance ratings for managers and supervisors to the achievement of Departmental and program-specific goals and outcomes; employees are aligned on a fiscal year cycle to help cascade standards to non-supervisory personnel, and a system accountability review is planned for 2004.

Evidence: Quarterly Review and Analysis Reports. Revised Performance Management Plans for Senior Executives (Form DL 1-2059, Rev. 10/2001) and for Supervisors and Managers (Form DL 1-382, Rev. 10/2001); briefings by DOL staff; and DOL goals supporting the Human Capital Initiative of the President's Management Agenda.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: There are no outstanding audit findings. The agency reviews obligations and outlays monthly. Additionally, BLS accounts for commitments. Generally accurate projection of end-of-year balances.

Evidence: Automated BLS financial systems. Monthly status reports. Quarterly estimates and actual obligations reported in apportionments.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: A mixed situation that satisfies the standard for "yes." BLS has a detailed and timely internal accounting system. It awarded a major survey contract through competition and uses competition for its labor hours contracts, as well as non-Federal contracts, such as contracts for supermarket-scanner data. However, there could be more use of competitive sourcing, such as through further position classification in the FAIR process.

Evidence: DOL 2001 Commercial Activities & Inherently Governmental inventories.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Notably, continuing reimbursable agreement with Census Bureau to collaborate on the Current Population Survey and the use of BLS data by the Bureau of Economic Analysis (BEA) to prepare, e.g., GDP estimates. Works with State agencies through the Workforce Investment Council (WIC). BLS is a member of the Interagency Council on Statistical Policy, a committee of representatives from 15 Federal agencies, which works to identify areas for collaboration.

Evidence: WIC authorized in Workforce Investment Act of 1996 (PL 105-220), Sec. 309.

Does the program use strong financial management practices?

Explanation: Since 2001, the annual audits have posted "clean opinions." BLS has conistently received 3 or fewer audit findings. In April 2003, DOL rated each agency's progress on the PMA scorecard. BLS received a progress rating of "green" on the financial management initiaitve. Strong and detailed internal accounting system.

Evidence: DOL, "Annual Report on Performance & Accountability," various years. GPRA document: www.bls.gov/bls/blsplan00.htm & www.bls.gov/bls/blsapp.htm

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Quarterly reviews of each BLS statistical program and meetings with the Commissioner. Also, the Department of Labor has recently tied its performance goals to performance ratings for managers; the new appraisal system *check is being cascaded throughout the Department.

Evidence: Revised DOL performance management plans for senior executives (Form DL1-2059, rev. 10/01) and for supervisors & managers (Form DL1-382, rev. 10/01).

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: Generally good progress in achieving long-term performance goals. Targets usually met but often are not adequately ambitious yet.

Evidence: Draft of performance table for FY 05 Budget.GPRA documents: www.bls.gov/bls/blsplan00.htm & www.bls.gov/bls/blsapp.htm

Does the program (including program partners) achieve its annual performance goals?

Explanation: Targets generally met.

Evidence: See above. FY 04 Congressional Budget Justification.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: Increased coverage of the service sector, for example, has been achieved without an inflation-adjusted program budget increase, as was conversion of a key employment survey to probability sampling. Similarly, certain detailed employment which was previously available only annually, is now available quarterly. However, the efficiency measure is new and limited to a single survey, there has been little use of the A-76 process, routine updating is sometimes considered an "efficiency improvement," and sometimes is slow to change program operations (see, e.g., q. 1.5) or reimbursable agreements.

Evidence: Draft of performance table for FY 2005 Budget.GPRA documents: www.bls.gov/bls/blsplan00.htm & www.bls.gov/bls/blsapp.htm

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: BLS releases key employment and unemployment data faster than other countries, according to the International Monetary Fund (IMF). Although this is not the case for consumer-inflation data (i.e., CPI), the U.S. is the only nation with any of the following desirable features for its consumer-inflation data: probability sampling used throughout for more representative data; combination of geometric-mean and Laspeyres formulas to show better how changes in relative prices affect the cost of living; and a "superlative" measure, also to reflect changes in relative prices. Making a different comparison, BLS' FY 2001 score of 74 on the American Customer Satisfaction Index (ACSI) -- although unpublished -- compares favorably with those of other statistical agencies that published their scores. (BLS will measure its ACSI satisfaction levels annually, starting with FY 2003.)

Evidence: IMF: www.imf.org/ ; dsbb.imf.org/ ; and Working Paper 96/103 at www4.statcan.ca/ ACSI: www.theacsi.org/ and FY 2005 budget submission.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Results of independent, quality, outside evaluations have shown effectiveness and, moreover, have been used to improve BLS practices.

Evidence: Followups to the Boskin Commission on the consumer price index, such as BLS-commissioned study by the National Research Council ("At What Price? Conceptualizing and Measuring Cost-of-Living and Price Indexes," 2002). Evaluation of employment data by the American Statistical Association, 1993.