Appendix B
Indicators of Agency Progress in Meeting Goals
Performance standards for federal statistical agencies were established through efforts of the Interagency Council on Statistical Policy under the Government Performance Responsibility Act for use in completing the Administration's Program Assessment Rating Tool (PART). These standards are used to establish product quality and program performance goals for federal statistical agencies.
The BTS Strategic Plan addresses six interrelated goals covering product quality and program performance.
Product Quality: Product quality encompasses many attributes, including (but not limited to) relevance, accuracy, and timeliness. The basic measures in this group relate to the quality of specific products, thereby providing actionable information to managers. These are ''outcome-oriented'' measures and are key to the usability of information products. In some sense, relevance relates to "doing the right things," while accuracy and timeliness relate to "doing things right."
Program Performance: Program performance encompasses balancing the dimensions of cost, dissemination, and mission accomplishment for the agency as a whole; operating efficiently and effectively; ensuring that customers receive the information they need; and serving the information needs of the Nation. Costs of products or programs may be used to develop efficiency measures. Dissemination involves making sure customers receive the information they need via the most appropriate mechanisms. Mission achievement means that the information the program produces makes a difference. Hence, three key dimensions are being used to indicate program performance: cost (input), dissemination (output), and mission achievement (outcome).
Performance evaluation indicators measure BTS' performance toward each of six strategic goals. Below we briefly describe the six strategic goals, map them to OMB's performance standards for federal statistical agencies, and describe the indicators for each.
Relevance
OMB Standard - Relevance is the degree to which products and services are useful and responsive to users' needs. Relevance of data products and analytic reports may be monitored through a professional review process and ongoing contacts with data users. Product relevance may be indicated by customer satisfaction with product content, information from customers about product use, demonstration of product improvements, comparability with other data series, agency responses to customer suggestions for improvement, new or customized products/services, frequency of use, or responses to data requests from users (including policy makers). Through a variety of professional review activities, agencies maintain the relevance, accuracy, and validity of their products, and encourage data users and other stakeholders to contribute to the agency's data collection and dissemination programs. Striving for relevance requires monitoring to ensure that information systems anticipate change and evolve to appropriately measure our dynamic society and economy. Qualitative or quantitative indicators may measure relevance.
BTS Goal - To anticipate the needs of decision makers and stakeholders by providing the information that is most useful and responsive to them through a thorough understanding of major transportation issues and trends.
BTS Performance Indicators - BTS has developed indicators for tracking information requests, product dissemination, and customer satisfaction with the BTS website. Relevance is evaluated through the following measures:
- Increase the number of congressional and government agency contacts regarding BTS information (increasing awareness in Congress and other government agencies of what we can provide).
- Increase the number of average daily unique visitors to the TranStats data warehouse site (increasing our customer base for our transportation data warehouse).
Accuracy
OMB Standard - Accuracy measures the important features of correctness, validity, and reliability of data and information products measured as degree of closeness to target values. For statistical data, accuracy may be defined as the degree of closeness to the target value and measured as sampling error and various aspects of nonsampling error (e.g., response rates, size of revisions, coverage, and edit performance). For analysis products, accuracy may be the quality of the reasoning, reasonableness of assumptions, and clarity of the exposition, typically measured and monitored through review processes. In addition, accuracy is assessed and improved by internal reviews, comparisons of data among different surveys, linkages of survey data to administrative records, redesigns of surveys, or expansions of sample sizes. Qualitative or quantitative indicators may measure accuracy.
BTS Goal - To provide high quality data, analysis, and information for transportation decisionmaking that is correct, valid, and reliable.
BTS Performance Indicators - BTS evaluates the accuracy of information products through the following measures:
- Maintain a minimum 80 percent response rate on the Commodity Flow Survey (CFS) (to reduce non-response bias).
- Increase the response rate on the Omnibus Household Survey (OHS) (to reduce non-response bias).
Timeliness
OMB Standard - Timeliness is the timing of information releases. Timeliness may be measured as time from the close of the reference period to the release of information, or customer satisfaction with timeliness. Timeliness may also be measured as how well agencies meet scheduled and publicized release dates, expressed as a percent of release dates met. Qualitative or quantitative indicators may measure timeliness.
BTS Goal - Reduce the lag time in data reporting so that decision makers have a nearly "real-time" view of the transportation system and factors affecting it.
BTS Performance Indicators - BTS has developed indicators for its major programs. Timeliness is evaluated through the following measures:
- Improve average response time for general data and information requests from customers both government and private sector (to improve customer responsiveness).
- Increase the percentage of periodic airline traffic data releases that occurred on schedule (enable our stakeholders and customers predictable delivery of our datasets).
- Ensure that the annual release of National Transportation Atlas Database (NTAD) CD is available prior to the start of the annual ESRI (Environmental Systems Research Institute) International User Conference on GIS and mapping software (to provide timely support for research, analysis, and decision-making across all modes of transportation).
- Reduce response time for map requests by the DOT Crisis Management Center (CMC) and in support of the Continuity of Operations (COOP) plans (to improve emergency response by providing current and timely information).
Cost
OMB Standard - Cost is the dollar amount used to produce data products or services. The development and use of financial performance measures within the Federal Government is an established goal, and the intent of such measures is to determine the "true costs" of various programs or alternative modes of operation at the Federal level. Examples of cost data include full costs of products or programs, return on investment, dollar value of efficiencies, and ratios of cost to products distributed. Quantitative indicators may measure cost.
BTS Goal - Reduce the cost of recurring programs in order to provide data and analysis in a cost-effective manner.
BTS Performance Indicators - The BTS cost measures are:
- Improve cost efficiencies in the production of BTS' major data reference reports (Transportation Statistics Annual Report, National Transportation Statistics, and the Pocket Guide to Transportation Statistics).
- Improve cost efficiencies in acquiring international freight data (Transborder data).
Dissemination
OMB Standard - Dissemination is the availability, accessibility, and distribution of information products and services. Typical measures include: on-demand requests fulfilled, product downloads, degree of accessibility, customer satisfaction with ease of use, number of participants at user conferences, citations of agency data in the media, number of Internet user sessions, number of formats in which data are available, amount of technical support provided to data users, exhibits to inform the public about information products, issuance of newsletters describing products, usability testing of web sites, and assessing compliance with Section 508 of the Rehabilitation Act, which requires Federal agencies to make their electronic and information technology accessible to people with disabilities. Qualitative or quantitative indicators may measure dissemination.
BTS Goal - To provide data and analysis products that are readily available, easily accessible, and widely distributed.
BTS Performance Indicators - The BTS measures for dissemination are:
- Increase the number of National Atlas Transportation Database (NTAD) CD-ROMs distributed (to broaden the customer base for this product).
- Increase the number of datasets downloaded from TranStats (to broaden customer usage for this product).
Mission Achievement
OMB Standard - Mission achievement is the effect of or satisfaction with the statistical programs. For government statistical programs, this dimension responds to the question-have we achieved our objectives and met the expectations of our stakeholders? Under this dimension, statistical programs document their contributions to the goals and missions of parent departments and other agencies, the Administration, Congress, and information users in the private sector and the general public. For statistical programs, this broad dimension involves meeting recognized societal information needs and also addresses the linkage between statistical outputs and programmatic outcomes.
However, identifying this linkage is far from straightforward. It is frequently difficult to trace the effects of information products on the public good. Such products often are necessary intermediate inputs in the creation of high visibility information whose societal benefit is clearly recognized. The statistics produced by statistical agencies are used to track the performance of programs managed by their parent or other organizations related to topics such as crime, education, energy, the environment, health, science, and transportation.
Moreover, beyond the direct and focused uses of statistical products and programs, the statistical agencies and their products serve a diverse and dispersed set of data users working on a broad range of applications. Users include government policy makers at the federal, state, and local levels, business leaders, households, academic researchers, analysts at public policy institutes and trade groups, marketers and planners in the private sector, and many others. Information produced by statistical agencies often is combined with other information for use in the decisionmaking process. Thus, the relationship between program outputs and their beneficial uses and outcomes is often complex and difficult to track. Consequently, agencies use both qualitative and quantitative indicators to make this linkage as explicit as feasible.
Quantitative measures may be used to reflect mission achievement. For example, customer satisfaction with the statistical agency or unit indicates if the agency or unit has met the expectations of its stakeholders. In the absence of preferred quantitative indicators, qualitative narratives can indicate how statistical agency products contribute to and evaluate progress toward important goals established for government.
BTS Goal - Provide (dissemination) high-quality data (accuracy) that fill transportation data gaps and advance their effective (relevancy and timeliness) use for transportation decisionmaking.
BTS Performance Indicators - BTS evaluates mission achievement through the following measures:
- Improve coverage of legislative mandates in the Transportation Statistics Annual Report (TSAR) (degree to which we support our legislative mandate).
- Track and increase the number of BTS products that contribute to the accomplishment of important, established government goals (meeting the agency mission).
|
