|
Introduction
The Bureau of Transportation Statistics (BTS), a
statistical agency within the Research and Innovative Technology Administration
(RITA), U.S. Department of Transportation (DOT), compiles, analyzes, and
publishes data and information needed for transportation decisionmaking, and
generally coordinates DOT statistical programs. BTS also manages the National Transportation Library and the Office of
Airline Information, and leads the federal effort in developing transportation
geo-data to fulfill the vision of the National Spatial Data Infrastructure
(NSDI).
BTS has a broad responsibility derived from legislation
(see BTS Strategic Plan 2003-2008, Appendix
A), through which Congress created the agency and established the scope of what
we do. BTS supports the highest
priorities of the Department of Transportation – to enhance safety, mobility, global connectivity,
environmental stewardship, and security (the five strategic goals of the
Department of Transportation).
BTS measures
its success in terms of outcomes in six
key areas that frame the strategic goals. These six dimensions and focus areas were developed by the statistical
agencies and statistical units represented on the Interagency Council on
Statistical Policy (ICSP):
- Relevance— the
degree to which products and services are useful and responsive to the needs
of our customers and stakeholders.
- Accuracy—the correctness,
validity, and reliability of data and information products.
- Timeliness—the
timing of information releases.
- Cost—the dollar
amount used to produce data products and services.
- Dissemination—the
availability, accessibility, and distribution of products and services.
- Mission Achievement— the effectiveness of RITA/BTS
information programs in satisfying our customers and stakeholders.
Performance standards for federal statistical
agencies were established through efforts of the Interagency Council on
Statistical Policy under the Government Performance and Results Act for use in
completing the Administration’s Program Assessment Rating Tool (PART). These
standards are used to establish product quality and program performance goals
for federal statistical agencies.
The BTS Strategic Plan addresses six
interrelated goals covering product quality and program performance.
Product Quality: Product quality
encompasses many attributes, including (but not limited to) relevance,
accuracy, and timeliness. The basic measures in this group relate to the
quality of specific products, thereby providing actionable information to
managers. These are ‘‘outcome-oriented’’ measures and are key to the usability
of information products. In some sense, relevance relates to “doing the right
things,” while accuracy and timeliness relate to “doing things right.”
Program Performance: Program performance encompasses balancing the
dimensions of cost, dissemination, and mission accomplishment for the agency as
a whole; operating efficiently and effectively; ensuring that customers receive
the information they need; and serving the information needs of the Nation.
Costs of products or programs may be used to develop efficiency measures.
Dissemination involves making sure customers receive the information they need
via the most appropriate mechanisms. Mission achievement means that the information the program produces makes a difference.
Hence, three key dimensions are being used to indicate program performance: cost (input), dissemination (output), and mission achievement (outcome).
Performance evaluation indicators measure
BTS’ performance toward each of six strategic goals. Below we briefly describe the six strategic
goals, map them to OMB’s performance standards for federal statistical agencies,
and describe the indicators for each. This is the first report of an annual series that will be updated each
year.
Relevance
OMB Standard – Relevance is the
degree to which products and services are useful and responsive to users’
needs. Relevance of data products and analytic reports may be monitored through
a professional review process and ongoing contacts with data users. Product
relevance may be indicated by customer satisfaction with product content,
information from customers about product use, demonstration of product
improvements, comparability with other data series, agency responses to
customer suggestions for improvement, new or customized products/services, frequency
of use, or responses to data requests from users (including policy makers).
Through a variety of professional review activities, agencies maintain the
relevance, accuracy, and validity of their products, and encourage data users
and other stakeholders to contribute to the agency’s data collection and
dissemination programs. Striving for relevance requires monitoring to ensure that
information systems anticipate change and evolve to appropriately measure our
dynamic society and economy. Qualitative or quantitative indicators may measure
relevance.
BTS
Goal - To anticipate the
needs of decision makers and stakeholders by providing the information that is
most useful and responsive to them through a thorough understanding of major
transportation issues and trends.
BTS
Performance Indicators - BTS
has developed indicators for tracking information requests, product
dissemination, and customer satisfaction with the BTS website. Relevance is evaluated through the following
measures:
- Increase the number of congressional and
government agency contacts regarding BTS information (increasing awareness in
Congress and other government agencies of what we can provide).
- Increase the number of average daily unique
visitors to the TranStats data warehouse site (increasing our customer base for
our transportation data warehouse).
Accuracy
OMB
Standard – Accuracy measures
the important features of correctness, validity, and reliability of data and
information products measured as degree of closeness to target values. For statistical
data, accuracy may be defined as the degree of closeness to the target value
and measured as sampling error and various aspects of nonsampling error (e.g.,
response rates, size of revisions, coverage, and edit performance). For analysis products, accuracy may be the
quality of the reasoning, reasonableness of assumptions, and clarity of the
exposition, typically measured and monitored through review processes. In addition, accuracy is assessed and improved
by internal reviews, comparisons of data among different surveys, linkages of
survey data to administrative records, redesigns of surveys, or expansions of
sample sizes. Qualitative or quantitative indicators may measure accuracy.
BTS
Goal - To provide high
quality data, analysis, and information for transportation decisionmaking that
is correct, valid, and reliable.
BTS
Performance Indicators - BTS
evaluates the accuracy of information products through the following measures:
- Maintain
a minimum 80 percent response rate on the Commodity Flow Survey (CFS) (to
reduce potential non-response bias).
- Increase the response rate on the Omnibus Household Survey (OHS) (to reduce potential non-response
bias).
Timeliness
OMB
Standard – Timeliness is the
timing of information releases. Timeliness may be measured as time from the
close of the reference period to the release of information, or customer satisfaction
with timeliness. Timeliness may also be measured as how well agencies meet
scheduled and publicized release dates, expressed as a percent of release dates
met. Qualitative or quantitative indicators may measure timeliness.
BTS
Goal - Reduce the lag time
in data reporting so that decision makers have a nearly “real-time” view of the
transportation system and factors affecting it.
BTS
Performance Indicators - BTS
has developed indicators for its major programs. Timeliness is evaluated through the following
measures:
- Improve average response time for general data
and information requests from customers both government and private sector (to
improve customer responsiveness).
- Increase the percentage of periodic data
releases (airline traffic and financial data, Air Travel Price Index, and
Transportation Services Index) that occurred by the schedule date (enable our
stakeholders and customers predictable delivery of our datasets).
- Ensure that the annual release of National
Transportation Atlas Database (NTAD) CD is available prior to the start of the
annual ESRI (Environmental Systems Research Institute) International User
Conference on GIS and mapping software (to provide timely support for research,
analysis, and decision-making across all modes of transportation).
- Reduce response time for map requests by the DOT Crisis Management Center (CMC) and in support of the Continuity of
Operations (COOP) plans (to improve emergency response by providing current and
timely information).
Cost
OMB
Standard – Cost is the
dollar amount used to produce data products or services. The development and
use of financial performance measures within the Federal Government is an established
goal, and the intent of such measures is to determine the “true costs” of
various programs or alternative modes of operation at the Federal level. Examples
of cost data include full costs of products or programs, return on investment, dollar
value of efficiencies, and ratios of cost to products distributed. Quantitative
indicators may measure cost.
BTS
Goal - Reduce the cost of
recurring programs in order to provide data and analysis in a cost-effective
manner.
BTS
Performance Indicators – The
BTS cost measures are:
- Improve
cost efficiencies in the production of BTS’ major data reference reports
(Transportation Statistics Annual Report, National Transportation Statistics,
and the Pocket Guide to Transportation Statistics).
- Improve
cost efficiencies in acquiring international freight data (Transborder data).
Dissemination
OMB
Standard – Dissemination is
the availability, accessibility, and distribution of information products and
services. Typical measures include: on-demand requests fulfilled, product downloads,
degree of accessibility, customer satisfaction with ease of use, number of
participants at user conferences, citations of agency data in the media, number
of Internet user sessions, number of formats in which data are available,
amount of technical support provided to data users, exhibits to inform the
public about information products, issuance of newsletters describing products,
usability testing of web sites, and assessing compliance with Section 508 of
the Rehabilitation Act, which requires Federal agencies to make their electronic
and information technology accessible to people with disabilities. Qualitative
or quantitative indicators may measure dissemination.
BTS Goal - To provide data and analysis products that are readily available, easily accessible, and widely
distributed.
BTS Performance Indicators – The BTS measures for dissemination are:
- Increase the number of National Atlas Transportation Database (NTAD) CD-ROMs distributed
(to broaden the customer base for this product).
- Increase the number of datasets downloaded from TranStats (to broaden customer usage for
this product).
Mission Achievement
OMB
Standard – Mission achievement is the effect of or satisfaction
with the statistical programs. For government statistical programs, this
dimension responds to the question—have we achieved our objectives and met the
expectations of our stakeholders? Under this dimension, statistical programs
document their contributions to the goals and missions of parent departments
and other agencies, the Administration, Congress, and information users in the
private sector and the general public. For statistical programs, this broad
dimension involves meeting recognized societal information needs and also
addresses the linkage between statistical outputs and programmatic outcomes.
However, identifying this linkage is far from
straightforward. It is frequently difficult to trace the effects of information
products on the public good. Such products often are necessary intermediate inputs
in the creation of high visibility information whose societal benefit is
clearly recognized. The statistics
produced by statistical agencies are used to track the performance of programs
managed by their parent or other organizations related to topics such as crime,
education, energy, the environment, health, science, and transportation.
Moreover, beyond the direct and focused uses
of statistical products and programs, the statistical agencies and their
products serve a diverse and dispersed set of data users working on a broad range
of applications. Users include government policy makers at the federal, state,
and local levels, business leaders, households, academic researchers, analysts
at public policy institutes and trade groups, marketers and planners in the
private sector, and many others. Information produced by statistical agencies
often is combined with other information for use in the decisionmaking process. Thus, the relationship between program
outputs and their beneficial uses and outcomes is often complex and difficult
to track. Consequently, agencies use both qualitative and quantitative indicators
to make this linkage as explicit as feasible.
Quantitative measures may be used to reflect
mission achievement. For example, customer satisfaction with the statistical agency
or unit indicates if the agency or unit has met the expectations of its
stakeholders. In the absence of preferred quantitative indicators, qualitative
narratives can indicate how statistical agency products contribute to and
evaluate progress toward important goals established for government.
BTS
Goal – Provide
(dissemination) high-quality data (accuracy) that fill transportation data gaps
and advance their effective (relevancy and timeliness) use for transportation
decisionmaking.
BTS
Performance Indicators - BTS
evaluates mission achievement through the following measures:
- Improve
coverage of legislative mandates in the Transportation Statistics Annual Report
(TSAR) (degree to which we support our legislative mandate).
- Track and increase the number of BTS products
that contribute to the accomplishment of important, established government
goals (meeting the agency mission).
Following is
a report of how well BTS has met its performance goals.
|
|