Detailed Information on the
Alcohol and Tobacco Tax and Trade Bureau Protect the Public Program Assessment

Establishing baselines and targets for new measures.

Establish a web-based "Flashboard" that will report TTB service standards, relevant performance measures, and other key bureau information such as workloads, so that stakeholders and customers can easily navigate from summary information to reports and/or other analytics.

Update TTB's web site to enhance search and navigation; improve content and format; redesign TTB.gov homepage; and employ plain language.

Implement new process to seek feedback through the use of customer satisfaction surveys for label and formula approval applications at the Alcohol, Labeling, and Formulation Division. Incorporate customer feedback to help ensure program services are prioritized to effectively target available resources, and ensure business processes operate effectively within prescribed standards of performance.

Refining performance measures to more accurately reflect the goals and achievements of the program.

Establishing clear guidelines and procedures to ensure that goals are very specific. Establishing written guidelines and supporting documentation for all aspects of the program.

Coordinating with other relevant federal agencies on consumer product safety labeling rules pertaining to consumable products.

Measure: Percent of electronically filed Certificate of Label Approvals applications.

Explanation:Calculated by dividing the number of e-filed applications by the total Certificate of Label Approval applications (COLA) submissions (paper and electronic).

Measure: Average number of days to process an original permit application at the National Revenue Center. (New measure, added February 2008)

Explanation:Permit applications are stamped when received and recorded when processed. The difference between the stamped date and the recorded datea is the amount of time used for calculating this metric. The NRC takes a simple average of the number of days it takes to process an original permit applications, during a specified period to calculate the results. "Processed" equals those permits issued, withdrawn, denied, or abandoned and includes Field Operations field time. "Processed" may also involve the date of a notice to "show cause." [Note: Considerable additional time is added to "Processed" time during the administrative process for denying an application.]

Measure: Percentage of instances where the utilization of International Trade Database System (ITDS) results in identifying importers without permits as a percentage of total permits on file at TTB's National Revenue Center. (New measure, added February 2008)

Explanation:The percentage of occurrences in which any individual or business importer has no known authorization (e.g., permit) to operate in the alcohol or tobacco industries in the United States, where instances in the ITDS fail to match those within the NRC's Integrated Revenue System.

Measure: National Revenue Center customer service survey results (New measure, added February 2008)

Explanation:The NRC conducts a customer survey to determine satisfaction levels among industry members. The survey's primary focus is to assess how well TTB helps applicants obtain a permit for starting a new business. The secondary focus is on claim processing.

Measure: National Revenue Center customer service survey results (New measure, added February 2008)

Explanation:The NRC conducts a customer survey to determine satisfaction levels among industry members. The survey's primary focus is to assess how well TTB helps applicants obtain a permit for starting a new business. The secondary focus is on claim processing.

Is the program purpose clear?

Explanation: Recent evaluations have examined the impact of USAID family planning (FP) assistance in Morocco over 30 years, Turkey over 12 years, and Brazil over eight years. The Morocco evaluation notes that "the collaboration between the MOH and USAID led to important results in FP indicators. [CPR]...rose from less than 20% in 1979-80 to 59% in 1997. [TFR] decreased dramatically from 7.0 children in 1979-80 to 3.1 children in 1997." (p.iv) The Turkey assessment notes accomplishments ranging from developing strategies for women's health and FP to securing contraceptives to strengthening training systems, to improving access and method choice, to monitoring and evaluation. The Brazil assessment notes that "USAID's program produced a number of significant achievements, many of which warrant replication throughout the country." (p.20)

Evidence: Congressional Justification, Strategic Plan, U.S. Code Title 27 (Intoxicating Liquors) specifically, chapter 8 - Federal Alcohol Administration Act, Internal Revenue Code, Subtitle E; Alcohol Beverage Labeling Act; Food, Drug and Cosmetic Act; Balanced Budget Act of 1997.

Does the program address a specific interest, problem or need?

Explanation: There is a need to ensure compliance with the Federal Alcohol Administration Act (FAA) and the Internal Revenue Code and protect the integrity of the alcohol products on the market through inspection of manufacturing sites and review of labels and formulas.

Evidence: There are instances of products found in the marketplace that contain unacceptable levels of pesticides, sulfites, methanols. There have also been cases of deceptive and fraudulent labeling and advertising of alcohol products. Although ATF is unable to provide a historical assessment of the magnitude of the problem, it tracks staff workload distribution and findings in its monthly metric report. In 2001, for example, 645 items were tested and 214 failures were found. Failures range from label deficiencies to overproof, unacceptable levels of methanols. Further, approximately 11% of individuals applying for permits are denied for reasons that may include criminal history.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: ATF's Alcohol and Tobacco consumer product safety activities impact interstate and international commerce. While these activities comprise a relatively small percentage of ATF's overall resources and may not be considered "significant" ATF is the only entity that oversees the integrity and safety of alcoholic products. The programs are designed to assure the integrity of the products, market, and individuals associated with the alcohol and tobacco industries through inspection, investigation and laboratory analysis.

Evidence: ATF spends approximately 2.8% percent ($21 M and 142 FTEs in FY 2002 and est. $23M and 142 FTEs for FY 2003) of its resources on consumer protection activities. To ensure consumer safety of alcoholic beverages, 100% of all credible information suggesting serious public health threats are investigated. In addition, product integrity inspections are conducted for high risk label claims, and alcohol beverages are secured from the marketplace for sampling purposes. The integrity of the market is assured through monitoring and investigation of trade practices and advertising. Undesirable elements including suspected terrorists are prevented from entry into the alcohol and tobacco industries through review and investigation of permit applications, of which approximately 11% are withdrawn, abandoned or denied.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: Although ATF's Alcohol Labeling and Formulation Division and National Laboratory Center work closely with the Food and Drug Administration, MOUs and specific authorities provided under the FAA delineate roles to avoid redundancy.

Evidence: Program are clearly defined in ATF orders. Existing MOUs with other Federal agencies have been or are being established to clearly defined roles. ATF possesses unique ability to provide technical assistance on alcohol beverage issues internationally.

Is the program optimally designed to address the interest, problem or need?

Explanation: While ATF continually evaluates risk associated with programs, ATF has only recently begun to develop formal performance measures for consumer protection programs. Programs have been subject to independent OIG reviews where no materials weaknesses were noted. Given that there is no conclusive evidence that another approach or method would be more efficient/effective, we are inclined to answer in the affirmative.

Evidence: None

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ATF has processes set up to develop annual plans for field and business centers activities, which are linked to ATF's strategic goals. ATFs senior executive led Strategic Leadership Team (SLT) sets the bureau's strategic plan. Detailed priority driven operating plans are developed in support of the strategic plan by program offices (e.g. Alcohol and Tobacco). These are used by field operations in developing detailed implementing plans, which are monitored by Headquarters and field supervisors. In support of this, long-term IT goals are developed by the executive led Information Resource Management Council (IRMC). The IRMC makes recommendations to the SLT. The Alcohol and Tobacco Corporate Board (ATCB) is a program office level version of the SLT. The ATCB has developed multi-year goals for increasing effectiveness and efficiency of alcohol and tobacco programs including technology advancements (e.g. chemically authenticate wine label claims).

Evidence: Long term performance goals and performance measures are delineated in: ATCB developed Strategic Goals document; the annual Alcohol and Tobacco operating plan; Certificate of Label Approvals (COLA) Online business case; systems integration business case; and ATF orders (e.g. Alcohol Beverage Sampling Program Procedural Guidelines and Adverse Action & Assessment Case Processing Guidelines). The performance goals established through FY 2008 include 1) increase the percentage of COLAs processed within 9 days to 70% of 2008; 2) 75% of label approval applications submitted and processed electronically; 3) Investigation of 100% of all credible information suggesting a serious threat to public health regarding alcohol products

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: Annual performance goals are clearly delineated in the ATCB Strategic Goal listing and the operating plan instructions to field operations, The implementing plans are reviewed by headquarters and field supervisors prior to adoption.

Evidence: Headquarters and field managers monitor performance on a regular basis to ensure progress is being demonstrated. Examples of annual goals are: 1) % of label approval applications submitted and processed electronically -- from 10% in FY 2003 to 75% in FY 2008; 2) Responses to unsafe products and product deficiencies (alcohol), constant at 190 through 2003, including investigation of 100% of all credible information suggesting a serious threat to public health regarding alcohol products. Additionally, 33% of all samples tested were found to be deficient regarding label claims. This number is expected to increase due to improvements in the alcohol beverage sampling targeting.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: Although ATF's planning process is highly collaborative and may involve State partners that share similar goals of product assurance and public safety, the States' commitment to the program's annual and long-term goals is not formalized.

Evidence:  

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: ATF uses innovative partnerships both internally and externally.

Evidence: ATF has partnerships with Customs, FDA, FTC though MOUs. Additionally, there are numerous instances of partnerships with state agencies to achieve common goals.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: There are sporadic GAO reports and OIG audits, but these generally focused on ATF's excise taxation and firearms related programs. While ATF is subject to independent customer service surveys, these surveys are neither sufficiently comprehensive nor specific to allow ATF to make significant improvements in its program management.

Evidence:  

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: In 1998, ATF restructured its financial/accounting systems to better align with its strategic goals.

Evidence: The ATF Congressional Justification, Accountability Report, regular meeting with its Strategic Leadership Team and Resource Management Committee.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: ATF has a system for identifying and correcting deficiencies in the strategic planning process.

Evidence: ATF's senior executive led Strategic Leadership Team (SLT) sets the bureau's strategic plan and meets on a regular basis to address strategic planning issues, including deficiencies. In support of this, long-term IT goals are developed by the executive led Information Resource Management Council (IRMC). The IRMC makes recommendations to the SLT and identifies potential strategic planning deficiencies in the IT arena. The Alcohol and Tobacco Corporate Board (ATCB) is a program office level version of the SLT. The ATCB has developed multi-year goals for increasing effectiveness and efficiency of alcohol and tobacco programs. Regular strategic planning meetings are held by both the ATCB and SLT to address any strategic planning deficiencies.

Are all regulations issued by the program/agency necessary to meet the stated goals of the program, and do all regulations clearly indicate how the rules contribute to achievement of the goals?

Explanation: ATF has a process whereby various directorates meet on a regular basis. The annual plan, including alcohol and tobacco strategic goals and priorities, is developed with program office and field operations, and is confirmed with all field managers prior to implementation. Additionally, managers' performance appraisals are integrated with achieving the goals set forth in the strategic plan, and specifically, Alcohol and tobacco strategic goals as published on ATF's intranet.

Evidence: The annual plan, including alcohol and tobacco strategic goals and priorities, is developed with program office and field operations, and is confirmed with all field managers prior to implementation. Alcohol and tobacco strategic goals and priorities are monitored and reported on internal monthly issues reports, as are industry metrics in monthly Metric Reports, and the status of field inspection programs through data mining of the real time information in ATF's case management system.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ATF has one efficiency measure for internal use related to its laboratory work response rate for alcohol and tobacco samples. However, internally, ATF tracks resource allocation throughout the year and when the allocation is inconsistent, program managers can isolate and analyze the situation. Further, to ensure accountability, the managers appraisal includes references to the annual operations plan, which is aligned with the annual performance plan.

Evidence: 1) Manager's appraisal template 2) Annual operations and implementation plan, 3))ATF's target of 50% performance based contracts by 2005.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: Fund for these programs are allocated to program divisions and are obligated for their intended purpose in a timely manner. In instances when there are potential problems, ATF's budget execution team works with the program office and Treasury to develop reprogramming proposals for Congress with OMB approval.

Evidence: ATF's financial management system allows for regular tracking of spending by project code, Regular reviews of spending are conducted. Treasury Annual Report, SF 132 (Apportionment), SF 133 (Report on Budget Execution and Budgetary Resources).

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Fund for these programs are allocated to program divisions and are obligated for their intended purpose in a timely manner. In instances when there are potential problems, ATF's budget execution team works with the program office and Treasury to develop reprogramming proposals for Congress with OMB approval.

Evidence: ATF's financial management system allows for regular tracking of spending by project code, Regular reviews of spending are conducted. Treasury Annual Report, SF 132 (Apportionment), SF 133 (Report on Budget Execution and Budgetary Resources).

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ATF has processes set up to develop annual plans for field and business centers activities, which are linked to ATF's strategic goals. Detailed operating plans are developed in support of the strategic plan by program offices (e.g. Alcohol and Tobacco). These are used by field operations in constructing detailed implementing plans, which are monitored by Headquarters and field supervisors.

Evidence: The program utilizes business cases for all major IT investments, which include performance goals and measures for improving efficiency and lowering costs. Currently there are two IT investment business cases for the COLA Online project and the Integrated Revenue Information System, both of which have performance expectations. For example the COLA Online project identifies the goal to reduce annual contracting costs by 50% by the third year after deployment of the new application.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: ATF utilizes a project cost accounting system to capture direct and indirect program /project cost. However, ATF does not capture full pension accrual costs.

Evidence: ATF's cost model estimates full costs including benefits and overhead associated with personnel and programs, but excludes pension accruals.

Does the program use strong financial management practices?

Explanation: ATF has had six consecutive unqualified audit opinions with no material weaknesses reported by auditors.

Evidence: Unqualified audit opinions by OIG.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: ATF has had a good record of responding to OIG recommendations and improving its management deficiencies consistent with the President Management Agenda such as aligning its programs and budget to better assess full costs of program. However, ATF will need to continue this record by strengthening the linkage between the cost information and program performance and in such areas as competitive sourcing.

Evidence: Congressional Justification, regular tracking of OIG Review corrective actions.

Did the program seek and take into account the views of affected parties including state, local and tribal governments and small businesses, in drafting significant regulations?

Explanation: REMOVE

Evidence:  

Did the program prepare, where appropriate, a Regulatory Impact Analysis (RIA) that comports with OMB's economic analysis guidelines and have these RIA analyses and supporting science and economic data been subjected to external peer review by qualified specialists?

Explanation:  

Evidence:  

Does the program systematically review its current regulations to ensure consistency among all regulations in accomplishing program goals?

Explanation:  

Evidence:  

In developing new regulations, are incremental societal costs and benefits compared?

Explanation:  

Evidence:  

Did the regulatory changes to the program maximize net benefits?

Explanation:  

Evidence:  

Does the program impose the least burden, to the extent practicable, on regulated entities, taking into account the costs of cumulative final regulations?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: To ensure the safety of the consumer regarding alcoholic beverages, 100% of all credible information suggesting a serious threat to public health are investigated. In addition, product integrity inspections are conducted for high risk label claims, and alcohol beverages are secured from the marketplace for sampling purposes. The integrity of the market is assured through monitoring and investigation of trade practices and advertising. Undesirable elements are prevented from entry into the alcohol and tobacco industries through review and investigation of permit applications, of which approximately 11% are withdrawn, abandoned or denied. ATF is still actively improving its measures in this program.

Evidence: GPRA report, Congressional Justifications, regular reporting of information in monthly metric reports of case statistics and status, and in the alcohol and tobacco monthly issues report of progress towards achieving outcome goals established in the alcohol and tobacco strategic goals document.

Does the program (including program partners) achieve its annual performance goals?

Explanation: ATF has numerous performance goals, however ATF is still in the process of developing formal performance measures to capture outcomes.

Evidence: GPRA report

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: ATF has a number of effectiveness measures, efficiency measures, and established milestones for internal use related to program performance as are delineated in the strategic planning document.

Evidence: The program utilizes business cases for all major IT investments, which include performance goals and measures for improving efficiency and lowering costs. Currently there are two IT investment business cases for the COLA Online project and the Integrated Revenue Information System, both of which have performance expectations. For example the COLA Online project identifies the goal to reduce annual contracting costs by 50% by the third year after deployment of the new application. An additional example of improved efficiencies is evidenced in the implementation of the recommendations of the Beverage Alcohol Streamlining Team (BAST), which reengineered the workflow and processes regarding product compliance based on a comprehensive review of the existing processes.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: ATF emphatically believes that its mission is unique because no other Federal entity protects the integrity of alcohol in the same capacity. However, there are other Federal entities such as the Food and Drug Administration (FDA) that has a similar program purpose of product assurance and safety that merits examination. FDA measures such as its inspection performance gap (percent of various entities inspected within x years) is an example of a transparent measure that could benefit ATF's ability to assess its own performance.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: There are no regularly scheduled independent and quality evaluations that is both comprehensive and able to indicate definitively that the program is effective and achieving results.

Evidence:  

Were programmatic goals (and benefits) achieved at the least incremental societal cost and did the program maximize net benefits?

Explanation:  

Evidence: