Evaluation of Performance Data

The USPTO reviews and evaluates its performance goals and measures based on customer, stakeholder, and employee input and the results of performance evaluations, as described below.

Customer, Stakeholder, and Employee Input. The USPTO has been a customer-driven organization for almost 10 years, when it began holding focus sessions with customers to determine their needs and expectations. Customer satisfaction efforts began by developing customer service standards relating to timeliness, accuracy, and responsiveness in delivering products and services. Next, customers were surveyed to establish baselines and subsequently measure progress toward customer satisfaction with the business lines in general and with specific goals and objectives. The USPTO recently completed its sixth annual survey of patent and trademark customers.

In 1998 and 1999 we collaborated with the National Academy of Public Administration to define and measure end outcomes of the U.S. patent and trademark systems. Independent inventors, patent and trademark practitioners, industry representatives, academicians, economists, USPTO senior executives, and experts in performance measurement participated in this exercise. The results were published and distributed to the participants in early FY 2000, and the results were used to make changes in operations and systems.

The USPTO also is committed to incorporating employee input into strategic and corporate planning processes. Employee satisfaction levels are one of the key components of the performance measures on USPTO’s balanced scorecards. Employee surveys were completed in both FY 2000 and FY 2001. These data have been used to develop initiatives to improve overall employee satisfaction. Several program evaluations were initiated or continued during FY 2001.

Baldrige Assessment. In January 2001, the USPTO conducted a second organization-wide self-assessment using the Baldrige criteria. Data were used to project key requirements for delivering ever-improving value to customers while maximizing overall effectiveness and productivity of the delivering organization. The participants met for four days and were divided into three teams where they collectively discussed each of the Baldrige criteria (i.e., Leadership, Strategic Planning, Customer and Market Focus, Information and Analysis, Human Resources, Process Management, and Business Results). After extensive discussion, each team identified several areas in which the USPTO had particular strengths and other areas where there were opportunities for improvement. Work will continue on implementing improvements as the USPTO continues its journey toward performance excellence.

Quality Reviews. The USPTO conducts ongoing reviews of the quality of patent and trademark examination. The focus of the review for patent applications is threefold: (1) identifying patentability errors, (2) assessing adequacy of the field of search and proper classification, and (3) assessing proper examination practice and procedures. For trademark applications, the review includes four areas: (1) substantive statutory criteria for registrability, (2) search for confusingly similar marks, (3) proper examination practice and procedures, and (4) proper application of judicial precedents. The information from these reviews helps the businesses identify necessary training with the goal of enhancing overall product quality and improving the consistency of examination. The results of these reviews provide analysis in the form of reports to patent and trademark management. These reports serve as a tool for educating examiners and examining attorneys. In addition to reporting specific errors, the analysis provides information on recurring problems and trends.

Management Control Reviews. The USPTO continued to promote work-at-home programs in 2001. The patent work-at-home (PWAH) pilot program was established in July 2001. The PWAH is being undertaken to improve the efficiency and effectiveness of our workforce, as well as improve employee recruitment and retention. The TWAH telecommuting program was expanded from 57 examining attorneys in FY 2000 to 89 examining attorneys in FY 2001, with a pilot that included paralegals. Measurements show that participants increased their productivity by increasing the number of hours worked on examinations.

The USPTO researched a number of alternative methods for assessing the productivity of our patent and trademark processes. A promising alternative is similar to the one used by the Bureau of Labor Statistics. This methodology was applied to a trademark productivity analysis, which examined changes in productivity from FY 1998 through FY 2001, and makes projections through FY 2008. The analysis examined the number of hours worked for all trademark employees and contractors and compared them with the number of trademark disposals. Results indicate that trademark productivity has remained about the same from FY 1998 to FY 2001; however, data also indicate trademark productivity is expected to increase steadily throughout the remainder of the study period. A similar methodology is currently under investigation for patents. While it is too soon to report definitive findings from these analyses, the methodology holds significant promise for showing productivity improvements as the USPTO transitions to a fully electronic organization.

Computer Security Initiatives. Recent initiatives taken to enhance information security include centralized user identification and password administration, upgraded physical access controls, streamlined security tasks with automated tools, and implementation of a campus-wide network intrusion detection system. The IT security Web page that provides information on USPTO's IT security policies, security awareness material, and virus information was enhanced, and accreditation of the IT infrastructure, PTOnet, and the Data Center was completed. Additionally, PKI, which facilitates secure communications and information processing for sensitive information, was deployed for the EFS and the PAIR system.

 Patent Application Publication No. US 2001/0046355 for an "optical fiber connector," filed electronically by IBM Corp. for Martin Schmatz of Switzerland (inventor), and published after 18 months. 

Review of Unconventional Threats to National Security. The USPTO has significantly enhanced its security efforts in response to the September 11, 2001, attacks on the United States. The agency is in a unique situation in that it occupies office space in 18 public buildings and is not the sole tenant of many of them. The USPTO is working with other federal agencies and property managers in Crystal City to enhance noncyber-based security. Some of the enhancements in place or in progress, include the following:

• Placed lobby level guards in all fully USPTO occupied buildings to check identification, packages, briefcases, etc.
• Increased roving guard patrols tenfold -- guard coverage now includes the parking garages, USPTO occupied space, stairways, building perimeters and all parking areas near the building.
• Ordered X-ray machines for mailrooms to inspect all incoming packages.
• Re-engineered USPTO’s Occupant Emergency Plan.
• Initiated weekly meetings with the Federal Bureau of Investigation and the Federal Protective Service.

Office of the Inspector General (OIG) Reviews. The OIG contributes to USPTO’s efforts to assure audit and evaluation coordination and coverage of USPTO goals. The OIG conducted the following types of audits and evaluations:

Financial Statement Audit. During FY 2000 financial statement audit, various tests and reviews of the primary accounting system and internal controls were conducted as required by the Chief Financial Officers' Act. In their FY 2000 and FY 2001 internal control report, the auditors reported no internal control material weaknesses or reported conditions. The auditors have issued an unqualified opinion on USPTO's FY 2000 and FY 2001 financial statements.

Program Evaluations. Several reviews of this type were conducted by the OIG. An example is the review of the USPTO performance measures included in the Department of Commerce’s FY 2000 Annual Performance Plan. The purpose of the review is to validate the measures and the data collection tools and methods. The results of the audit showed that management controls were in place and operating effectively regarding the collection, validation and reporting of performance measures. In addition, the report stated that the USPTO was committed to developing and producing quality performance measures. Several minor recommendations were reported and have subsequently been implemented by the USPTO.

Discontinued Goals and Measures. As a result of an assessment to focus performance goals and measures on core mission activities, the following goals and measures as stated will no longer be tracked as part of the Government Performance and Results Act. Many of the measures identified below will continue to be monitored as part of USPTO’s normal ongoing operations but will not be used to assess compliance with overall agency goals.

Intellectual Property Policy Goal: Strengthen intellectual property protection in the United States and abroad, making it more accessible, affordable and enforceable.

	FY 1999 Actual	FY 2000 Actual	FY 2001 Target	FY 2001 Actual
Measure: Increase in technical assistance to developing countries moving to a market economy – number of countries provided technical assistance.	93	126	125	78
Discussion: Target not met. Not able to recruit staff to conduct technical assistance as a result of hiring freeze in place for part of the fiscal year. Will recruit new staff in FY 2002 to increase technical assistance.
Measure: Increase in technical assistance to developing countries moving to a market economy – number of technical assistance activities completed.	99	106	105	88
Discussion: Target not met. Not able to recruit staff to conduct technical assistance as a result of hiring freeze in place for part of the fiscal year. Will recruit new staff in FY 2002 to increase technical assistance.

Patent Goal: Enhance the quality of patent products and services, transition to e-Government, and optimize patent processing time.

	FY 1999 Actual	FY 2000 Actual	FY 2001 Target	FY 2001 Actual
Measure: Percent of allowed applications where a significant question relating to quality of the examination process was raised.	13.9	7.7	7.0	5.1
Discussion: Target met.
Measure: Percent customer satisfaction with setting forth positions clearly in written communications.	63	63	66	62
Discussion: Target not met. The USPTO has been surveying customers of the patent process since 1995. Overall satisfaction remained virtually the same until 1998 with significant improvement in 1999 and 2000. Our survey contractors labeled this 5 percent increase "statistically significant." They have cautioned us that repeated "significant" increases in overall satisfaction are highly unusual.
Measure: Percent customer satisfaction with results of the search of prior art.	64	61	64	63
Discussion: Target not met. The USPTO has been surveying customers of the patent process since 1995. Overall satisfaction remained virtually the same until 1998 with significant improvement in 1999 and 2000. Our survey contractors labeled this 5 percent increase "statistically significant." They have cautioned us that repeated "significant" increases in overall satisfaction are highly unusual.
Measure: Percent customers satisfied with returning phone calls in one day.	57	61	66	64
Discussion: Target not met. The USPTO has been surveying customers of the patent process since 1995. Overall satisfaction remained virtually the same until 1998 with significant improvement in 1999 and 2000. Our survey contractors labeled this 5 percent increase "statistically significant." They have cautioned us that repeated "significant" increases in overall satisfaction are highly unusual.
Measure: Percent customers satisfied with directing callers to the proper office or person.	64	69	71	70
Discussion: Target not met. The USPTO has been surveying customers of the patent process since 1995. Overall satisfaction remained virtually the same until 1998 with significant improvement in 1999 and 2000. Our survey contractors labeled this 5 percent increase "statistically significant." They have cautioned us that repeated "significant" increases in overall satisfaction are highly unusual.
Measure: Average days to mail a filing receipt.	23	64	30	48
Discussion: Target not met. Transition from government FTE to contractor personnel took longer than planned. Contractor personnel will be fully trained and expected to meet FY 2002 targets.
Measure: Percent of filing receipts produced accurately.	73.3	80.5	84.0	89.2
Discussion: Target met.
Measure: Percent employee satisfaction on survey question, "how satisfied am I with my job?"	48	58	61	65
Discussion: Target met.
Measure: Rank in survey results of employee satisfaction in Government.	N/A	N/A	N/A	N/A
Discussion: Government-wide employee satisfaction survey not conducted.
Measure: Average number of first office actions and disposals (production units).	223,099	235,883	231,954	240,633
Discussion: Target met.
Measure: Number of patent disposals.	214,556	234,344	226,700	239,493
Discussion: Target met.
Measure: Percent applications receiving first office actions within 14 months of filing while factoring in term reductions.	83.1	81.2	78	74.3
Discussion: Target not met. The government-wide hiring freeze prevented the timely hiring of patent examiners. The hiring of patent examiners will enable us to meet our target.
Measure: Percent applications receiving actions after an applicant‘s amendment within four months.	97.4	98.3	98.0	98.1
Discussion: Target met.
Measure: Percent applications receiving actions after a Board decision within four months.	N/A	76.9	84.0	84.1
Discussion: Target met.
Measure: Percent applications granted within four months after issue fee payment.	N/A	89.1	87.0	92.2
Discussion: Target met.
Measure: Percent patents granted that do not qualify for term extension for exceeding 36 months.	N/A	N/A	86.0	87.2
Discussion: Target met.
Measure: Percent annual business return on e-Government initiatives.	N/A	N/A	N/A	N/A
Discussion: The TEAM system is the major measurable initiative in which the USPTO has launched a business return study in FY 2001. TEAM is planned for FY 2004 to provide an electronic record as the official legal record of patent application processing. Based on preliminary projections the USPTO will realize a net cost avoidance of $31.2 million over a six year period resulting in a return on investment of 18 percent, and a payback period of five years and five months.
Measure: Percent of patent applications filed electronically.	N/A	N/A	2	<1
Discussion: Target not met. EFS requires substantial up-front IT investment by the customers. The USPTO is exploring incentives for the customers who file electronically.
Measure: Percent of annual growth of external customers using USPTO e-Government systems.	N/A	N/A	10	387
Discussion: Target exceeded.
Measure: Percent of employees relying on USPTO e-Government environment to perform their work.	N/A	N/A	10	86
Discussion: Target exceeded.

Trademark Goal: Enhance the quality of trademark products and services, transition to e-Government, and optimize trademark processing time.

 Trademark Performance Measures

	FY 1999 Actual	FY 2000 Actual	FY 2001 Target	FY 2001 Actual
Measure: Percent of customers reporting satisfaction with clear written communication.	77	77	77	79
Discussion: Target met.
Measure: Percent of customers reporting satisfaction with correct information in the TMOG.	74	76	77	71
Discussion: Target not met. The process by which the TMOG is published was greatly enhanced in the last quarter of FY 2001. As a result the TMOG is now available in an "on-line" PDF searchable format directly from our Web site. This enhancement has enabled us to minimize errors. Results should be revealed in the next customer satisfaction survey.
Measure: Percent of customers satisfied with the office returning phone calls in one day.	59	53	65	55
Discussion: Target not met. All Trademark employees received customer service training in the second half of FY 2001. Emphasis included the importance of returning phone calls, and the timeframe in which we do so. Results of the improvement are expected in the next customer satisfaction survey.
Measure: Percent of customers satisfied with the delivery of filing receipts. Correct filing receipts mailed in one day for electronic filing.	N/A	100	93	81
Discussion: Target not met. Electronic filing receipts are returned in less than one day. Customers who have not filed electronically responded to the survey.
Measure: Percent of customers satisfied with the delivery of filing receipts. Correct filing receipts mailed in 14 days for paper filing.	33	27	85	43
Discussion: Target not met. Although we fell short of our target, we improved significantly over FY 2000. As we continue to encourage electronic filing, this measure is expected to decline significantly.
Measure: Rank in survey results of employee satisfaction as measured against other government agencies.	N/A	N/A	N/A	N/A
Discussion: Government-wide employee satisfaction survey not conducted.
Measure: Percent of trademark applications filed electronically.	8.3	14.9	30	24
Discussion: Target not met. The number of applications filed electronically continues to climb, even as the number of paper filings drops. Although we failed to reach our goal, economic factors such as those affecting the number of applications filed overall, impacted the results of this measure.
Measure: Percent customers communicating electronically in all aspects of correspondence.	N/A	N/A	N/A	N/A
Discussion: We have yet to identify an acceptable measure for this goal but continue to encourage electronic communication with our organization, as well as ways of measuring it.

< Previous Page | Next Page >