This is the accessible text file for GAO report number GAO-08-97T entitled 'Elections: Further Testing Could Provide Increased but Not Absolute Assurance That Voting Systems Did Not Cause Undervotes in Florida's 13th Congressional District' which was released on October 3, 2007. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Statement before the Task Force on Florida-13, Committee on House Administration, House of Representatives: For Release on Delivery: Expected at 4:00 p.m. EDT: Tuesday, October 2, 2007: Elections: Further Testing Could Provide Increased but Not Absolute Assurance That Voting Systems Did Not Cause Undervotes in Florida's 13th Congressional District: Statement of Dr. Nabajyoti Barkakati: Senior-Level Technologist: Applied Research and Methods: GAO-08-97T: GAO Highlights: Highlights of GAO-08-97T, a statement before the Task Force on Florida- 13, Committee on House Administration, House of Representatives. Why GAO Did This Study: In November 2006, about 18,000 undervotes were reported in Sarasota County in the race for Florida’s 13th Congressional District (FL-13). After the contesting of the election results in the House of Representatives, the task force unanimously voted to seek GAO’s assistance in determining whether the voting systems contributed to the large undervote in Sarasota County. GAO agreed with the task force on an engagement plan, including the following review objectives: (1) What voting systems were used in Sarasota County and what processes governed their use? (2) What was the scope of the undervote in Sarasota County in the general election? (3) What tests were conducted on the voting systems in Sarasota County prior to the general election and what were the results of those tests? (4) Considering the voting systems tests conducted after the general election, are additional tests needed to determine whether the voting systems contributed to the undervote? To conduct its work, GAO met with officials from the State of Florida, Sarasota County, and Election Systems and Software (ES&S)—the voting systems manufacturer—and reviewed voting systems test documentation. GAO analyzed election data to characterize the undervote. On the basis of its assessments of prior testing and other activities, GAO identified potential additional tests for the Sarasota County voting systems. What GAO Found: In the 2006 general election, Sarasota County used voting systems manufactured by ES&S, specifically iVotronic direct recording electronic (DRE) voting systems during early and election day voting and the Unity election management system, which handles the election administration functions, such as ballot design and election reporting. GAO’s analysis of the 2006 general election data from Sarasota County did not identify any particular voting machines or machine characteristics that could have caused the large undervote in the FL-13 race. The undervotes in Sarasota County were generally distributed across all machines and precincts. GAO’s analysis found that some of the prior tests and reviews conducted by the State of Florida and Sarasota County provide assurance that certain components of the voting systems in Sarasota County functioned correctly, but they are not enough to provide reasonable assurance that the iVotronic DREs did not contribute to the undervote. Specifically, GAO found that assurance is lacking in three areas, and proposes that tests be conducted to address those areas. First, because there is insufficient assurance that the firmware in all the iVotronic DREs used in the election matched the certified version held by the Florida Division of Elections, GAO proposes that a firmware verification test be conducted on a representative sample of 115 (of the 1,499) machines that were used in the general election. Second, because an insufficient number of ways to select a candidate in the FL-13 race were tested, GAO proposes that a test be conducted to verify all 112 ways that GAO identified to select a candidate. Third, because no prior tests were identified that address the effect of a miscalibrated iVotronic DRE on the undervote, GAO proposes that an iVotronic DRE be deliberately miscalibrated to verify the accurate recording of ballots under these conditions. GAO expects these three tests would take 2 weeks, once the necessary arrangements are made. Should the task force ask GAO to conduct the proposed tests, several matters would need to be addressed before testing could begin, including obtaining access to the iVotronic DREs that have been subject to a sequestration order, arranging for a test site, obtaining some commercially available test tools, developing test protocols and detailed test procedures, and arranging for the video recording of the tests. Sarasota County election officials have indicated that they can help GAO access the machines and provide a test site between November 26 and December 7, 2007. Although the proposed tests could help provide increased assurance, they would not provide absolute assurance that the iVotronic DREs did not cause the large undervote in Sarasota County. The successful conduct of the proposed tests could reduce the possibility that the voting systems caused the undervote and shift attention to the possibilities that the undervote was the result of intentional actions by voters or voters that did not properly cast their votes on the voting system. To view the full product, including the scope and methodology, click on [hyperlink, http://GAO-08-97T]. For more information, contact Keith Rhodes at (202) 512-412 or rhodesk@gao.gov, or Naba Barkakati at (202) 512-4499 or arkakatin@gao.gov. [End of section] Mr. Chairman and Members of the Task Force: I am pleased to appear before the task force today to present the findings on our review of voting equipment used in Florida's 13th Congressional District (Florida-13), which we are conducting in response to your request of May 25, 2007. In November 2006, about 18,000 undervotes were reported in Sarasota County in the race for Florida's 13th Congressional District.[Footnote 1] Following the contesting of the election results in the House of Representatives, the task force met and unanimously voted to seek GAO's assistance in determining whether the voting systems contributed to the large undervote in Sarasota County. On June 14, 2007, we met with the task force and agreed upon an engagement plan, which included the following review objectives: (1) What voting systems and equipment were used in Sarasota County and what processes governed their use? (2) What was the scope of the undervote in Sarasota County in the general election? (3) To what extent were tests conducted on the voting systems in Sarasota County prior to the general election and what were the results of those tests? (4) Considering the tests that were conducted on the voting systems from Sarasota County after the general election, are additional tests needed to determine whether the voting systems contributed to the undervote? To conduct our work, we met with officials from the Sarasota County Supervisor of Elections, the Florida Department of State and Division of Elections, and Election Systems and Software (ES&S), the manufacturer of the voting systems used in Sarasota County. We reviewed voting system documentation, including standards documents, audit and testing documentation, submissions from the contestant and contestee, and selected Florida election laws and rules. In Sarasota County, election officials demonstrated how the ES&S voting system was used to support the 2006 general election. To determine the scope of the undervote in Sarasota County, we collected election data from the Supervisor of Elections and analyzed it to determine whether the undervote could be attributed to particular voting machines or machine characteristics. Specifically, we examined ballot image logs and event logs from the voting systems and technician and incident reports generated by elections staff from Sarasota County on election day. We also conducted various statistical analyses to characterize the undervote and to identify whether a subset of machines or precincts may have caused the large undervote. We reviewed test documentation and interviewed officials involved with testing from ES&S, the Florida Division of Elections, and the Sarasota County Supervisor of Elections. To determine the need for additional tests, we also reviewed the tests conducted following the election, including those conducted or sponsored by the Florida Division of Elections, including the parallel testing, the examination of Sarasota County's election procedures and practices, and the source code review conducted at Florida State University's Security and Assurance in Information Technology (SAIT) laboratory. We reviewed the final reports of these tests and also met with the leader of the source code review team. Following the agreement to and execution of a non-disclosure agreement with the Florida Department of State and ES&S, we obtained access to the iVotronic source code and reviewed it to further our understanding of the system and to verify some of the source code review's findings. We analyzed the available information and identified a key set of voting system objectives that, if implemented properly, would provide reasonable assurance that the voting systems did not malfunction and cause the large undervote in Sarasota County. Using these objectives, we used the results of testing previously conducted and assessed the extent to which these key voting system objectives could be met. For those objectives that could not be adequately assured, we assessed the significance of those objectives and identified tests that could be conducted to help try to assure those key voting system objectives were met. For each test, we identified resources that would be required, including time and manpower. We provided a draft of this report to the Florida Department of State, ES&S, and the Sarasota County Supervisor of Elections for their review and comments. The Florida Department of State and ES&S also conducted a sensitivity review to ensure that business proprietary information is not disclosed in this statement. We conducted our work from June to September 2007 in Washington, D.C.; Tallahassee and Sarasota, Florida; and Omaha, Nebraska. Results in Brief: In the 2006 general election, Sarasota County used voting systems manufactured by ES&S, specifically iVotronic direct recording electronic (DRE) voting systems during early and election day voting and the Unity election management system, which handles the election administration functions, such as ballot design and election reporting. Our independent analysis of the 2006 general election data from Sarasota County confirmed the large undervote in the race for Florida's 13th Congressional District, but did not identify any particular voting machines or machine characteristics that could have caused the large undervote in the election. The undervotes in Sarasota County for the congressional race were generally distributed across all machines and precincts. We found that some of the prior tests and reviews provide assurance that the voting systems in Sarasota County functioned correctly, but they are not enough to provide reasonable assurance that the iVotronic DRE voting systems did not contribute to the undervote. For example, prior reviews provide reasonable assurance that the Unity election management system did not contribute to the undervote, and the votes captured by iVotronic DREs at the precincts match the voter count from precinct records within acceptable margins of error. Portions of the Florida state audit, such as the firmware comparison and parallel tests, provided useful information, but the results could not be applied to all the iVotronic DREs used in the election because the number of machines tested was too small. Additionally, the machines were not tested for all different ways a voter can select a candidate in the congressional race. We also did not find any prior testing that would help us understand the effects of a miscalibrated touch screen. To address these issues, we propose that (1) a firmware verification test, (2) a ballot test, and (3) a calibration test be conducted to try to obtain further assurance that the iVotronic DREs used in Sarasota County during the 2006 general election did not cause the undervote. The firmware verification test would compare the firmware in a representative sample of iVotronic DREs with the certified version of firmware. The ballot test would exercise 112 ways to select a candidate on 10 iVotronic DREs. The calibration test would deliberately miscalibrate an iVotronic DRE that uses the certified software and verify the functioning of the machine. We expect the testing would take 2 weeks using a staff of about 6 to 8 people, once the necessary arrangements have been made. Although the proposed tests would provide increased assurance, they would not conclusively eliminate the machines as a cause of the undervote. Before commencing the testing, we would need to obtain access to the iVotronic DREs that have been subject to a sequestration order in the state court system of Florida, arrange for a test site, obtain some commercially available software and hardware for the firmware comparison test, develop test protocols and detailed test procedures, and arrange for video recording of the test. Sarasota County election officials have indicated that working around the county's election schedules, they could help us access the machines and provide a test site between November 26 and December 7, 2007. Our proposed tests could help reduce the possibility that the undervote was caused by the iVotronic DREs. However, even after completing the tests, we would not have absolute assurance that the iVotronic DREs did not play any role in the large undervote. Absolute assurance is impossible to achieve because we are unable to recreate the conditions of the election in which the undervote occurred. By successfully conducting the proposed tests, we could reduce the possibility that the iVotronic DREs were the cause of the undervote and shift attention to the possibilities that the undervote was the result of intentional actions by the voter or voters that did not properly cast their votes on the voting system. Draft copies of this statement were provided to the Secretary of State of Florida, the Supervisor of Elections of Sarasota County, and ES&S for their review and comment. The Florida Department of State provided technical comments, which we incorporated. The Sarasota County Supervisor of Elections did not provide us comments. In its comments, ES&S stated that it believes that the collective results of prior testing have demonstrated that the voting systems worked properly in Florida's 13th Congressional District race, and that the focus should be on testing the effect of the ballot display on the undervote. We disagree that the prior test results adequately demonstrate that the voting systems could not have contributed to the undervote. Our analysis identified three areas where further testing could provide increased assurance that the undervote was not caused by the voting systems. We agree with ES&S that the large undervote in Florida's 13th Congressional District race could have been caused by voters who intentionally undervoted or voters who did not properly cast their ballots, potentially because of issues related to the human interaction with the ballot. However, our review focused on whether the voting systems could have contributed to the large undervote. ES&S also provided technical comments, which we incorporated as appropriate. Background: The 13th Congressional District of Florida comprises DeSoto, Hardee, Sarasota, and parts of Charlotte and Manatee Counties. In the November 2006 general election, there were two candidates in the race to represent the 13th Congressional District: Vern Buchanan, the Republican candidate, and Christine Jennings, the Democratic candidate. The State of Florida certified Vern Buchanan the winner of the election. The margin of victory was 369 votes out of a total of 238,249 votes counted. Table 1 summarizes the results of the election and shows that the results from Sarasota County exhibited a significantly higher undervote rate than in the other counties in the congressional district. Table 1: Results from 2006 General Election for Florida Congressional District 13: County: Charlotte; Buchanan: 4,460; Jennings: 4,277; Undervotes: 225; Total ballots cast: 8,962; Percentage undervote: 2.51. County: DeSoto; Buchanan: 3,471; Jennings: 3,058; Undervotes: 142; Total ballots cast: 6,672; Percentage undervote: 2.13. County: Hardee; Buchanan: 2,629; Jennings: 1,686; Undervotes: 269; Total ballots cast: 4,584; Percentage undervote: 5.87. County: Manatee; Buchanan: 50,117; Jennings: 44,432; Undervotes: 2,274; Total ballots cast: 96,828; Percentage undervote: 2.35. County: Sarasota; Buchanan: 58,632; Jennings: 65,487; Undervotes: 18,412; Total ballots cast: 142,532; Percentage undervote: 12.92. County: Total; Buchanan: 119,309; Jennings: 118,940; Undervotes: 21,322; Total ballots cast: 259,578; Percentage undervote: [Empty]. Source: GAO analysis of Florida Division of Elections, Charlotte County, DeSoto County, Hardee County, Manatee County, and Sarasota County data. Note: Numbers do not add up because of overvotes - where voters select more than the maximum number of candidates allowed in a race; in this case, a ballot that had votes for both Buchanan and Jennings. [End of table] In Florida, the Division of Elections in the Secretary of State's office helps the Secretary carry out his or her responsibilities as the chief election officer. The Division of Elections is responsible for establishing rules governing the use of voting systems in Florida. Voting systems cannot be used in any county in Florida until the Florida Division of Elections has issued a certification of the voting system's compliance with the Florida Voting System Standards.[Footnote 2] The Florida Voting Systems Certification program is administered by the Bureau of Voting Systems Certification in the Division of Elections. An elected supervisor of elections is responsible for implementing elections in each county in Florida in accordance with Florida election laws and rules. The supervisor of elections is responsible for the purchase and maintenance of the voting systems as well the preparation and use of the voting systems to conduct each election. Sarasota County Used ES&S Voting Systems in 2006 General Elections: In the 2006 general election, Sarasota County used voting systems manufactured by ES&S. The State of Florida has certified different versions of ES&S voting systems. The version used in Sarasota County was designated ES&S Voting System Release 4.5, Version 2, Revision 2, and consisted of iVotronic DREs, a Model 650 central count optical scan tabulator for absentee ballots, and the Unity election management system. It was certified by the State of Florida on July 17, 2006. The certified system includes different configurations and optional elements, several of which were not used in Sarasota County. The election management part of the voting system is called Unity; the version that was used was 2.4.4.2. Figure 1 shows the overall election operation using the Unity election management system and the iVotronic DRE. Figure 1: Overview of Election Operation Using the Unity Election Management System and iVotronic DRE: [See PDF for image] This figure is an illustration of flow of election operations, as follows: Election Data Manager: Program, store and format election data for a jurisdiction; flows to: Ballot Image Manager: Create paper ballot layouts and create information for personalized electronic ballot to be used in iVotronic DREs; flows to: Hardware Programming Manager: Copy election definitions for iVotronic DREs into PEBs; flows to: Precinct Count (iVotronic DRE): iVotronic DREs are used to capture votes and tabulate votes for a precinct; flows to: Data Access Manager, then to Election Reporting Manager, then to election reports. Precinct Count (iVotronic DRE) also: Copy precinct-level votes from PEBs into Election Reporting Manager and ballot images and event logs from compact flash cards. Source: GAO. [End of figure] Sarasota County used iVotronic DREs for early and election day voting. Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware version 1.1 with firmware version 8.0.1.2.[Footnote 3] Some of the iVotronic DREs are configured with Americans with Disabilities Act (ADA) functionality, which includes the use of audio ballots. The iVotronic DRE uses a touch screen--a pressure-sensitive graphics display panel--to display and record votes (see fig. 2). Figure 2: The iVotronic DRE Voting System and Its Components. [See PDF for image] This is a photograph of the iVotronic DRE Voting System, highlighting the following components: connection to communications pack and printer (removed during voting); Vote button; Personalized electronic ballot; and Touch screen. Source: GAO. [End of figure] The machine has a storage case that also serves as the voting booth. The operation of the iVotronic DRE requires using a personalized electronic ballot (PEB), which is a storage device with an infrared window used for transmission of ballot data to and from the iVotronic DRE. The iVotronic DRE has four independent flash memory modules, one of which contains the program code--firmware--that runs the machine and the remaining three flash memory modules store redundant copies of ballot definitions, machine configuration information, ballots cast by voters, and event logs. The iVotronic DRE includes a VOTE button that the voter has to press to cast a ballot and record the information in the flash memory. The iVotronic DRE also includes a compact flash card that can be used to load sound files onto iVotronic DREs with ADA functionality. The iVotronic DRE's firmware can be updated through the compact flash card. Additionally, at the end of polling, the ballots and audit information are to be copied from the internal flash memory module to the compact flash card. To use the iVotronic DRE for voting, a poll worker activates the iVotronic DRE by inserting a PEB into the PEB slot after the voter has signed in at the polling place. After the poll worker makes selections so that the appropriate ballot will appear, the PEB is removed and the voter is ready to begin using the system. The ballot is presented to the voter in a series of display screens, with candidate information on the left side of the screen and selection boxes on the right side (see fig. 3). Figure 3: Second Ballot Page Showing the Congressional and Gubernatorial Races in Sarasota County's 2006 General Election: [See PDF for image] This figure is an illustration of the ballot used in the general election. Source: Sarasota County Supervisor of Elections. [End of figure] The voter can make a selection by touching anywhere on the line, and the iVotronic DRE responds by highlighting the entire line and displaying an X in the box next to the candidate's name. The voter can also change his or her selection by touching the line corresponding to another candidate or by deselecting his or her choice. "Previous Page" and "Next Page" buttons are used to navigate the multipage ballot. After completing all selections, the voter is presented with a summary screen with all of his or her selections (see fig. 4). From the summary screen, the voter can change any selection by selecting the race. The race will be displayed to the voter on its own ballot page. When the voter is satisfied with the selections and has reached the final summary screen, the red VOTE button is illuminated, indicating the voter can now cast his or her ballot. When the VOTE button is pressed, the voting session is complete and the ballot is recorded on the iVotronic DRE. In Sarasota County's 2006 general election, there were nine different ballot styles with between 28 and 40 races, which required between 15 and 21 electronic ballot pages to display, and 3 to 4 summary pages for review purposes. Figure 4: First Summary Page in Sarasota County's 2006 General Election: [See PDF for image] This figure is an illustration of the summary ballot used in the general election. Source: Sarasota County Supervisor of Elections. [End of figure] Analysis of Election Data Shows that Undervote Was Distributed across All Machines and Precincts: Our analysis of the 2006 general election data from Sarasota County does not identify any particular voting machines or machine characteristics that could have caused the large undervote in Florida's 13th Congressional District race. The undervotes in Sarasota County for the congressional race were generally distributed across all machines and precincts. Using voting system data that we obtained from Sarasota County, we found that 1,499 iVotronic DREs recorded votes in the 2006 general election; 84 iVotronic DREs recorded votes during early voting, and 1,415 iVotronic DREs recorded votes on election day.[Footnote 4] Using these data, we verified that the vote counts for the contestant, contestee, and undervotes match the reported vote totals for Sarasota County in Florida's 13th Congressional District race. As can be seen in table 2, the undervote rate in early voting was significantly higher than in election day voting.[Footnote 5] Table 2: Undervotes in Florida's 13th Congressional District Race during Early and Election Day Voting: Machines: All voters: 1,499; Early voters: 84; Election day voters: 1,415. Ballots cast: All voters: 119,919; Early voters: 30,877; Election day voters: 89,042. Undervotes: All voters: 17,846; Early voters: 5,445; Election day voters: 12,401. Undervote rate: All voters: 14.88%; Early voters: 17.63%; Election day voters: 13.93%. Source: GAO analysis of Sarasota County data. [End of table] The range of the undervote rate for all machines was between 0 and 49 percent, with an average undervote rate of 14.3 percent. When just the early voting machines are considered, the undervote rate ranged between 5 and 28 percent. The largest number of undervotes cast on any one machine on election day was 39. While the range of ballots cast on any one machine on election day was between 1 and 121, the median number of ballots cast on any one machine was 66. The range of undervote rate by precinct was between 0 and 41 percent, and the average undervote by precinct was about 14.8 percent. Prior Tests and Reviews Provide Some Assurance, but Do Not Provide Reasonable Assurance That the iVotronic DREs Did Not Contribute to the Undervote: Prior to the elections, Sarasota County's voting systems were subjected to several different tests that included testing by the manufacturer, certification testing by the Florida Division of Elections, testing by independent testing authorities, and logic and accuracy testing by Sarasota County's Supervisor of Elections. After the 2006 general election, an audit of Sarasota County's election was conducted by the State of Florida that included a review of the iVotronic source code, parallel tests, and an examination of Sarasota County's election procedures. Although these tests and reviews provide some assurance, as do certain controls that were in place during the election, that the voting systems in Sarasota County functioned correctly, they do not provide reasonable assurance that the iVotronic DREs did not contribute to the undervote. Prior Tests and Reviews of Sarasota County's Voting Systems Provide Useful Information, but Have Some Shortcomings: According to ES&S officials, ES&S tested the version of the iVotronic DRE that was used in Sarasota County in 2001-2002, but they could not provide us documentation for those tests because the documentation had not been retained. The Florida Division of Elections conducted certification testing of the iVotronic DRE and the Unity election management system before Sarasota County acquired the system from the manufacturer. The certification process included tests of the election management system and the conduct of mock primary and general elections on the entire voting system. ES&S Voting System, Release 4.5, Version 2, Revision 2, was certified by the Florida Division of Elections on July 17, 2006. According to Florida Division of Elections officials, testing of each version focuses on the new components, and components that were included in prior versions are not as vigorously tested. The 8.0.1.2 version of the iVotronic firmware was first tested as a part of ES&S Release 4.5, Version 1, which was certified in 2005. Version 2 introduced version 2.4.4.2 of the Unity Election Management System, which was certified in August 2005. Certification testing was conducted on software that was received from an independent test authority, who witnessed the building of the firmware from the source code. An independent test authority also conducted environmental testing of the iVotronic DRE in 2001 that was relied upon by the Florida Division of Elections for certification. A logic and accuracy test was conducted by Sarasota County on October 20, 2006, on 32 iVotronic DREs, and it successfully verified that all ballot positions on all nine ballot styles could be properly recorded. In addition, the use of a provisional ballot and audio ballot were tested, as well as machines configured for early voting with all nine ballot styles. After the 2006 general election, the Florida Division of Elections conducted an audit of Sarasota County's 2006 general election that included two parallel tests, an examination of the certified voting system and conduct of election by Sarasota County's elections office, and an independent review of the iVotronic DRE firmware's source code. After the conduct of this audit, the audit team concluded that there was no evidence that suggested the official election results were in error or that the voting systems contributed to the undervote in Sarasota County.[Footnote 6] The parallel tests were performed using 10 iVotronic DREs--5 used in the 2006 general election and 5 that were not used. Four of the machines in each test replicated the votes cast on four election day iVotronic DREs. The fifth machine in each test used an ad hoc test script that involved picking a random vote pattern along with a specific vote selection pattern picked from 10 predetermined vote patterns for the 13th Congressional District for each ballot cast. The audit report asserts that testing a total of 10 machines is more than adequate to identify any machine problems or irregularities that could have contributed to undervotes in the Florida-13 race. However, we concluded that the results from the testing of 10 machines cannot be applied to all 1,499 iVotronic DREs used during the 2006 general election because the sample was not random and the sample size was too small. In examining whether voting systems that were used in Sarasota County matched the systems that were certified by the Florida Division of Elections, the Florida audit team examined the Unity election management system and the firmware installed on six iVotronic DREs. The audit team confirmed that the software running on the Unity election management system and the firmware in the six iVotronic DREs matched the certified versions held in escrow by the Florida Division of Elections. On the basis of its review, the audit team concluded that there is no evidence to indicate that the iVotronic DREs had been compromised or changed. We agree that the test verifies that those six machines were not changed, but any extrapolation beyond this cannot be statistically justified because the size of the sample is too small. Therefore, these tests cannot be used to obtain reasonable assurance that the 1,499 machines used in the general election used the certified firmware. A software review and security analysis of the iVotronic firmware version 8.0.1.2 was conducted by a team led by Florida State University's SAIT Laboratory. The eight experts in the software review team attempted to confirm or refute many different hypotheses that, if true, might explain the undervote in the race for the 13th Congressional District. In doing so, they made several observations about the code, which we were able to independently verify. The software review and our verification of the observations were helpful, but a key shortcoming was the lack of assurance whether the source code reviewed by the SAIT team or by us, if compiled, would correspond to the iVotronic firmware that was used in Sarasota County for the 2006 election. According to ES&S and Florida Division of Elections officials, in May 2005 an independent testing authority witnessed the process of compiling the source code and building the version of firmware that was eventually certified by the Florida Division of Elections. According to ES&S officials, if necessary, ES&S can recreate the firmware from the source code, but the firmware would not be exactly identical to the firmware certified by the Florida Division of Elections because the embedded date and time stamp in the firmware would be different. The software review team also looked for security vulnerabilities in software that could have been exploited to cause the undervote. Although the team found several software vulnerabilities, the team concluded that none of them were exploited in Sarasota in a way that would have contributed to the undervote. We did not independently verify the team's conclusion. Reasonable Assurance of Some Voting System Objectives Has Been Achieved: The Unity election management system and the iVotronic DREs are the major voting system components that may require testing to determine whether they contributed to the large undervote in Sarasota County. Our review of tests already conducted and documentation from the election provide us reasonable assurance that the key functions of the Unity election management system--election definition and vote tabulation-- did not contribute to the undervote. The election definitions created using the Unity election management system are tested during logic and accuracy testing to demonstrate that they include all races, candidates, and issues and that each of the items can be selected by a voter. The votes tabulated on the iVotronic DRE at each precinct matched the data uploaded to the Unity election management system, and the totals from the precinct results tapes agree with that obtained by Unity. Further, the state audit confirmed that the Unity election management system software running in Sarasota County matched the escrowed version certified by the Florida Division of Elections. We have reasonable assurance that the number of ballots recorded by the iVotronic DREs is correct because this number is very close to the number of people recorded on the precinct registers as showing up at the polling places to vote either during early voting or on election day. This assurance also allows us to conclude that issues, such as votes cast by "fleeing voters"--votes that are cast by poll workers for voters who leave the polling place before pressing the button to cast the vote--and the potential loss of votes during a system shutdown, did not affect the undervote in this election. If these issues had occurred, they would have caused a discrepancy between the number of voters who sign in at the polling place to vote and the public counts recorded on the iVotronic DREs. We have reasonable assurance that provisional ballots were appropriately handled by the iVotronic DREs and the Unity election management system. We also verified that during the Florida certification test process, the Division of Elections relied on successful environmental and shock testing conducted by an independent test authority. Reasonable Assurance That All iVotronic DREs Used in the 2006 General Election Used Software Certified by the Florida Division of Elections Is Lacking: We found that prior testing and activities do not provide reasonable assurance that all iVotronic DREs used in Sarasota County on election day were using the hardware and firmware certified for use by the Florida Division of Elections. Sarasota County has records indicating that only certified versions were procured from ES&S, and the firmware version is checked in an election on the zero and results tapes. However, because there was no independent validation of the system versions, we cannot conclude that no modifications were made to the systems that would have likely made them inconsistent with the certified version. As we previously mentioned, the firmware comparison of only 6 iVotronic DREs in the state audit is insufficient to support generalization to all 1,499 iVotronic DREs that recorded votes during the election. Without reasonable assurance that all iVotronic DREs are running the same certified firmware, it is difficult for us to rely on the results of other testing that has been conducted, such as the parallel tests or the logic and accuracy tests. The Ability of Voters to Make Selections in Different Ways and Have Their Votes Properly Recorded Has Not Been Fully Tested: Prior testing of the iVotronic DREs only verified 13 of the 112 ways that we identified that a voter may use to select a candidate in Florida's 13th Congressional District race. Specifically, on an iVotronic DRE, a voter could (1) initially select either candidate or neither candidate (i.e. undervote), (2) change the vote on the initial screen, and (3) use a combination of page back and review screen options to change or verify his or her selection before casting the ballot. By taking into account these variations, our analysis has found at least 112 different ways a voter could make his or her selection in Florida's 13th Congressional District race, assuming that it was the only race on the ballot. Out of 112 different ways to select a candidate in the congressional race, Florida certification tests and the Sarasota County logic and accuracy tests verified 3 ways to select a candidate; and the Florida parallel tests verified 10 ways to select a candidate--meaning that of the 112 ways, 13 have been tested. By not verifying these different ways to select a candidate, we do not have reasonable assurance that the system will properly handle expected forms of voter behavior. The Effect of Miscalibrated iVotronic DREs Is Unclear: During the setup of the iVotronic DRE, sometimes referred to as the clear and test process, the touch screens are calibrated by using a stylus to touch the screen at 20 different locations. The calibration process is designed to align the display screen with the touch screen input. It has been reported that a miscalibrated machine could affect the selection process by highlighting a candidate that is not aligned with what the voter selected. We identified two reported cases on election day where the miscalibration of the iVotronic DRE led to its closure and discontinued use for the rest of the day. While a miscalibrated machine could certainly make an iVotronic DRE harder to use, it is not clear it would have helped to contribute to the undervote. We did not identify any prior testing or activities that would help us understand the effect of a miscalibrated iVotronic DRE on the undervote. Further Tests Could Provide Increased but Not Absolute Assurance That the iVotronic DREs Used in the Election Did Not Cause the Undervote: On the basis of our analysis of all prior test and audit activities, we propose that a firmware verification test, a ballot test, and a calibration test be conducted to try to obtain increased assurance that the iVotronic DREs used in Sarasota County during the 2006 general election did not cause the undervote. We propose that the firmware verification testing be started first, once the necessary arrangements have been made, such as access to the needed machines and the development of test protocols and detailed test procedures. Once we have reasonable assurance that the iVotronic DREs are running the same certified firmware, we could conduct the ballot test and calibration test on a small number of machines to determine whether it is likely the machines accurately recorded and counted the ballots. If the firmware verification tests are successfully conducted, we would have much more confidence that the iVotronic DREs will behave similarly when tested. If there are differences in the firmware running on the iVotronic DREs, we would need to reassess the number of machines that need to be tested for ballot testing and calibration testing in order for us to have confidence that the test results would be true for all 1,499 iVotronic DREs used during the election. In other words, if we are reasonably confident that the same software is used in all 1,499 machines, then we are more confident that the results of the other tests on a small number of machines can be used to obtain increased assurance that the iVotronic DREs did not cause the undervote. Although the proposed tests would provide increased assurance, they would not conclusively eliminate the machines as a cause of the undervote. Conduct Firmware Testing to Verify That the Firmware in the iVotronic DREs Used in Sarasota County Matches the Certified Version: We propose to conduct a firmware verification test using a statistical sampling approach that can provide reasonable assurance that all 1,499 iVotronic DREs are running the certified version of firmware. The exact number of machines that would be tested depends on the confidence level desired and how much error can be tolerated. We propose drawing a representative sample from all the iVotronic DREs that recorded votes in the general election. With a sample size of 115 iVotronic DREs, which would be divided between sequestered and nonsequestered machines, and assuming that there are no test failures, we would be able to conclude with a 99 percent confidence level that no more than 4 percent of the 1,499 iVotronic DREs used in the election were using uncertified firmware. We suggest a test approach similar to what was used by the Florida Division of Elections when it verified the firmware for 6 iVotronic DREs. We estimate that the firmware testing for 115 machines could be conducted in about 5 to 7 days and would require about 5 or 6 people, once the necessary arrangements have been made. The machines would be transported to a test facility specified by Sarasota County election officials where we could perform the test. The activities involved in conducting a firmware validation test would include locating and retrieving the selected iVotronic DRE from the storage facility, transporting it to the test facility, opening the DRE, extracting the chip with the firmware, reading the contents of the chip using a specialized chip reader, and conducting a comparison between the contents and the certified firmware to determine if any differences exist. To conduct this test, we would need commercially available specialized hardware and software similar to that used by the Florida Division of Elections in its firmware comparison test. Conduct Ballot Testing of iVotronic DREs to Confirm Correct Operation: We propose conducting ballot testing on 10 iVotronic DREs, each configured with one of the nine different ballot styles, with the 10th machine configured as an early voting machine with all nine ballot styles. We would test 112 ways to select a candidate on the early voting machine. On the election day machines, we would test the 112 different ways distributed across the 9 machines in a random manner, meaning each machine would on average record 12-13 ballots. Assuming that (1) reasonable assurance is obtained that all iVotronic DREs used during the election were using the same certified firmware, and (2) we found no failures during the ballot testing, this testing would provide increased assurance that the iVotronic DREs used during the election, both in early voting and in election day voting, were able to accurately record and count ballots when using any of the 112 ways to select a candidate in the Florida-13 race. We would plan to code each ballot by including an identifier in the write-in candidate field for either the U.S. senator or governor's race. Using this write-in coding, we could examine the ballot image and confirm that each ballot was accurately recorded and counted by the iVotronic DRE. Any encountered failures would also be more rapidly attributed to a specific test case, and we would be able to more readily repeat the test case to determine if we have a repeatable condition. Testing 112 ways to select a candidate on a single machine would also provide us some additional assurance that the volume of ballots cast on election day did not cause a problem. We note that casting 112 ballots on a single machine is more than that cast on over 99 percent of the 1,415 machines used on election day. We estimate the ballot testing would take about 2 to 3 days and require the equivalent of 2 people, once the necessary arrangements have been made. Deliberately Miscalibrate an iVotronic DRE to Understand the Effect on the Undervote: Because little is known about the effect of a miscalibrated machine on the behavior of an iVotronic DRE, we propose to deliberately miscalibrate an iVotronic DREs and verify the functioning of the machine. We propose to identify different ways to miscalibrate a ballot and to test ballots on the miscalibrated iVotronic DRE to verify that it still properly records votes. With this test we would confirm whether (1) the review screen displays the same selection in the Florida-13 race as was highlighted in the selection screen, and (2) that the vote is recorded as it was displayed on the review screen. Again, we would plan to use the write-in candidate option to verify the proper recording of the ballot. This test would demonstrate whether the system correctly records a vote for the race and hence whether it contributed to the undervote. We estimate that the calibration test could be completed in about 1 day by 2 people, once the necessary arrangements have been made. Several Matters Remain to Be Addressed to Conduct Further Testing: Should the task force ask us to conduct the proposed testing, we want to make the task force aware of several other matters that would need to be addressed before we could begin testing. These activities would require some time and resources to complete before testing could commence. First, we would need to gain access to iVotronic DREs that have been subject to a sequestration order in the state court system of Florida. If we do not have access to the needed machines, we would be unable to obtain reasonable assurance that the machines used on election day were using certified software, and without this assurance, the results from prior tests and any results of our ballot and calibration tests would be less meaningful because we would be unable to apply the results to all 1,499 iVotronic DREs used during the election. Second, we would need to agree upon an appropriate facility for the tests. Sarasota County Supervisor of Elections has indicated that we can use its warehouse space, but because of upcoming elections in November and January, the only time the election officials would be able to provide us this space and the necessary support is between November 26 and December 7, 2007. If testing cannot be completed during this time period, Sarasota County officials stated that they would not be able to assist us until February 2008. Third, some tests may require commercially available specialized software, hardware, or other tools to conduct the tests. We would need to make arrangements to either borrow or to purchase such testing tools before commencing testing. Fourth, in order to conduct any tests, we would need to develop test protocols and detailed test procedures and steps. We also anticipate that we would need to conduct a dry run, or dress rehearsal, of our test procedures to ensure that our test tools function properly and that our time estimates are reasonable. Finally, we would need to make arrangements for video recording of our testing. It would be our preference to have a visual record of the tests to document the actual test conduct and to facilitate certain types of test analysis. Other Observations on Touch Screen Voting Systems: We recognize that human interaction with the ballot layout could be a potential cause of the undervote. Although we have not explored this issue in our review, we note that there is an ongoing academic study that is exploring this issue using voting machines obtained from ES&S. We believe that such experiments could be useful and could provide insight into the ballot layout issue. During our review, we noted that several suggestions have been offered as possible ways to establish that voters are intentionally undervoting and to provide some assurance that the voting systems did not cause the undervote. First, a voter-verified paper trail could provide an independent confirmation that the touch screen voting systems did not malfunction in recording and counting the votes from the election. The paper trail would reflect the voter's selections and, if necessary, could be used in the counting or recounting of votes. This issue is recognized in the Florida State University SAIT source code review as well as the 2005 and draft 2007 Voluntary Voting Systems Guidelines prepared for the Election Assistance Commission. We have previously reported on the need to implement such a function properly.[Footnote 7] Second, explicit feedback to voters that a race has been undervoted and a prompt for voters to affirm their intent to undervote might help prevent many voters from unintentionally undervoting a race. On the iVotronic DREs, such feedback and prompts are provided only when the voter attempts to cast a completely blank ballot, but not when a voter undervotes in individual races. Third, offering a "none of the above" option in a race would provide voters with the opportunity to indicate that they are intentionally undervoting. The State of Nevada provides this option in certain races in its elections. Decisions about these or other suggestions about ballot layout or voting system functions should be informed by human factors studies that assess their effectiveness in accurately recording voters' preferences, making voting systems easier to use, and preventing unintentional undervotes. Conclusions: The high undervote encountered in Sarasota County in the 2006 election for Florida's 13th Congressional District has raised questions about whether the voting systems accurately recorded and counted the votes cast by eligible voters. Other possible reasons for the undervote could be that voters intentionally undervoted or voters did not properly cast their ballots on the voting systems, potentially because of issues relating to the interaction between voters and the ballot. The focus of our review has been to determine whether the voting systems--the iVotronic DREs, in particular--contributed to the undervote. We found that the prior reviews of Sarasota County's 2006 general election have provided valuable information about the voting systems. Our review found that in some cases we were able to rely on this information to eliminate areas of concern. This allowed us to identify the areas where increased assurances were needed to answer the questions being raised. Accordingly, the primary focus of the tests we are proposing is to obtain increased assurance that the results of the prior reviews and our proposed testing can be applied to all the iVotronic DREs used in the election. Our proposed tests involving the firmware comparison, ballot testing, and calibration testing could help reduce the possibility that the undervote was caused by the iVotronic DREs. However, even after completing the tests, we would not have absolute assurance that the iVotronic DREs did not play any role in the large undervote. Absolute assurance is impossible to achieve because we are unable to recreate the conditions of the election in which the undervote occurred. By successfully conducting the proposed tests, we could reduce the possibility that the iVotronic DREs were the cause of the undervote and shift attention to the possibilities that the undervote was the result of intentional actions by the voter or voters that did not properly cast their votes on the voting system. Comments and Our Evaluation: We provided draft copies of this statement to the Secretary of State of Florida, the Supervisor of Elections of Sarasota County, and ES&S for review and comment. The Florida Department of State provided technical comments, which we incorporated. The Sarasota County Supervisor of Elections appreciated the opportunity to review the draft, but provided us no comments. In its comments, ES&S stated that it believes that the collective results of testing already conducted on the Sarasota County voting systems have demonstrated that they performed properly and as they were designed to function and that all votes were accurately captured and counted as cast in Florida's 13th Congressional District race. Further, ES&S asserts that tests and analyses should be conducted to examine the effect of the ballot display on the undervote, which it believes is the most probable cause of the undervote. We disagree that the collective results of testing already conducted on the Sarasota County voting systems adequately demonstrate that the voting systems could not have contributed to the undervote in the Florida-13 race. First, as we have cited, we do not have adequate assurance that all the iVotronic DREs used in Sarasota County used the firmware certified by the Florida Division of Elections. Without this assurance, it is difficult for us to apply the results from the other tests to all 1,499 machines that recorded votes during the election because we are uncertain that all machines would have behaved in a similar manner. Further, we believe that expected forms of voter behavior to select a candidate in the Florida-13 race were not thoroughly tested. While ES&S asserts that such processes would have no effect on the iVotronic DRE's ability to capture and record a voter's selection, we did not identify testing that verified this. Further, while ES&S states that the testing of a deliberately miscalibrated iVotronic DRE would result in a clearly visible indication of which candidate was selected, we could not identify any testing that demonstrated this. We acknowledge that the large undervote in Florida's 13th Congressional District race could have been caused by voters who intentionally undervoted or voters who did not properly cast their ballots, potentially because of issues related to the human interaction with the ballot. However, the focus of our review, as agreed with the task force, was to review whether the voting systems could have contributed to the large undervote. ES&S also provided technical comments, which we incorporated as appropriate. Mr. Chairman, this completes my prepared statement. I would be happy to respond to any questions you or other members of the task force may have at this time. Contacts and Acknowledgments: For further information about this statement, please contact Keith Rhodes, Chief Technologist, at (202) 512-6412 or rhodesk@gao.gov, or Naba Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other key contributors to this statement include James Ashley, James Fields, Jason Fong, Cynthia Grant, Geoffrey Hamilton, Richard Hung, John C. Martin, Jan Montgomery, Jennifer Popovic, Sidney Schwartz, and Daniel Wexler. [End of section] Footnotes: [1] Undervotes occur when the number of choices selected by the voter is fewer than the maximum allowed for that contest. In this case, it means ballots that did not record a selection for either candidate in the congressional contest. [2] Florida Department of State, Florida Voting System Standards, Form DS-DE 101 (Jan. 12, 2005). [3] The certified version of ES&S Voting System Release 4.5, Version 2, Revision 2, specifies the use of iVotronic hardware version 1.0. According to Florida Division of Election officials, hardware version 1.1 of the iVotronic DRE has been available since at least 2004 and should have been included as a part of the certification for ES&S Voting System Release 4.5, Version 2, Revision 2. According to ES&S officials, iVotronic firmware version 8.0.1.2 runs in exactly the same manner on hardware versions 1.0 and 1.1. [4] Election day voting is the casting of ballots on election day at polling places. Absentee and early voting are programs that permit eligible persons to vote prior to election day. Absentee voting is conducted by mail in advance of election day and early voting is generally in-person voting in advance of election day at specific polling locations. [5] Early and election day ballots include provisional ballots cast during those respective stages of voting and included in the vote totals. 160 provisional ballots were included in the vote totals. 37 provisional ballots were excluded. Because the absentee ballots were not cast using iVotronic voting systems, we did not verify the absentee ballot counts. When absentee ballots are included, a total of 142,532 ballots were cast and a total of 18,412 undervotes were recorded. [6] Florida Department of State, Audit Report of the Election Systems and Software, Inc.'s, iVotronic Voting System in the 2006 General Election for Sarasota County, Florida (Tallahassee, Florida: Feb. 2007), and Security and Assurance in Information Technology Laboratory, Florida State University, Software Review and Security Analysis of the ES&S iVotronic 8.0.1.2 Voting Machine Firmware (Tallahassee, Florida: Feb. 23, 2007). [7] GAO, Elections: Federal Efforts to Improve Security and Reliability of Electronic Voting Systems Are Under Way, but Key Activities Need to Be Completed, GAO-05-956 (Washington, D.C.: Sept. 21, 2005). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, DC 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Gloria Jarmon, Managing Director, JarmonG@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548: Public Affairs: Susan Becker, Acting Manager, BeckerS@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548: