Chapter 2: Conformity Assessment Process
2.1 Overview
Conformity assessment encompasses the examination and testing of software and firmware; tests of hardware under conditions simulating the intended storage, operation, transportation, and maintenance environments; inspection and evaluation of system documentation; and operational tests to validate system performance and functioning under normal and abnormal conditions. Conformity assessment also evaluates the completeness of the manufacturer's developmental test program, including the sufficiency of manufacturer tests conducted to demonstrate compliance with stated system design and performance specifications, and the manufacturer's documented quality assurance and configuration management practices. The assessment addresses individual system components or elements as well as the integrated system as a whole.
Beginning in 1994, the National Association of State Election Directors (NASED) began accrediting Independent Test Authorities for the purpose of conducting qualification testing of voting systems. The qualification testing process was originally based on the 1990 voting system standards and evolved to encompass the new requirements contained in the 2002 version of the standards.
The Help America Vote Act (HAVA) directs the U.S. Election Assistance Commission (EAC) to provide for the testing, certification, decertification, and recertification of voting system hardware and software by accredited laboratories. HAVA also introduces different terminology for these functions. Under the EAC process, test labs are "accredited" and voting systems are "certified." The term "standards" has been replaced with the term "VVSG."
Conformity assessment may be performed by one or more accredited test labs that together perform the full scope of tests required. Assessment may be coordinated across accredited test labs so that equipment and materials tested by one accredited test lab can be used in the tests performed by another accredited test lab.
When multiple accredited test labs are being used, the development of the test plan (see Part 2: Chapter 5: “Test Plan (test lab)”) and the test report (see Part 2: Chapter 6: ”Test Report (test lab)”) must be coordinated by a lead accredited test lab. The lead test lab is responsible for ensuring that all testing has been performed and documented in accordance with the VVSG and is ultimately responsible for the summary finding of conformance (see Requirement Part 2: 6.1-F).
Whether one or more accredited test labs are used, the testing generally consists of three phases:
- Pre-test activities;
- Chapter 3: overview of general testing approaches;
- Testing; and
- Post-test activities.
2.2 Scope of Assessment
The conformity assessment process is intended to discover vulnerabilities that, should they appear in actual election use, could result in failure to complete election operations in a satisfactory manner. This involves
- Operational accuracy in the recording and processing of voting data, as measured by report total error rate;
- Operational failures or the number of unrecoverable failures under conditions simulating the intended storage, operation, transportation, and maintenance environments for voting systems;
- System performance and function under normal and abnormal conditions; and
- Completeness and accuracy of the system documentation and configuration management records to enable purchasing jurisdictions to effectively install, test, and operate the system.
Conformity assessment involves several different kinds of testing, including
- Inspections, where the conformity of the voting system and manufacturer practices for configuration management and quality assurance are evaluated via expert review;
- Hardware testing, where the ability of the system to tolerate the physical conditions of its operation, transportation and storage is evaluated;
- Functional testing, where the conformity of the voting system's observable behaviors is evaluated;
- Performance testing, where the satisfaction of specified benchmarks is either evaluated in specific tests or monitored concurrent with other testing;
- Usability testing, where the performance is evaluated with human test subjects; and
- Vulnerability testing, where the system's resistance to attack is evaluated.
Voting system hardware, software, communications and documentation are examined and tested to determine suitability for elections use. Examination and testing address the broad range of system functionality and components, including system functionality for pre-voting, voting, and post-voting functions. All products for election use are tested in accordance with the applicable procedures.
Tests are conducted for new systems seeking initial testing as well as for modified versions of systems that have been previously tested.
Not all systems are required to complete every category of testing. Consistent with Requirement Part 2: 5.1-D, the test lab may find that proven performance of COTS hardware, software and communications components in commercial applications other than elections obviates the need for certain specific evaluations. However, as most functional testing exercises the complete system, COTS components are always tested together with other components of the voting system. Similarly, if a previous version of the same system has been tested, the test lab may find that complete retesting would be redundant, but some tests that exercise the entire system are always conducted. The background and rationale for these decisions regarding the scope of testing must be documented in the test plan.
The accredited test lab determines which tests are necessary to reassess a modified system based on a review of the nature and scope of changes and other submitted information including the system documentation, manufacturer test documentation, configuration management records, and quality assurance information. The accredited test lab may determine that a modified system is subject only to limited retesting if the manufacturer demonstrates that the change does not affect demonstrated compliance with these VVSG for:
- Performance of voting system functions;
- Voting system security and privacy;
- Overall flow of system control; and
- The manner in which ballots are defined and interpreted, or voting data are processed.
Limited testing is intended to facilitate the correction of defects, the incorporation of improvements, the enhancement of portability and flexibility, and the integration of vote counting software with other systems and election software.
In all cases, the system documentation and configuration management records are examined to confirm that they completely and accurately reflect the components and component versions that comprise the voting system.
2.3 Testing Sequence
Tests and inspections required by these VVSG need not be conducted in any particular order. Test labs should organize the test campaign to maximize overall testing effectiveness, to test in as efficient a manner as possible, and to minimize the amount of regression testing that is incurred when nonconformities are found and corrected. Test anomalies and errors are communicated to the system manufacturer throughout the process.
2.4 Pre-Test Activities
Pre-test activities include the request for initiation of testing and the pre-test preparation.
2.4.1 Initiation of testing
Conformity assessment is conducted at the request of the manufacturer. The manufacturer must:
- Request the performance of conformity assessment from among the accredited testing laboratories;
- Enter into formal agreement with the accredited test lab for the performance of testing; and
- Prepare and submit materials required for testing consistent with the requirements of the VVSG.
Conformity assessment is conducted for the initial version of a voting system as well as for all subsequent revisions to the system that are to be used in elections. As described in Part 3: 2.2 “Scope of Assessment”, the nature and scope of testing for system changes or new versions is determined by the accredited test lab based on the nature and scope of the modifications to the system and on the quality of system documentation and configuration management records submitted by the manufacturer.
2.4.2 Pre-test preparation
Pre-test preparation encompasses the following activities:
- The manufacturer and accredited test lab enter into an agreement for the testing to be performed by the accredited test lab;
- The manufacturer prepares and submits a TDP to the accredited test lab. The TDP consists of the materials described in Part 2: Chapter 3: ”Technical Data Package (manufacturer)”;
- The accredited test lab performs an initial review of the TDP for completeness and clarity and requests additional information as required;
- The manufacturer provides additional information if requested by the accredited test lab;
- The test lab witnesses the production of the implementation for testing;
- The manufacturer delivers to the accredited test lab all hardware and software needed to perform testing.
2.4.2.1 Documentation submitted by manufacturer
The manufacturerSHALL submit to the test lab a Technical Data Package conforming to the requirements of Part 2: Chapter 3: ”Technical Data Package (manufacturer)”.
Applies to: Voting system
DISCUSSION
The manufacturer must submit all the documentation necessary for the identification of the full system configuration submitted for evaluation and for the development of an appropriate test plan by the accredited test lab for conducting conformity assessment. This documentation collectively is referred to as the Technical Data Package (TDP). The TDP provides information that defines the voting system's design, method of operation, and related resources. It provides a system overview and documents the system's functionality, hardware, software, security, test specifications, operations procedures, maintenance procedures, and personnel deployment and training requirements. It also documents the manufacturer's configuration management plan and quality assurance program. If another version of the system was previously tested, the TDP would also include appropriate system change notes.
Source: [VVSG2005] II.1.5
2.4.2.2 Voting equipment submitted by manufacturer
Manufacturers may seek to market a complete voting system or an interoperable component of a voting system. In all instances, manufacturers must submit for testing the specific system configuration that will be offered to jurisdictions or that comprises the component to be marketed plus the other components with which the component is to be used. Under no circumstances will a component be assessed except as part of a complete voting system, and that assessment is valid only when that component is used with that same system (see Part 1: 2.3 “Conformance Designations”).
If needed for compliance with Part 3: 2.4.3.4 “Unmodified COTS verification”, the manufacturer SHALL supply the system with the COTS components omitted, for subsequent integration performed by or witnessed by the test lab.
Applies to: Voting system
DISCUSSION
See Part 3: 2.4.3.4 “Unmodified COTS verification”.
Source: New requirement.
The hardware submitted for conformity assessment SHALL be equivalent, in form and function, to the actual production version of the hardware units specified for use in the TDP.
Applies to: Voting system
Source: [VVSG2005] II.1.6.a
The firmware and software submitted for conformity assessment SHALL be the exact firmware and software that will be used in production units.
Applies to: Voting system
Source: [VVSG2005] II.1.6.b
Developmental prototypes SHALL NOT be submitted unless the manufacturer can show that the equipment to be tested is equivalent to standard production units both in performance and construction.
Applies to: Voting system
Source: [VVSG2005] II.1.6.c
Benchmark directory listings SHALL be submitted for all software/firmware elements (and associated documentation) included in the manufacturer's release as they would normally be installed upon setup and installation.
Applies to: Voting system
Source: [VVSG2005] II.1.6.d
2.4.3 Initial system build by test lab
The following requirements describe how test labs are to perform build of voting system software by the test lab.
Previously built voting system software being updated may be able to use the requirements found Part 3: 2.4.3.3 “Updating previously built voting system software executable code” to create the updated executable code including application logic, border logic, and third party logic.
2.4.3.1 Build environment establishment
The test lab SHALL assemble the build environment(s) used to create executable code including application logic, border logic, and third party logic.
Applies to: Voting system
Source: [EAC06] Section 5.6.1.2 and [VVSG2005] II.1.8.2.4
At least one representative from the manufacturer SHALL witness the assembly of the build environment.
Applies to: Voting system
Source: [EAC06] Section 5.6 and [VVSG2005] II.1.8.2.4
A representative from the test lab SHALL create a build environment establishment record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location the build environment was established; names, affiliations, and signatures of all people present; copies of the procedures used to assemble the build environment; list of software and hardware used to establish the build environment; and the voting system associated with the build environment.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL obtain the software and hardware required to establish the build environment.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-C documents the software and hardware required to assemble the build environment.
The test lab SHALL obtain COTS software and hardware required to assemble the build environment from the open market.
Applies to: Voting system
DISCUSSION
Note: manufacturers are required to supply non-COTS hardware and software as part of Requirement Part 3: 2.4.2.2-A.
The test lab SHALL remove any previously stored information on erasable storage media in preparation for using the media to assemble the build environment.
Applies to: Voting system
DISCUSSION
The purpose of this requirement is to prepare erasable storage media for use by the build environment. The requirement does not require the prevention of previously stored information leakage or recovery. Simply deleting files from file systems, flashing memory cards, and removing electrical power from volatile memory satisfies this requirement.
Source: [EAC06] Section 5.6.1.1
The test lab SHALL use the procedures found in the TDP to assemble the build environment.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-D documents the procedures to assemble the build environment. Test lab personnel can have manufacturers provide guidance during the assembly of the build environment, but test lab personnel must perform the actual assembly.
Source: [EAC06] Section 5.6.1.2
The test lab SHALL document as part of the build environment establishment record the reason for any deviation from assembly procedures found in the TDP.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-D documents the procedures used to assemble the build environment.
Source: [EAC06] Section 5.9
When digital signatures are associated with software, the test lab SHALL verify digital signatures before using the software for the build environment.
Applies to: Voting system
DISCUSSION
The digital signatures associated with the build environment may be from the manufacturer of the software, National Software Reference Library (NSRL), or other authoritative sources.
Source: [EAC06] Section 5.6.2.1
The test lab SHALL record as part of the build environment establishment record the results of digital signature verification including who generated the signature.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL copy the binary image of the assembled build environment to unalterable storage media.
Applies to: Voting system
DISCUSSION
This requirement creates a snapshot of the build environment before it is used to build the voting system software executable code. Unalterable storage media includes technology such as a CD-R, but not CD-RW.
The test lab SHALL create a digital signature for the binary image of the build environment, and include the digital signature on the unalterable storage media with the binary image.
Applies to: Voting system
Source: [EAC06] Section 5.6.1.3
2.4.3.2 Build of voting system software executable code
Previously built voting system software being updated may be able to use Requirement Part 3: 2.4.3.3 to create the updated executable code including application logic, border logic, and third party logic.
The test lab SHALL build the executable code including application logic, border logic, and third party logic of the voting system using the established build environment.
Applies to: Voting system
DISCUSSION
The build environment is established using the requirements in Part 3: 2.4.3.1 “Build environment establishment”.Source: [EAC06] and [VVSG2005] II.1.8.2.4
At least one representative from the manufacturer SHALL witness the build of executable code including application logic, border logic, and third party logic of the voting system.
Applies to: Voting system
Source: [EAC06] Section 5.6
A representative from the test lab SHALL create an executable code build record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location of the build; names, affiliations, and signatures of all people present; filenames of the source code and resulting executable code; voting system software version; name and version of the voting system (including certification number, if possible); and copies of the procedures used to build the voting system software executable code.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL validate manufacturer digital signatures on voting system software source code before placing source code on the build environment.
Applies to: Voting system
DISCUSSION
Requirement Part 3: 2.6.2.4-D requires manufacturers to provide voting system software source code with digital signatures as part of the TDP.
Source: [EAC06] Section 5.6.2.1
The results of digital signature validation including who generated the signature SHALL be part of the executable code build record for voting system software.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL use the procedures found in the TDP to build the voting system software executable code including application logic, border logic, and third party logic.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-E documents the procedures to build voting system software executable code. Test lab personnel can have manufacturers provide guidance during the build of the voting system executable code, but test lab personnel must perform the actual build.
Source: [EAC06] Section 5.6.3
The test lab SHALL document as part of the executable code build record the reason for any deviation from build procedures found in the TDP.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-E documents the procedures to build voting system software executable code.
Source: [EAC06] Section 5.9
After voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL copy the binary image of the build environment (including source and executable code) to unalterable storage media.
Applies to: Voting system
DISCUSSION
This requirement creates a snapshot of the build environment after it has been used to build voting system software executable code. Unalterable storage media includes technology such as a CD-R, but not CD-RW.
Source: [EAC06] Section 5.6.2.3
After voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL create a digital signature for the binary image of the build environment (including source and executable code), and include the digital signature on the unalterable storage media with the binary image.
Applies to: Voting system
Source: [EAC06] Section 5.6.2.2
2.4.3.3 Updating previously built voting system software executable code
The following voting system software build requirements apply when updates to previously built voting system software has occurred. These requirements assume the original build environment can be used to create the updated software and a significant portion of original software is not being updated. If the original build environment cannot be used or a significant portion of the original software is being updated, then the requirements of Part 3: 2.4.3.1 “Build environment establishment” and Part 3: 2.4.3.2 ”Build of voting system software executable code”.
At least one representative from the manufacturer SHALL witness the establishment of the post build environment associated with the previously built voting system software, and the build of the updated voting system software executable code including application logic, border logic, and third party logic.
Applies to: Voting system
DISCUSSION
This requirement does not modify the requirement found in Section 5.6 of the EAC Testing and Certification Program Manual [EAC06] requiring a representative from both the manufacturer and test lab to be present during the build.
Source: [EAC06] Section 5.6
The test lab SHALL establish the build environment using the original post build environment binary image associated with the previously built voting system software.
Applies to: Voting system
DISCUSSION
Requirements Part 3: 2.4.3.2-A.7 and Part 3: 2.4.3.2-A.8 create the post build binary image of the original built voting system software developed by the manufacturer. If the test lab does not posses the required hardware and software to create the build environment then Requirements Part 3: 2.4.3.2-A.7 and Part 3: 2.4.3.2-A.8 apply. This requirement extends the requirement found in [EAC06] Section 5.6.4.1 and 5.6.4.2 by explicitly stating the original build environment needs to be established.
Source: [EAC06] Section 5.6.4.1 and 5.6.4.2
The test lab SHALL remove previously stored information on erasable storage media in preparation for using the media to establish the build environment.
Applies to: Voting system
DISCUSSION
The purpose of this requirement is to prepare the erasable storage media for use by the original post build environment. The requirement does not require the prevention of previously stored information leakage or recovery. Simply deleting files from the file system, flash memory cards, and removing electrical power from volatile memory satisfy this requirement.
Source: [EAC06] Section 5.6.1.1
The test lab SHALL verify the digital signature of the original post build binary image associated with the previously built voting system software before using the binary image to establish the build environment.
Applies to: Voting system
DISCUSSION
This requirement does not modify the requirement found in Section 5.6.4.1 of the EAC Testing and Certification Program Manual [EAC06] that states the file signature of the build environment needs to be verified before use.
Source: [EAC06] Section 5.6.4.1
The result of digital signature verification including who generated the signature SHALL be part of the original post build environment establishment record.
Applies to: Voting system
Source: [EAC06] Section 5.9
A representative from the test lab SHALL create an original post build environment establishment record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location the original post build environment was established; names, affiliations, and signatures of all people present; copies of the procedures used to assemble the original post build environment; list of software and hardware used to establish the original post build environment; and the voting system associated with the original post build environment.
Applies to: Voting system
DISCUSSION
This requirement updates the requirement found in Section 5.9 of the EAC Testing and Certification Program Manual [EAC06] by specifying the information needed to be documented when establishing the build environment.
Source: [EAC06] Section 5.9
The test lab SHALL build the executable code including application logic, border logic, and third party logic of the updated voting system software.
Applies to: Voting system
DISCUSSION
This requirement does not modify the requirement found in Section 5.6.4.2 of the EAC Testing and Certification Program Manual [EAC06] that states the executable files are created; and extends the requirement found at Section 1.8.2.4 of [VVSG2005] Volume II in [VVSG2005] by requiring the use of the build environment established in Part 3: 2.4.3.1 “Build environment establishment”.
Source: [EAC06] Section 5.6.4.2 and [VVSG2005] II.1.8.2.4
The test lab SHALL validate manufacturer digital signatures on updated voting system software source code before placing the updated source code on the build environment.
Applies to: Voting system
DISCUSSION
This requirement modifies the requirement found in Section 5.6.4.2 of the EAC Testing and Certification Program Manual [EAC06] by constraining the verification to digital signature from a “file signature” (which could be a hash value or digital signature); extends 5.6.2.1 by specifying the verification to happen before software is installed on the build environment; and does not call for the digital signature of the build environment to be verified before installing the source code.
Source: [EAC06] Section 5.6.4.2
The result of digital signature verification including who generated the signature SHALL be part of the updated voting system software build record.
Applies to: Voting system
DISCUSSION
Requirement Part 3: 2.6.2.4-D requires manufacturers to provide voting system software source code with digital signatures as part of the TDP. This requirement updates the requirement found in Section 5.9 of the EAC Testing and Certification Program Manual [EAC06] by specifying the results of digital signature verification needs to be documented as part of the record when building the executable code.
Source: [EAC06] Section 5.9
The test lab SHALL use the procedures found in the TDP to build the updated voting system software executable code including application logic, border logic, and third party logic.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-G documents the procedures to build the updated voting system software executable code. Test labs can have manufacturers assist in building of the updated voting system software executable code. This requirement extends the requirement found in Section 5.6.4.2 of the [EAC06] by specifying the use of the manufacturer supplied procedures to build the updated voting system software.
Source: [EAC06] Section 5.6.4.2
A representative from the test lab SHALL create an executable code build record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location of the build; names, affiliations, and signatures of all people present; filenames of the source code and resulting executable code; voting system software version; name and version of the voting system (including certification number, if possible); and copies of the procedures used to build the updated voting system software executable code.
Applies to: Voting system
DISCUSSION
This requirement updates the requirement found in Section 5.9 of the [EAC06] by specifying the information needed to be documented when creating updated executable code.
Source: [EAC06] Section 5.9
After updated voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL copy the binary image of the updated build environment (including source and executable code) to unalterable storage media.
Applies to: Voting system
DISCUSSION
This requirement creates a snapshot of the updated build environment after it has been used to build the updated voting system software executable code. Unalterable storage media includes technology such as a CD-R, but not CD-RW. This requirement differs from the requirement found in Section 5.6.2.3 of the [EAC06] by creating the binary image after, instead of before, the updated software executable code has been built.
Source: [EAC06] Section 5.6.2.3
After updated voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL create a digital signature for the binary image of the updated build environment (including source and executable code), and include the digital signature on the unalterable storage media with the binary image.
Applies to: Voting system
DISCUSSION
This requirement differs from the requirement found in Section 5.6.2.2 of the [EAC06] by creating a digital signature on the binary image after the software executable code has been built as opposed to a “file signature” which could be a hash value or digital signature before the software executable code is built; although requirement 5.6.3.1 of the EAC Testing and Certification Program Manual requires “file signatures” for updated executable code.
Source: [EAC06] Section 5.6.2.2
2.4.3.4 Unmodified COTS verification
The following requirements describe how test labs are to verify that products identified as COTS are unmodified when used by the voting system.
The manufacturer SHALL document the procedures used to assemble and configure unmodified COTS components into the system supplied in Requirement Part 3: 2.4.2.2-A.
Applies to: Voting system
DISCUSSION
Test labs will assemble and configure unmodified COTS components into the voting system using the documentation provided by this requirement. Requirement Part 2: 4.4.1-A subitem e identifies all COTS components in the voting system, and Requirement Part 2: 3.8-D requires configuration data for unmodified COTS to be documented.
Source: COTS verification process per STS and CRT consensus, June 2006
Test labs SHALL obtain COTS components identified in Requirement Part 2: 4.4.1-A subitem 5 from open market suppliers of COTS components.
Applies to: Voting system
DISCUSSION
Test labs will procure the COTS components “off-the-shelf” from suppliers of the COTS components.
At least one representative from the test lab and manufacturer SHALL witness the assembly and configuration of the COTS components into the voting system.
Applies to: Voting system
The test lab SHALL assemble and configure the COTS components into the voting system.
Applies to: Voting system
The test lab SHALL document and maintain a record of the COTS assembly and configuration that includes, at a minimum: a unique identifier for each record; the time and date and location of the voting system build; names, affiliations, and signatures of all people present; copies of the procedures used to assemble and configure the COTS components; and identification of the voting system.
Applies to: Voting system
The test lab SHALL document deviations from the manufacturer documentation submitted for assembly and configuration of the COTS components.
Applies to: Voting system
2.5 Testing
Testing encompasses the preparation of a test plan, the establishment of the appropriate test conditions, the use of appropriate test fixtures, the witness of the system build and installation, the maintenance of test data, and the evaluation of the data resulting from tests and examinations.
2.5.1 Test plan
The accredited test lab SHALL prepare a test plan to define all tests and procedures required to assess conformity to the VVSG, including:
- Verifying or checking equipment operational status by means of manufacturer operating procedures;
- Establishing the test environment or the special environment required to perform each test;
- Initiating and completing operating modes or conditions necessary to evaluate the specific performance characteristics under test;
- Measuring and recording the value or range of values for the characteristics to be tested, demonstrating expected performance levels;
- Verifying, as above, that the equipment is still in normal condition and status after all required measurements have been obtained;
- Confirming that documentation submitted by the manufacturer corresponds to the actual configuration and operation of the system; and
- Confirming that documented manufacturer practices for quality assurance and configuration management comply with the VVSG.
Applies to: Voting system
DISCUSSION
Requirements on the content of the test plan are contained in Part 2: Chapter 5: ”Test Plan (test lab)”.Source: [VVSG2005] II.1.8.2.1
2.5.2 Test conditions
The accredited test lab may perform the tests in any facility capable of supporting the test environment.
Preparations for testing, arrangement of equipment, verification of equipment status, and the execution of procedures SHALL be witnessed by at least one independent, qualified observer, who SHALL attest that all test and data acquisition requirements have been satisfied.
Applies to: Voting system
Source: [VSS2002] II.9.6.2.2.a
When a test is to be performed at "standard" or "ambient" conditions, this SHALL refer to a nominal laboratory or office environment with a temperature in the range of 20.0 °C to 23.9 °C (68 °F to 75 °F) and prevailing atmospheric pressure and relative humidity.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.2.b
When a test is to be performed at conditions other than "standard" or "ambient," the test SHALL be performed at the required temperature and electrical supply voltage, regulated within the following tolerances:
- Temperature ± 2.2 °C (± 4 °F)
- AC electrical supply voltage ± 2 V
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.2.c
2.5.3 Test fixtures
Except as provided in Requirement Part 3: 2.5.3-B, the test lab SHALL NOT use simulation devices or software that bypass portions of the voting system that would be exercised in an actual election.
Applies to: Voting system
DISCUSSION
Devices or software that closely and validly simulate actual election use of the system are permissible. If a tabulator is specified to count paper ballots that are manually-marked with a specific writing utensil, it is not valid to substitute ballots that were mechanically marked by a printer. However, ballots that were marked according to manufacturer instructions can sometimes be recycled through a tabulator without invalidating the test. Limitations on this practice are provided in Requirement Part 3: 5.2.3-D.
The test lab may bypass the user interface of an interactive device in the case of environmental tests that:
- Would require subjecting test "voters" to unsafe or unhealthy conditions; or
- Would be invalidated by the presence of a test "voter."
The test lab may bypass the user interface of an interactive device in the case of environmental tests that:
Applies to: Voting system
2.5.4 Test data requirements
A test log of the procedure SHALL be maintained. This log SHALL identify the system and equipment by model and serial number.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.5.a
Test environment conditions SHALL be recorded.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.5.b
All operating steps, the identity and quantity of simulated ballots, annotations of output reports, the elapsed time for each procedure step, observations of equipment performance, and, in the case of non-operating hardware tests, the condition of the equipment SHALL be recorded.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.5.c
2.5.5 Test practices
The accredited test lab SHALL conduct the examinations and tests defined in the test plan to determine compliance with the voting system requirements described in Part 1 and Part 2.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.6
If any failure, malfunction or data error is detected, its occurrence and the duration of operating time preceding it SHALL be recorded for inclusion in the analysis of data obtained from the test.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.6.a
If a logic defect is responsible for the incorrect recording, tabulation, or reporting of a vote, the test campaign SHALL be terminated and the system SHALL be rejected.
Applies to: Voting system
DISCUSSION
Conformity assessment is not quality assurance. If a critical software defect is found, the system cannot be considered trustworthy even after the known fault is corrected, because the cases that the test lab does not have the opportunity to test can be expected to conceal similar faults. Any subsequent testing of a system based on or derived from the rejected system requires a new application and starting over.
Source: [GPO90] 7.1.1, [VSS2002] Overview, [VVSG2005] II.1.8.2.6.b
If a logic defect is found that is not responsible for the incorrect recording, tabulation, or reporting of a vote, the test campaign SHALL be suspended and the system returned to the manufacturer for correction and quality assurance.
Applies to: Voting system
DISCUSSION
Rejection may be a foregone conclusion if sufficient evidence has been collected to show that the reliability benchmark is not satisfied (see Part 3: 5.3.3 “Reliability”). Notwithstanding that, the manufacturer will be given the opportunity to correct noncritical software defects. Revisions to the software must be performed within the manufacturer's quality assurance and configuration management processes and must undergo manufacturer regression testing before the conformity assessment process is resumed. When it is resumed, the test plan should be revised to include regression testing for the change that was made.
Source: [VVSG2005] II.1.8.2.6.b, clarified and strengthened
If the anomaly is other than a logic defect, and if corrective action is taken to restore the equipment to a fully operational condition within eight hours, then the test campaign may be resumed at the point of suspension.
Applies to: Voting system
DISCUSSION
Rejection may be a foregone conclusion if sufficient evidence has been collected to show that the reliability benchmark is not satisfied (see Part 3: 5.3.3 “Reliability”). Notwithstanding that, the manufacturer may replace a component that has suffered a random failure, or the manufacturer may opt to suspend the test campaign in order to correct a hardware design defect that caused a nonrandom failure.
Source: [VVSG2005] II.1.8.2.6.c
If the test campaign is suspended for an extended period of time, the accredited test lab SHALL maintain a record of the procedures that have been satisfactorily completed. When testing is resumed at a later date, repetition of the successfully completed procedures may be waived provided that no design or manufacturing change has been made that would invalidate the earlier test results.
Applies to: Voting system
DISCUSSION
The considerations for resumption of testing are similar to those of Requirement Part 2: 5.1-D.
Source: [VVSG2005] II.1.8.2.6.d
The test campaign may resume after a deficiency is found if:
- The manufacturer submits a design, manufacturing, or packaging change notice to correct the deficiency, together with test data to verify the adequacy of the change;
- The examiner of the equipment agrees that the proposed change is responsive to the full scope of the deficiency;
- Any previously failed tests are passed by the revised system; and
- The manufacturer attests that the change will be incorporated into all existing and future production units.
Applies to: Voting system
DISCUSSION
Consistent with configuration management, the corrected system is formally a different system from the one that failed. The failure of the previous version is never "purged" entirely; rather, a new revision of the system is found not to suffer the same defect.
Source: [VVSG2005] II.1.8.2.6.e, clarified
2.6 Post-Test Activities
2.6.1 Voting system software version recommended for certification
The following requirements specify the version of the voting system software executable code including application logic, border logic, and third party logic that test labs included as part of a specific voting system recommended for certification.
2.6.1.1 Voting system software version
The test lab SHALL include voting system software executable code including application logic, border logic, and third party logic resulting from either an initial or final test lab build as part of the specific voting system recommended for certification.
Applies to: Voting system
DISCUSSION
The term “test lab build” refers to the voting system software executable code (including application logic, border logic, and third party logic) resulting from the test lab creating the executable code using (a) test lab procured equipment and build tools (such as compilers, linkers, etc.) and (b) source code and build procedures provided by the manufacturers. Note the test lab build is the result of using the requirements found in Part 3: 2.4.3 “Initial system build by test lab”.
Source: [VVSG2005] II.1.8.4.2
When no updates or modifications to the voting system software executable code including application logic, border logic, and third party logic has occurred since the initial test lab build, the test lab SHALL submit the executable code from the initial test lab build as part of the specific voting system recommended for certification.
Applies to: Voting system
When updates or modifications to the voting system software executable code including application logic, border logic, and third party logic has occurred since the initial test lab build, the test lab SHALL submit the executable code from a final test lab build as part of the specific voting system recommended for certification.
Applies to: Voting system
When required by Requirement Part 3: 2.6.1.1-A.2, the test lab SHALL use the requirements found in Part 3: 2.4.3 “Initial system build by test lab” to create a final test lab build of voting system software executable code including application logic, border logic, and third party logic
Applies to: Voting system
Source: [VVSG2005] II.1.8.4.2
2.6.2 Software distribution requirements for repositories, test labs, and manufacturers
The following requirements describe how voting system software must be distributed by test labs, voting system software manufacturers, and repositories such as the National Software Reference Laboratory (NSRL) to support traceability back to a reference version of the voting system software from a test lab, manufacturer, or repository. This traceability provides the basis for verifying that software installed on programmed devices of the voting system is certified voting system software. Although these requirements apply only to test labs, manufacturers, and repositories, other organizations that distributed voting system software such as jurisdictions may apply these requirements to support traceability back to reference versions of voting system software they distribute.
2.6.2.1 Software distribution package requirements
Software distribution packages are used to distribute software between different parties. Software distribution packages contain software from voting system manufacturers, third party manufacturers, test labs, repositories, and jurisdictions. The software contained on software distribution packages include voting application software, election specific software, installation software, third party software, and software integrity information.
Test labs, manufacturers, and repositories SHALL establish software distribution package master copies from which copies are created and distributed.
Applies to: Voting system
DISCUSSION
Software is traceable back to a software distribution package master copy containing the software. Copies of software distribution packages can be distributed on via modifiable media (physically on CD-RWs, memory cards, and hard drives; or electronically via email, FTP, and Websites) since digital signatures are created as part of software distribution packages. (See Requirement Part 3: 2.6.2.1-F)
A master copy creation record SHALL be created that includes at a minimum: the unique identifier of the record; the unique identifier of the master copy; the type of unalterable storage media containing the master copy; time, date, and location the master copy was created; name(s), affiliation(s), and signature(s) of the people present during the creation of the master copy; name and version of the software distribution package; the name, version and certification number (if certified) of the voting system; identifiers of the software components (such as filename(s)) in the software distribution package; location of software components in the software distribution package; and the digital signature algorithm used to sign the contents of the software distribution package.
Applies to: Voting system
A software distribution package master copy SHALL be stored on unalterable storage media.
Applies to: Voting system
DISCUSSION
Unalterable storage media includes technology such as a CD-R, but not CD-RW.
A copy creation record SHALL be created that includes at a minimum: the unique identifier of the master copy; the distribute mechanism for the copy; time, date, and location the copy was created; name(s), affiliation(s) and signature(s) of the people present during the creation of the copy; and the contact information (title, organization, address, phone number, email address, etc.) for the organizations or people to whom copies were distributed.
Applies to: Voting system
DISCUSSION
Copies of software distribution packages can be distributed on via modifiable media (physically on CD-RWs, memory cards, and hard drives; or electronically via email, FTP, and Websites) since digital signatures are created as part of software distribution packages. (See Requirement Part 3: 2.6.2.1-F)
The master copy and copy creation records SHALL be made on unalterable storage media.
Applies to: Voting system
DISCUSSION
Unalterable storage media includes technology such as a CD-R, but not CD-RW.
Test labs manufacturers and repositories, including the National Software Reference Library (NSRL), SHALL retain the master copy of software distribution packages and associated records until notified by the national certification authority that they can be archived.
Applies to: Voting system
Software distribution packages SHALL contain a separate human readable file that provides at a minimum: the name and version of the software distribution package; the unique identifier of the master copy; the name, version, certification number (if certified) of the voting system; and the algorithm used to create digital signatures for the contents of the software distribution package. (See Requirement Part 3: 2.6.2.1-F).
Applies to: Voting system
DISCUSSION
Binary document formats and text containing markup tags are not considered human-readable. Applications may generate such documents, but it must also provide the functionality to render those documents in human-readable form (e.g., by including the necessary reader application).
Software distribution packages SHALL contain a separate human readable file that provides at a minimum the following information for each component within the software distribution package: software component identifier (such as filename), software manufacturer name, software product name, software version, and component location within the software distribution package (such as the full directory path to the file or archive containing the file or memory addresses).
Applies to: Voting system
DISCUSSION
Binary document formats and text containing markup tags are not considered human-readable. Applications may generate such documents, but it must also provide the functionality to render those documents in human-readable form (e.g., by including the necessary reader application).
When software distribution packages use archive files to hold multiple software components, the archive files SHALL be generated using algorithms and file formats in common usage.
Applies to: Voting system
DISCUSSION
Some commonly used archive files include but are not limited to zip, gz, and tarbz2.
The full directory path and filename of archive files SHALL be used as the full directory path for the files within the archive.
Applies to: Voting system
Software distribution packages SHALL contain digital signatures for each software component contained within the software distribution package.
Applies to: Voting system
DISCUSSION
Digital signatures are generated for the un-archived forms of each of the software files as well as archive files.
Software distribution packages SHALL contain, at a minimum, digital signatures generated by the test lab, manufacturer, or repository that created the software distribution package.
Applies to: Voting system
Digital signatures SHALL be stored in a non-proprietary standard data format as part of the software distribution package.
Applies to: Voting system
DISCUSSION
Some non-proprietary standard data formats for digital signatures include IETF RFC 3852: Cryptographic Message Syntax (CMS), RSA Public Key Cryptographic Standard #7: Cryptographic Message Syntax Standard, W3C XML-Signature Syntax and Processing.
Each piece of physical media used for software distribution packages SHALL be labeled on an external surface of the media including at a minimum: the test lab, manufacturer, or repository that created the media; the creation date of the media; unique identifier of the media (such as a serial number); software distribution package name and version; whether the software has been certified or not; and the name, version, and certification number (if certified) of the voting system.
Applies to: Voting system
DISCUSSION
Each piece of media needs to be uniquely identifiable even if the pieces contain the same information in order to support traceability. These requirements apply to master copies of software distribution packages since they are required to be stored on unalterable media. (See Requirement Part 3: 2.6.2.1-A.2).
Each piece of physical media used for software distribution packages SHALL contain a digital signature generated by the creating test lab, manufacturer, or repository covering the entire contents of the media.
Applies to: Voting system
DISCUSSION
The binary image refers to the complete contents of the physical media as a whole. A binary image of physical media may contain multiple files.
2.6.2.2 Repository software distribution requirements
Repositories receive voting system software (source and executable code) that has been certified from test labs or the national certification authority. Repositories may receive non-voting specific software from third party manufacturers and election specific software such as ballot definition files from jurisdictions. Repositories must handle software properly to insure that the software in their possession does not get modified or released to parties without appropriate approvals. However, repositories may be compelled to release software they possess to comply with court orders. Repositories can be described based on the type of service they provide: escrow, notary, and distribution. Escrow repositories hold software they receive until formal requests for the software are received and approved. Notary repositories use software they receive to generate software integrity information (such as digital signatures or hash values) which can be used to verify the integrity of the piece of software. Notary repositories distribute software integrity information, but they do not distribute the voting software or the software used to generate the software integrity information. Distribution repositories provide software they receive to parties approved by the owner of the software. Note that a single repository may provide one or more of the repository services (escrow, notary and distribution). The National Software Reference Library (NSRL) is an example of a notary repository that currently generates software integrity information in the form of hash values. Since source code is not provided to the NSRL, the NSRL only generates software integrity information for executable code.
The repository SHALL publicly document the process used to request copies of the software distribution packages (including associated documentation) from the repository.
Applies to: Voting system
DISCUSSION
Manufacturer approval may be required for release for software considered in intellectual property and needs to be reflected in the request process. Copies of software distribution packages can be distributed on via modifiable media (physically on CD-RWs, memory cards, and hard drives; or electronically via email, FTP, and Websites) since digital signatures are created as part of software distribution packages (see Requirement Part 3: 2.6.2.1-F). When copies of a software distribution package are created, Requirement Part 3: 2.6.2.1-A.3 requires a record to be produced.
The repository SHALL verify the digital signatures associated with software are valid before creating a software distribution package master copy containing the software.
Applies to: Voting system
DISCUSSION
In general, the digital signatures verified by repositories will be generated by test labs, the national certification authority, and possibly jurisdictions.
Results of digital signature verifications including the source of the signature SHALL be part of the creation record of software distribution package master copies created by the repository.
Applies to: Voting system
Distribution, escrow, and notary repositories SHALL create software distribution package master copies containing software received from test labs, the national certification authority, and jurisdictions.
Applies to: Voting system
DISCUSSION
Distribution, escrow, and notary repositories received software distribution packages created by test labs, the national certification authority, and possibly jurisdictions. This requirement establishes software distribution package master copies that support traceability of voting system software back to the repository. Requirement Part 3: 2.6.2.1-A.2 requires software distribution package master copies to be on unalterable media. Requirement Part 3: 2.6.2.1-F requires digital signatures for each software component contained in the software distribution package. Requirement Part 3: 2.6.2.1-A.5 requires repositories to retain software distribution package master copies until notified by the national certification authority.
Notary repositories SHALL create software distribution package master copies containing software reference integrity generated by the repository for software received from test labs, the national certification authority, and jurisdictions.
Applies to: Voting system
DISCUSSION
This requirement establishes software distribution package master copies that support traceability of software integrity information for voting system software back to the notary repository. Requirement Part 3: 2.6.2.1-A.2 requires software distribution package master copies to be on unalterable media. It requires digital signatures for each software component contained in the software distribution package. It also requires repositories to retain software distribution package master copies until notified by the national certification authority.
A distribution or escrow repository SHALL provide copies of the software distribution packages they create to parties that follow the repositories request process (see Requirement Part 3: 2.6.2.2-A).
Applies to: Voting system
DISCUSSION
This requirement allows distribution and escrow repositories to provide the software distribution package they create to parties that follow the request process documented by Requirement Part 3: 2.6.2.2-A. Manufacturer approval may be required for release for software considered in intellectual property and needs to be reflected in the request process of the distribution and escrow repository. Copies of software distribution packages can be distributed on via modifiable media (physically on CD-RWs, memory cards, and hard drives; or electronically via email, FTP, and Websites) since digital signatures are created as part of software distribution packages (see Requirement Part 3: 2.6.2.1-F). When copies of a software distribution package are created, Requirement Part 3: 2.6.2.1-A.3 requires a record to be produced.
A notary repository SHALL provide copies of software distribution packages containing software integrity information generated by the repository to parties that follow the repository’s request process (see Requirement Part 3: 2.6.2.2-A).
Applies to: Voting system
DISCUSSION
This requirement allows notary repositories to provide the software integrity information they create for voting system software to parties that follow the request process documented by Requirement Part 3: 2.6.2.2-A.
2.6.2.3 Test labs software distribution requirements
The test lab SHALL create a software distribution package master copy containing the source and executable code from the test lab build of the voting system software.
Applies to: Voting system
DISCUSSION
This requirement establishes the software distribution package master copy that supports traceability of voting system software source and executable code back to the test lab.
Source: [EAC06] Section 5.6.3.1
The test lab SHALL create a software distribution package master copy containing configuration files, installation programs, and third party software to be installed on programmed devices of the voting system.
Applies to: Voting system
DISCUSSION
This requirement establishes the software distribution package master copy that supports traceability of configuration files, installation programs, and third party software to be installed on programmed devices of the voting system back to the test lab.
Source: [EAC06] Section 5.6.3.1 and 5.6.3.3
The test lab SHALL provide copies of the software distribution packages containing the source and executable code from the test lab build, build environment pre- and post-build binary images, and other software to be installed on programmed devices of the voting system (configuration files, installation programs, and third party software) to the manufacturer, National Software Reference Library (NSRL), and a designated national repository.
Applies to: Voting system
DISCUSSION
This requirement requires test labs to provide a complete copy of the voting system software to the manufacturer, the national certification authority, and National Software Reference Library (NSRL).
The test lab SHALL provide copies of the software distribution packages containing a complete set or subset of the source and executable code from the test lab build, build environment pre- and post-build binary images, and other software to be installed on programmed devices of the voting system (configuration files, installation programs, and third party software) to parties approved by the manufacturer.
Applies to: Voting system
DISCUSSION
This requirement allows test labs to provide complete or partial copies of the voting system software to parties approved by the manufacturer.
Source: [EAC06] Section 5.6.2.4, 5.6.3.2, 5.7.1-5
2.6.2.4 Manufacturer software distribution requirements
The manufacturer SHALL use software distribution packages for voting system software the manufacturer distributes.
Applies to: Voting system
The manufacturer SHALL create a software distribution package master copy containing source code of voting system software including application logic, border logic, and third party logic.
Applies to: Voting system
DISCUSSION
This requirement establishes the software distribution package master copy that supports traceability of configuration files, installation programs, and third party software to be installed on programmed devices of the voting system back to the test lab. Manufacturers will include a copy of this software distribution package as part of their TDP as required by Requirement Part 3: 2.6.2.4-D.
The manufacturer SHALL create a software distribution package master copy containing configuration files, installation programs, and third party software to be installed on programmed devices of the voting system.
Applies to: Voting system
DISCUSSION
This requirement establishes the software distribution package master copy that supports traceability of configuration files, installation programs, and third party software to be installed on programmed devices of the voting system back to the test lab. Manufacturers will include a copy of this software distribution package as part of their TDP as required by Requirement Part 3: 2.6.2.4-D.
As part of the TDP, the manufacturer SHALL provide a copy of the software distribution packages from the Requirements Part 3: 2.6.2.4-A.
Applies to: Voting system
2.6.3 Final test report
The accredited test lab may issue interim reports to the manufacturer, informing the manufacturer of the testing status, findings to date, and other information.
The accredited test lab SHALL prepare a test report conforming to the requirements of Part 2: Chapter 5: “Test Plan (test lab)”.
Applies to: Voting system
Source: [VVSG2005] II.1.8.3.b
Where a system is tested by multiple accredited test labs, the lead accredited test lab SHALL prepare a consolidated test report.
Applies to: Voting system
Source: [VVSG2005] II.1.8.3.c