Q&As about NIST Evaluation of Laboratories that Test Voting Systems

Background

The Help America Vote Act (HAVA) of 2002 directs the National Institute of Standards and Technology (NIST) to support the U.S. Election Assistance Commission (EAC) in its accreditation of laboratories qualified to conduct the testing, certification, decertification, and recertification of voting systems as provided under the act.  NIST processes for carrying out this responsibility are as open and transparent as possible to facilitate the public's understanding of how laboratories that test voting systems are evaluated.  

Q. What is NIST's role in the accreditation of laboratories that test voting systems?
HAVA requires that NIST conduct an evaluation of independent, non-federal laboratories to determine their competence to test voting system hardware and software for conformance to federal standards.  HAVA also specifies that NIST recommend to the EAC those laboratories that are qualified to test voting system hardware and software. The EAC will make the final decision to accredit a Voting System Testing Laboratory (VSTL) to test and certify equipment under EAC requirements.  

Q.  What process is NIST using to evaluate laboratories?
NIST is relying on assessments conducted by its National Voluntary Laboratory Accreditation Program (NVLAP) as a basis for determining the competence of candidate laboratories to test voting system hardware and software for conformance to federal standards.  

Laboratory accreditation is a formal recognition that a laboratory is competent to carry out specific tests.   It also allows a laboratory to determine whether it is performing its work correctly and to appropriate standards.  

Expert technical assessors evaluate all aspects of laboratory operation that affect the production of test data, using recognized criteria and procedures.  General criteria are based on the international standard ISO/IEC 17025, General requirements for the competence of testing and calibration laboratories, used for evaluating laboratories throughout the world.  Laboratory accreditation bodies use this standard specifically to assess factors relevant to a laboratory's ability to produce precise, accurate test data, including the technical competence of staff; validity and appropriateness of test methods; and testing and quality assurance of test and calibration data.  NVLAP includes this standard in NIST Handbook 150: NVLAP Procedures and General Requirements, available at http://www.nist.gov/nvlap.  

Q.  How does NVLAP determine the competency of laboratories to test voting systems?
o be accredited by NVLAP, a laboratory must demonstrate both its general technical competence and its competence to perform a core set of voting system tests. Currently, laboratories are using proprietary test methods and test cases to determine that a voting system meets existing federal standards. The federal standards are the 2002 Voting System Standards (VSS) (see http://www.eac.gov/voting_sys_cert.htm) and EAC­ adopted 2005 Voluntary Voting System Guidelines (VVSG 1.0) (see http://www.eac.gov/vvsg_intro.htm).  Technical criteria for the NVLAP voting system testing laboratory accreditation program are contained in NIST Handbook 150-22: NVLAP Voting System Testing, available at http://www.nist.gov/nvlap.  

Q.  Why are laboratories using proprietary test methods?
Currently, no uniform set of tests exists to determine that a voting system meets federal standards. With the support of the EAC, in 2007 NIST began to develop a uniform set of non-proprietary tests to be used in conjunction with the next version of the Voluntary Voting System Guidelines (VVSG 1.1). The availability and use of these open tests will improve consistency and comparability among testing laboratories.  

Q.  What specific tests must a laboratory meet to be judged competent?
The NVLAP assessment includes a thorough evaluation of all aspects of laboratory operation that affect the production of voting system hardware and software test data, including the adequacy of the laboratory's equipment and facilities; its system for documenting what testing is conducted; technical staff qualifications; and staff training requirements.  The core of the assessment takes place during an on-site review and focuses on the laboratory's demonstrated competence to do the following:  

• Review the overall system design and technical specifications for a voting system to ensure that it conforms to federal standards.
• Compare the voting system components submitted for qualification to the vendor's technical documentation (also known as a configuration audit).
• Review the "source code," the voting system's "human readable" programming instructions to verify that the code is unmodified and that settings are correct.
• Conduct an audit of the voting system's configuration to ensure that all functions work as expected and match the system's manual and other documentation.
• Test the capabilities of the voting system as a whole to ensure that it meets the requirements and that the function of any source code included in the system but not designed to meet these requirements is identified.
• Verify that the voting system reliably records votes accurately at its maximum processing volume for a specified period of time.
• Ensure that the voting system provides adequate security to prevent unauthorized access or data interception and/or disruption.

For more details on these core tests, see attachment.

Q. What does the accreditation process entail?
The accreditation process includes the time that it takes for a laboratory to submit its application and supporting documents for review by NVLAP. This process can take from nine to 18 months; including the time it takes for a laboratory to submit documentation for review by NVLAP, schedule a pre-assessment and an official on-site assessment, review assessment information from the on-site visit, and clear up any non-conformities.  Laboratories that do not achieve accreditation within 12 months of their initial application must reapply to NVLAP to keep their application active.

Q.  Does NVLAP conduct follow-up assessments?
Yes, to ensure continued compliance with accreditation criteria, all NVLAP-accredited laboratories undergo another onsite assessment during the first year following initial accreditation and every two years thereafter.  NVLAP also can conduct monitoring visits at any time during the accreditation period; these may be unannounced.  

Q. What documentation related to the accreditation of VSTLs is publicly available and what is not?
NIST makes publicly available non-proprietary information, including the report generated by NVLAP assessors as a result of the on-site assessment of each accredited laboratory and the laboratory responses to the on-site findings. NIST also makes publicly available the assessor checklist used during on-site assessments.  By law, NIST must protect proprietary information. This includes details of a laboratory's specific testing methods and protocols.  

Q.  Will the EAC have access to proprietary data?
To assist the EAC in making the final decision on accrediting a Voting System Testing Laboratory (VSTL), the EAC has access to all the data, including proprietary information.

Q. Does NIST also accredit vendors of electronic voting systems?
No, NIST does not accredit vendors of electronic or other types of voting systems.  The EAC has established requirements for voting system vendors and their relationship with testing laboratories, contained in its Testing and Certification Program Manual, January 2007.