Chapter 3: Introduction to General Testing Approaches

3.1 Inspection

Inspection is the examination of a product design, product, process, or installation and the determination of its conformity with specific requirements or, on the basis of professional judgment, with general requirements. [ISO04a]

Inspection is indicated when there is no operational test for assessing conformity to a given requirement. Inspection can be as simple as a visual confirmation that a particular design element or function is present or review of documentation to ensure inclusion of specific content, or it can be as complex as formal evaluation by an accredited specialist.

Logic verification is an example of inspection. Although formal proofs can be checked automatically, the determination that the premises correctly describe the behavior of the system requires professional judgment.

Source code inspections and architecture reviews are also types of inspections.

3.2 Functional Testing

Functional testing is the determination through operational testing of whether the behavior of a system or device in specific scenarios conforms to requirements. Functional tests are derived by analyzing the requirements and the behaviors that should result from implementing those requirements. For example, one could determine through functional testing that a tabulator reports the correct totals for a specific simulated election day scenario.

Functional testing is indicated when the requirements on the behavior of a system or device are sufficiently precise and constraining that conformity can be objectively demonstrated.

Strategies for conducting functional testing are broadly characterized as either "black-box" or "white-box." However, a given test is neither black-box nor white-box. That distinction pertains to the strategy by which applicable tests are developed and/or selected, not to the tests themselves. For example, if a given input is tested because it is a special case in the functional specification of the system, then it is black-box testing; but if that same input is tested because it exercises an otherwise unused block of code found during the review of source code, then it is white-box testing.

Functional testing can be performed using a test suite or it can be open-ended.

3.3 Performance Testing (Benchmarking)

Performance testing, a.k.a. benchmarking, is the measurement of a property of a system or device in specific scenarios. For example, one could determine through performance testing the amount of time that a tabulator takes to report its totals in a specific simulated election day scenario.

What distinguishes performance testing from functional testing is the form of the experimental result. A functional test yields a yes or no verdict, while a performance test yields a quantity. This quantity may subsequently be reduced to a yes or no verdict by comparison with a benchmark, but in the case of functional testing there is no such quantity to begin with (e.g., there is no concept of "x % conforming" for the requirement to support 1-of-M voting – either it is supported or it is not).

Performance testing is indicated when the requirements supply a benchmark for a measurable property.

Usability testing is an example of performance testing. The property being measured in usability testing involves the behavior of human test subjects.

3.4 Vulnerability Testing

Vulnerability testing is an attempt to bypass or break the security of a system or a device. Like functional testing, vulnerability testing can falsify a general assertion (namely, that the system or device is secure) but it cannot verify the security (show that the system or device is secure in all cases). Vulnerability testing is also referred to as penetration testing. Vulnerability testing can be performed using a test suite or it can be open-ended. Vulnerability testing involves the testing of a system or device using the experience and expertise of the tester; using the knowledge of system or device design and implementation; using the publicly available knowledge base of vulnerabilities in the system or device; using the publicly available knowledge base of vulnerabilities in similar system or device; using the publicly available knowledge base of vulnerabilities in similar and related technologies; and using the publicly available knowledge base of vulnerabilities generally found in hardware and software (e.g., buffer overflow, memory leaks, etc.).

3.5 Interoperability Testing

Interoperability testing is the determination through operational testing of whether existing products are able to cooperate meaningfully for some purpose. It consists of bringing together existing products, configuring them to work together, and performing a functional test to determine whether the operation succeeds.

Conformance testing and interoperability testing are fundamentally different. Conformance testing focuses on the relationship of a given product to the standard. As defined in Appendix A, this is what "testing" normally means throughout the VVSG. Interoperability testing, on the other hand, focuses on the practical cooperation of two or more products, irrespective of any standard. Conformance to a standard is neither necessary nor sufficient to achieve interoperability.

Because interoperability testing focuses on practical cooperation, the use of test scaffolding is to be avoided. All of the components should be actual product.