EAC - Voluntary Voting System Guidelines

Name : | Cem Kaner
Organization : | N/A
Post Date : | 9/30/2005
General Comments
Comment : | In modern programming, the 100-trillion-test example would be a single subroutine (a function or a method) of a larger program. A voting system might have thousands of subroutines of this complexity. Thus, assessments of any testing strategy must be considerations of the adequacy of the testing effort and not of the completeness. The testing effort is guaranteed to be incomplete, but it might be adequate. Independent testing under government regulations or the most widely used industry standards involves creation of extensive documentation and the maintenance of extensive records that allow a manager, client or auditor to determine what testing has been done, what was tested, when the testing was done, and what the results were. Such records can be of great value, for many purposes. I am not suggesting that the Guidelines relax or revise such requirements. However, recordkeeping comes at a price. If you start with an infinite task, you cannot have enough time or money to complete it. Given a finite budget, you can only do so much. The more you spend on formal planning (and on written description of your plans), on high level and detailed design (and written description of the designs and the many drafts of the designs), on the written description of the individual tests and the tracking of the executions of these tests, and on the many types of reports of testing progress and results, the less you have left for actually running tests and evaluating their results. If each test has a high cost for planning, design, description and tracking, there is a significant impact on the total number of tests that can be run within the budget.