Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

CRT:Voting system logic testing

(Votetest)

David Flater2008-01-23

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Players and field

• Congress– Help America Vote Act (HAVA)– National Voting Rights Act– Section 508– Americans with Disabilities Act

• Election Assistance Commission (EAC)– Voluntary Voting System Guidelines (VVSG)– Manufacturer registration, lab accreditation,

certification, …

• Technical Guidelines Development Committee (TGDC)

• NIST– National Voluntary Lab Accreditation Program (NVLAP)

• Voting system manufacturers• Voting jurisdictions• State and local election officials• Concerned citizens• Professional advocates• Academics• Reporters & bloggers

(and they all have lawyers)

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

• Manufacturer-driven activities• Conformity assessment

– Physical configuration audit– Documentation and design reviews– Electromagnetic compatibility and

environmental testing– Logic testing (Votetest)– Volume test (mock election)– CRT benchmarks– STS and HFP testing

• Election Assistance Commission (EAC) certification

• Jurisdiction acceptance testing and certification

• Deployment• Monitoring

Logic testing in context

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Goals

• Status quo: test labs are on their own to develop conformance tests for the Voluntary Voting System Guidelines (VVSG)

• Conservative goal: reduce variability and cost of testing by providing test labs with tools and materials useful in constructing test suites

• Ambitious goal: further reduce variability and cost by providing a canonical test suite

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Choosing the right tools

• “Testing target:” the object of conformity assessment– A.k.a. Implementation/Device/System Under Test

• Different kinds of testing targets need different testing approaches

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Differences from other testing targets

• Automatic testing is not feasible– Don’t have standard interfaces to get data in and

results out– Voters are part of the process (people in the loop)– Unanticipated nonfatal errors must be detected

• Cost of executing tests is a major issue– Significant time and effort to prepare election

definitions, ballot styles, and test ballots or voters for each test case

– Labor costs for people in the loop– Politics: any increase in total cost for certification

will be considered an unfunded mandate

• More is not better– A vote is a vote (logically)– As we increase the number of votes counted

• Cost of testing increases proportionally• Return on investment diminishes rapidly

• Context: one step in a long process– Volume test (mock election), logic verification,

etc.

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

The requirements• Normative reference: the next iteration of the VVSG (in public

review)• Logic must correctly handle all voting variations that the

manufacturer claims to support• Everything must work through the complete elections and voting

processElection definition

Ballot definition

Configuration and calibration of equipment

Logic and accuracy testing

Vote gathering

Tabulation Reconciliation Reporting

1 of M voting X X X X X X X X

N of M voting X X X X X X X X

Cumulative voting X X X X X X X X

Ranked order voting

X X X X X X X X

In-person voting X X X X X X X X

Absentee voting X X X X X X X X

Provisional / challenged ballots

X X X X X X X X

Write-ins X X X X X X X X

Review-required ballots

X X X X X X X X

Primary elections X X X X X X X X

Split precincts X X X X X X X X

Ballot rotation X X X X X X X X

Straight party voting

X X X X X X X X

Cross-party endorsement

X X X X X X X X

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Testing strategy

• All tests are end-to-end tests that exercise the complete elections and voting process

• Small number (10-100) of carefully selected tests– Cover each voting variation with a simple, synthetic test

(around 10 ballots, 1 contest)– Similarly cover all meaningful pairs of voting variations– Few slightly larger tests (around 100 ballots, multiple contests)

based on real sample ballots– Few miscellaneous tests (e.g., boundary cases)

• Test scripts to be “realized” according to the specifics of the target

• Test oracle• No big tests in this test suite

– Context: The big volume test (mock election) provides a significant test of all supported voting variations together

• Punt devilish details– Some requirements are too implementation-dependent– Some requirements are incidental to every scenario– Provided test descriptions but not test cases– Test lab is responsible for complete coverage

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Votetest release strategy

• First release– Based on draft VVSG– “Basic test suite”– Tools and materials– Needs review and feedback

• Second release?– If consensus is that basic test suite is not enough– If there are problems to correct– Sync with finalized VVSG (if applicable)

• Maintenance and support– Keep up with VVSG maintenance (interpretations, errata)– Correct operational issues and coverage gaps as they arise

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Votetest contents

• Data model that supports all draft VVSG voting variations

• SQL* schema that realizes the data model and the tabulation logic specified in the draft VVSG

• Test cases formalized as SQL scripts– We don’t know the interface to the test target– SQL used as surrogate language– Execute as written on the supplied database– Must be translated into whatever is required by the

test target

• Report generator to display results from test oracle• Expected test results• Documentation• Bonus: test generator* Schema uses extensions to ISO SQL

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Votetest environment

Test case execution

Voting system environment

Test case (SQL) Expected resultsReport generatorDatabase

Test case (translated) Actual resultsVoting system Report generator

Translate Compare

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Usability of logic test tools and materials

• Technical expertise befitting an accredited test lab is assumed and required

• Test cases formalized as SQL scripts– More precise than informal test scripts– Automated translation is possible

• The expected output from each test case is provided as a plain text report– Test lab does not need to get the infrastructure to run

on their machines to use the test scripts– Sanity check for running installations

• No huge up-front investment– Hardware requirements: one surplus PC– Software requirements: all free software

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

######################################################################

BEGIN TEST CASE OUTPUT 2007-12-27 15:52:52-05

######################################################################

$Id: 1-basic-1ofM.sql 415 2007-12-27 16:34:15Z dflater $

Small 1-of-M contest, no write-ins, no rejected ballots.Ballot styles: 1Reporting contexts: 1

[... Integrity checks deleted ...][... View materialization log deleted ...]

-------------------------------------------------------------------------------Report for context Precinct 1 generated 2007-12-27 15:52:52-0500

BALLOT COUNTS

Configuration Read Counted------------- ---- -------Total 12 12 Blank 1 1Precinct 1 Style 12 12 Blank 1 1

VOTE TOTALS

President, vote for at most 1Car Tay Fower 4Tayra Tree 3Beeso Tu 2Oona Won 1Nada Zayro 0Overvotes 1Undervotes 1Counted ballots 12Balance 0-------------------------------------------------------------------------------

Report total volume: 76 - Includes optional reporting of blank ballots. - Excludes separate reporting of ballots cast vs. read.

######################################################################

END TEST CASE OUTPUT 2007-12-27 15:52:52-05

######################################################################

• Print header• Reset database

to baseline state• Load test data• Run integrity

checks

• Generate report

• Print footer

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

The oracle

• Design requirement is correctness not performance

• Logic model of draft VVSG translated as transparently as possible into SQL views– Limited expressiveness of SQL means

fewer ways to introduce faults (vs. programming)

– Good news: the logic model itself translates with minimal overhead

– Bad news: straight party voting and write-in reconciliation add a level of complexity

• Informal verification of correctness included in documentation

• Demonstrated scalability up to 2 million ballots

• Results of simple tests are manually confirmed

• Test suite + saved output + shell script = automated regression test

X

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Status as of 2008-01-23

• 3 baseline tests (no optional voting variations required)• 19 single-variation tests covering 12 optional voting

variations• 66 two-variation tests covering 63 combinations of two

voting variations– The other 3 combinations are not meaningful

• 1 three-variation test• 3 tests based on sample ballots• Total of 92 tests

• Working on documentation and presentation• Could improve test generator and do more samples tests• Needs NIST internal review, integration with other test

efforts• No public release yet

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Challenges

• Can’t review prior art—everything claimed as trade secret• Draft VVSG is a moving target—Standards and Advisory

Boards• Accretive release strategy—pressure to get it right the first

time• Realism—no two jurisdictions are alike• Politics

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Demo—Disclaimers

• For demonstration purposes only, we are about to execute a test case in an emulated environment

• This configuration has problems and is not recommended for production use

• The nonfatal error shown below should be ignored

could not remove file or directory "base/55958": Directory not empty

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

Votetest environment

Test case execution

Voting system environment

Test case (SQL) Expected resultsReport generatorDatabase

Test case (translated) Actual resultsVoting system Report generator

Translate Compare

THE DEM

Soft

ware

Dia

gn

ost

ics

an

d C

on

form

an

ce

Test

ing

End of presentation