Boulder County is once again doing a pretty comprehensive and savvy election audit, and you are invited to nominate some batches to be targeted for hand-counting! They have a press release on it.
The general auditing goal is "software independence", so we don't have to trust the black-box voting system we have. We do that by looking for the best independent evidence we can find for whether the preliminary outcomes of the election are correct. Specifically, we take a detailed report on the election results by batch, and carefully (but mostly randomly) select batches for hand-counting, to compare with the preliminary results.
The theory behind this is laid out at Principles and Best Practices for Post-Election Audits.
Boulder produced unusually fine-grained data for this audit, which helps greatly with the efficiency of the audit and confidence levels achieved. In fact, at a meeting at the American Statistical Association two weeks ago, we published a statement on the importance of small-batch reporting. Rather than reporting by big precincts, Boulder reported in batches averaging 150 ballots each. Vote counts for all 41 contests in each of batch are available.
Boulder also continues to pioneer a "budget-based" audit approach, which tries to maximize the confidence we can get out of auditing a given number of ballots. The initial budget for this audit is 21,000 ballot-contests (e.g. one contest on each of 21,000 ballots). So I'm guessing they'll be auditing about 140 units, which is an impressive number. But since the batches are small, the counting should be more accurate than with full-precinct audits, and the overall effort is reduced, without sacrificing statistical efficacy.
They're seeking nominations (e.g. of "unusual" results) for a given contest in a given audit unit and would love you to apply your own analysis or intuition. E.g. say "I nominate the 'City of Louisville Ballot Question 2A' contest in audit unit p005_mb_568 because the winner seems to have a surprisingly large share in that one" (which probably isn't true, but you get the idea). More details would be welcome. One technique used last year by auditing guru Mark Lindeman was constructing a model to predict votes in one contest based on how voters in the same batch voted in other contests, and identifying vote counts that seemed out of character.
November 11 is the deadline to submit nominations. They will probably choose some at random, and I'm hoping they'll also choose some on merit.
Check out the audit page and all the data at
http://www.bouldercounty.org/clerk/elections/2009ElectionAudit.htm
Helpful "Usage Notes" are in the third zip file there (the only one you really need, which contains the processed data to be audited and the spreadsheet that will actually implement the audit), i.e.
http://www.bouldercounty.org/clerk/elections/Audit%20Files/AuditSetup-Contest_MBB_Selection.zip
It also includes their detailed plan. The plan doesn't quite match the current state-of-the-art for dealing with discrepancies (represented by Philip Stark's work in California), and having an effective canvass of the chain of custody and reconciliation of the number of ballots in each ballot style is also necessary. Verifying chain-of-custody for mail ballot elections is a very very hard task of course. But this audit is way ahead of most of the country. Hillary Hall, Don Hayden, and the rest of the team should be commended on their work.
See other election info here: http://www.bouldercounty.org/clerk/elections/
This is all based on the audit we did last year in Boulder: http://bcn.boulder.co.us/~neal/elections/boulder-audit-08-11/
Your comments and analysis would be most welcome!
|
I'm happy to report two pieces of good news for ElectionAudits. I'll be giving a talk in July at OSCON titled ElectionAudits: a Django App for Advanced Election Auditing.
You can use "OS09PG" as a discount code to get 15% off during OSCON registration.
Also, the April 2009 issue of Python Magazine has an interview with me about ElectionAudits. It is available from the site as a paid pdf download (68 pages in the April issue, which also includes other voting coverage, like Mitch Trachtenberg's open source Ballot Browser!). Thanks to these collaborations between open source geeks and visionary clerks like our own Hillary Hall, and Humboldt County's Carolyn Crnich, we're making significant progress!
The Daily Camera's Laura Snider wrote a very good piece on the Boulder audit at New election audit targets close races - County is the first to use a 'risk-limiting' audit.
See all the data and results of Boulder's audit for lots more details.
I think we were the first in Colorado to do a risk-limiting audit, and one of the first in the nation, but I know that e.g. in California's Marin county last February Philip Stark did a risk-limiting audit of a ballot measure, and his team is doing them in 4 counties (Yolo, Humboldt, Marin and Santa Cruz) for the recent general election.
I should also update my previous article on Audit Problems in Colorado, where I said that the Colorado audit rule doesn't require an audit of the actual counting process. As a result of the lawsuit last year, the Secretary of State did require, as a special condition of certification for several election system in Colorado, that counties audit some contests for a single actual election-day mail-in batch. I'm looking forward to seeing how the counties actually implemented that. See the conditions of use for the details.
The best story I've seen so far on the "paper dust" investigation in Boulder is from Laura Snider at the Boulder Daily Camera. Or maybe I'm biased since she called me and put in a quote or two 8-)
http://www.dailycamera.com/news/2008/nov/05/boulder-countys-counting-crawls/
From the limited information I have, it seems to me that the magnitude of the problem is relatively small and might not be noticed by counties that don't look closely at their ballots via a good audit or the sort of screen images that the BallotNow system provides (despite its many flaws, that is an interesting aspect of the Hart system).
So how closely are other counties looking at their ballots, from systems known to have problems?
Snider writes:
Last winter, Secretary of State Mike Coffman de-certified all Hart scanning equipment because the scanners "failed to count votes accurately when there are extraneous marks on the ballot." ... Coffman's testing board recommended that he institute regulations requiring county officials to review every ballot, looking for the stray marks. Instead, Coffman chose to re-certify the machines, which are used in 47 Colorado counties, without the extra regulations.
That creates the possibility that some counties using the Hart scanners may be having dust problems and not know it. Improved audits were required by the conditions of use imposed by the Secretary of State.
But who besides Boulder in Colorado is really doing good audits of mail-in ballots?
Is your county addressing the many flaws in Colorado's auditing procedures (see my related article) ?
ElectionAudits Software News
Help audit the election!
Proper audits are the gold standard of election integrity - they can tell whether the election system actually counted the real ballots properly. Unfortunately, they are rarely done properly. We need better audits of our elections in order to promote confidence in the results. Thankfully, there are resources to make it easier than it has been in the past, both in terms of procedures and statistical research, and in terms of open source software. If you are willing to talk to your local elections officials, you can make a difference. Read the Background section below for more information.
You can:
Join the ElectionAudits Team - this will get you on the mailing list.
Ask your local officials if they follow the Principles and Best Practices for Post-Election Audits
Attend a Logic and Accuracy Test (LAT) of your county election system
Get data from the LAT and test the ElectionAudits software
Ask when the report your county intends to audit will be released
Ask when the random selection will be done and how. Make sure the selection isn't done before they release what they intend to audit (like most of Colorado unfortunately!)
To do my part, I've worked with the Boulder County Clerk and started the ElectionAudits open source software project to help us audit elections with appropriate statistical significance, and I need help there too! If you're skilled with Python or Django (especially on Windows since it is developed on Ubuntu Linux), database query optimization, xml, xslt, css, setuptools for packaging cross-platform easy_install eggs, statistics, documentation or the like, see the ElectionAudits Software Home Page, download the latest code, and please lend a hand:
http://neal.mcburnett.org/electionaudits/
Background
We all know there are many questions being raised about the systems used to count votes in our elections. The bottom line, as noted in a paper by MIT voting and security expert Ron Rivest and NIST voting expert John Wack, is the need for "Software Independence":
http://en.wikipedia.org/wiki/Software_independence
meaning that election software should not be a critical component which, if it fails, can threaten the results of an election.
That is why we've moving from using unauditable paperless DRE machines to systems based on paper ballots that can be audited. And we're slowly getting there. But even for optical scanner systems we also need to check the software that does the actual tallying and reporting of the results.
Those paper ballots are of little use if we never audit them. And we still have a long way to go on effective audits of elections: the voting systems don't support audits well, the laws and regulations are sparse and often inadequate, and the topic is confusing to many.
This prompted California Secretary of State Debra Bowen to establish Post Election Manual Tally Requirements and issue California proposed emergency regulations for them.
Along with that, recently, experts in elections, statistics, computer science, risk assessment and related fields have been working on better audits, coordinated by ElectionAudits.org, the nation's clearinghouse for election audit information. The latest results are the Principles and Best Practices for Post-Election Audits.
I've been very active in that effort, and convinced Boulder County to do a complete audit, far better than what is currently required in Colorado, starting with an audit of the 2008 Primary in Boulder:
http://bcn.boulder.co.us/~neal/elections/2008-boulder-primary-audit/
Based on that experience, it was clear we needed software to help deal with the inadequacy of the reports available from our Hart InterCivic BallotNow system, and to make it clearer how to base the work on good statistics. So the software includes the "varsize.py" statistical software from Ron Rivest to guide efficient sampling of the audit units.
See the demo and more at http://neal.mcburnett.org/electionaudits/
Colorado needs to update its election auditing procedures. Rules that follow the "Principles and Best Practices for Post-Election Audits"
at
http://electionaudits.org/principles would be the best. California Secretary of State Debra Bowen has put forward a good first step with these Post Election Manual Tally Requirements:
http://www.sos.ca.gov/elections/voting_systems/ttbr/post_election_req.pdf
and has responded to pushback from counties with these proposed emergency regulations:
http://josephhall.org/tmp/post_election_audit_regs.pdf
In Colorado the problems are worse than the existing California rules. Below is a list of places where Colorado doesn't follow the Best Practices. Colorado should follow the lead of Boulder County, which is addressing all of the following problems, as described at http://bcn.boulder.co.us/~neal/elections/2008-boulder-primary-audit/.
No real audit for mail-in ballots or central-count systems
Mail-in ballots will be heavily used in this election, and will typically be counted on "central count" systems. But the Post-Election Audit Rule, 11.5.4.5, from the Secretary of State is out of compliance with the Colorado Revised Statutes 1-7-514, for the case of central-count election machines, and does not specify an audit at all. It calls instead for a post-election test on live ballots. The purpose of an audit is to ensure that the device worked during the actual counting process, not that it can be made to work in a separate count afterward. The "Logic and Accuracy Test" rules already address testing. For precinct-count scanners and DREs, the law and rule are consistent in calling for audits of previously reported election results. The rule for central count machines should call for no less.
The root problem is that some central count systems used here don't report the original election day results that correspond to each batch of paper ballots fed in to the scanner.
The existing precinct-level reports can't be used, because in Colorado the physical mail-in ballots are typically all mixed together, not sorted into piles by precinct. When it comes time to audit a particular precinct from the report, the auditors would have to go through all of the mail-in ballots to find and audit the ballots for that precinct.
The alternative is for counties to sort the physical mail-in or early paper ballots into piles by precinct as they arrive. Many other states do this, although auditing the early votes on DRE machines remains a big problem, since the rolls of paper containing the VVPATs contain ballots from all precincts, mixed together.
The best solution is to either have the vendors do a simple report by batch, or to find another way to generate audit reports containing the results of each physical batch of paper ballots, as Boulder does.
Random selection prior to auditable public results
The choice of which machines have been selected to be audited is available for misuse before the results to be audited are typically produced and shared with the public, since the SoS selects machines within 24 hours after the close of polls. Thus anyone intent on corrupting the results would have an opportunity to be sure to only corrupt them on machines that won't be audited.
Instead, the machines or batches to be audited should not be selected until clerks first publicly release the unofficial results for each contest on each machine (or batch). And that audit report is what should be audited.
Random selection not transparent
The selection should also should be in public, using dice (Cordero, Wagner and Dill 2006) or perhaps a publicly verifiable random selection method based on public random numbers. But it seems that software is used to do the "random" selection, and it is not clear how the public can be sure it really is random. In the 2008 Primary, races that were uncontested or not even in the right county were selected, as documented in my Boulder 2008 Primary Audit writeup.
Audit of central count machines too small, doesn't comply with the law
Only a small fraction of central-count ballots are audited in many counties (a maximum of 500 per central count machine even if it counts e.g. 40000 ballots - Rule 11.5.4.5 - that would be 1.25%). This is not even the "specified percentage" which is specifically called for in Colorado Revised Statues 1-7-514: "where a central count voting device is in use in the county, the rules promulgated by the secretary pursuant to subsection (5) of this section shall require an audit of a specified percentage of ballots counted within the county."
Not a high enough percentage
Even in cases where there are 5% of ballots audited, the contests to be audited are spread out among all the machines to be audited, so very little auditing is done on each contest. E.g in Boulder County, 5% of the machines is perhaps 16 machines total, and the 71 contests to be audited are spread out over those machines. So only perhaps 5 out of the 71 contests in play will be audited on each machine selected, implying only 0.35% of the ballots per contest. Races that are uncontested are included in the selection, which is a waste of effort. California, by contrast, requires a 1% audit per contest.
Lack of paper audit trail in some counties
Some counties are not requiring that a paper record exist for all ballots, and those electronic DRE "ballots" can't be audited at all.
Wasteful focus on landslide contests, and little confidence for tight contests
The audit is wasteful for contests with a wide margin of victory and inadequate for races with a tight margin of victory, and is not designed to limit the risk of certifying an invalid result. State-wide contests that have a wide margin don't need much auditing at all - a Judge approval contest with a 32% margin might merit selecting only a half dozen machines state-wide, while a race with a one percent margin needs one or two hundred - more than one audit unit per county.
This is an early version of this report. Updated: Fri Feb 8 12:48:48 MST 2008
Longmont council race audit, Jan 2008
On January 29, 2008, the the Boulder County Elections Division ran the Longmont Special Municipal Election for the city of Longmont, Colorado, to choose a new City Council member:
http://www.co.boulder.co.us/clerk/elections/Jan2008_Longmont.htm
An actual audit using Hart's BallotNow
It was an all-mail-in election, with a single contest and four candidates. The counting process used central scanners (the Hart InterCivic BallotNow equipment). Because these scanners had been recently decertified in Colorado, the County Clerk, Hillary Hall, recommended a new way of auditing the results: checking the hand-count results against the original machine count of those ballots, rather than against machine recounts of the selected batches. Thus this would be a real audit, not the partial recounts disguised as audits, as have been done previously in Colorado.
The clerk also recommended a 10% audit of the results via a hand count. The city council, noting the small size of the election, asked for a larger audit: 20%.
The reason that real audits have not been done (though they are required by state law) is that most of the machines don't produce enough information to do an audit. In particular, the Hart BallotNow/Tally system can produce only precinct results, not results with subtotals by contiguous batches of ballots that can later be identified and hand-counted to compare with the machine count. So when central-count scanners are used to tabulate ballots that have not been sorted by precinct, it is impractical to find all the ballots for a given precinct so that the machine-reported precinct results can be audited.
For this election the clerk recommended doing extra steps during tabulation so that an independent report of vote counts for each batch could be generated. Note that the Hart system separates the BallotNow scanner from the Tally computer. BallotNow scans the ballots into image files, interprets them, and uses human intervention to interpret ballots that seem overvoted. Tally integrates input from BallotNow and from other optional components like DREs and generates tallies and reports. The only documented way to get a tally for an individual MBB is to generate the running totals before and after tallying the MBB and subtract the before-and-after results for each contest.
The process used by the clerk's team involved separating the 13961 ballots into 146 batches of about 100 votes each. Each batch was run separately through BallotNow. The team used 146 separate memory cards (called "Mobile Ballot Boxes" or MBBs) to store the scanning results (not yet tabulated into vote count subtotals) of each batch on a separate MBB. Each MBB was then read into the Tally system. The Tally system was then told to generate a full running tabulation and report in comma-separated-values (CSV) format after each MBB. The report gives the TOTAL for each contest for all MBBs (that is, for all counted batches) read up to that point. Each of the 146 election reports was saved, and then a spreadsheet was used to generate individual subtotals by batch, by subtracting the results of the previous total from the desired total.
A spreadsheet is available with these sheets:
BatchSummary sheet: The "auditable vote-count report" that was audited.
ResolutionNotes sheet: Notes on some anomalies during the scanning process
Unusual Batch Search: my quick-and-dirty analysis of which batches had the largest margin for the winner vs. the runner-up.
The overall result was 7595 (54%) for the winner (Gabe Santos) and 4338 (31%) for the first runner up (Richard Juday), a winning margin of 23%. The total number of votes was 13961.
The margins in the individual batches ranged from 47 votes to -3 votes (that is, 3 more votes for runner-up Juday than Santos). The average winning margin for Santos was 23 votes (out of 100) per batch.
These data are useful for analyses of the statistics of the variation in the batch counts. It would seem likely that in many cases the mail-in processing would cause some randomization of the ballots so that batch counts could be expected to be more uniform than precinct counts. I don't know if the batches are labeled by whether the ballots were hand-delivered or mailed in, and it is possible that voters who chose mailing vs hand-delivery would have different candidate preferences. I wonder if there are any other data sets out there like this.
The handwritten audit tabulation records the hand tallies for the 28 batches that were audited. There were two discrepancies of a single vote each. One (probably batch D-065) was determined to be a miscount by the hand-counters, and the other (batch C-063) was a difference in interpretation of a ballot. The BallotNow system shows ballots that it thinks were overvoted to a "resolution team" during normal election-day counting. The team inspects the ballot (actually the scanned image, projected on a screen) to see if the voter's intent was clearly and legally indicated. The original resolution team said this particular vote was an overvote, by law, but the hand-counters said it was a vote for one of the candidates. The intent seemed clear, but the clerk says the law requires that this particular ballot be counted as an overvote. I haven't seen the ballot or studied the law, which she says should probably be clarified or changed....
TODO
- Explain how batches were selected.
- Provide statistical confidence analysis.
- Note advantages of having more batches.
- Document time necessary for batch preparation, hand-counting ballots, etc.
- Advice for future audits:
- Design audit size to meet a given confidence level, not a set percent of the ballots: easier for wide margins, more accurate for close contests
- Auditable report should be published before selecting batches to audit.
- Improvements to random selection process.
- Discuss issue with identifying voter intent.
- Links to "best practices" for audits.
Acknowledgments
Thanks to Boulder Clerk Hillary Hall and her team for proposing and implementing this historic audit, and the Longmont City Council for its support. Thanks to Mary Eberle for her public testimony, private pursuasion, and editing talents.
See also
County to audit city ballots: 20 percent of special election votes to be examined by hand http://www.timescall.com/News_Story.asp?id=6032
1% Manual Tally Observer Report of CA Congressional District 11 January 10, 2007 by Judy Bertelson http://www.countedascast.com/docs/CD11_Manual_Tally_Report_Jan01.pdf
David Wagner: Notes on the 1% Manual Tally in Yolo County (2006) http://www.yoloelections.org/news/snews/reactions.pdf
Posts