Complexity

Poppies, acrylic on canvas, 36”x36”

This is the most complex painting I ever made. It took 2 months of 7 day weeks and 10 hour days to complete. I don’t ever want to put that kind of time into a painting again. It looks nice, but less detail can look just as nice, even if less fascinating. Now I have to learn the same lesson with data analysis.

For the last two weeks, I’ve been working on something that I wanted to share last week. However, it’s not done yet because it is very complex. I am comparing 15 state voter roll databases that contain a total of over 126,000,000 records. Because of the amount of data involved, it can take hours to implement any small change in the analysis. I hope I am finally near the finish line, and think I am, but want to be sure before I publish.

The goal is to come up with a way to estimate clone counts in states with incomplete data with reasonable confidence and a statistically sound methodology. The idea is simple: count clones in states that have complete data, then count again as if they had the same incomplete data found in other states. Figure the median difference, and apply that ratio to states with incomplete data, like Wisconsin.

The irony of Wisconsin is that they charge more for a copy of their voter rolls than any state I’ve looked at to date, and the quality of the rolls is worse. In most states, voter rolls are free. Wisconsin charges $12,500 per snapshot, almost as if they want to circumvent public disclosure requirements by making it prohibitively expensive. Then, they don’t include birth date, birth year, or age, making it difficult to ascertain with confidence that a record is a clone.

Wisconsin does include phone numbers and email, but both appear to be voluntary fields because very few records have either. Regardless, the amount that have one or both are enough to be reasonably confident that the number of clones in this limited group alone swamps much bigger states like Texas and Florida.

My hope is that once all the scripts work and the results are in, I’ll be able to write something like, ‘Of Wisconsin’s two million records that match on name, it can be estimated that about 850 thousand are clones attached to the same 400 thousand names, with a 95% confidence interval.’

It won’t tell us which of those records can be confidently described as clones, but it will tell us about how many are likely present.

The only reason I’m making this preliminary post is that I think it was about time to let you know what I’m doing. And as a secondary prize, here is a huge watercolor I made in about 45 minutes, just so you know that detail isn’t everything.

Tuolumne Meadow, Yosemite, watercolor on paper, 55”x 45”