Political polls are like nutrition science - both are junk food for your mind

Garbage in, garbage out

With the election coming up this week, I’m going to repost a newsletter I wrote after the last presidential election, with a few edits.

The original article was titled “Garbage In, Garbage Out,” but I changed the title this year to better reflect the point - both nutrition science and political polls often serve to mislead more than inform.

Spending your mental energy on political polling and election forecasting is just like paying attention to the latest study on how coffee is bad for you or intermittent fasting is dangerous.

Political polling and election forecasting take up a lot of time and energy for a lot of people this time of year.

It’s not uncommon for many of us to start our morning by “checking the polls” and monitoring trivial changes in levels of support with large margins of error.

This information doesn’t have to be accurate for it to occupy our attention and make us anxious.

And while it’s possible that following the polls inspires action, I suspect that usually it’s just like reading about a new nutrition study.

For most of us, they’re both just mental junk food.

Election forecasting and nutritional science are both fields populated by brilliant people doing high level analytics

When you’re analyzing data to draw conclusions that people will use to make important life decisions, it’s important that your models make sense.

What’s striking about both political forecasting and nutritional science is that they’re both being done by incredibly smart people.

Look at the details of Nate Silver’s methodology. It’s amazingly detailed, with a model that analyzes decades of economic data, adjusts polls by historical accuracy of the pollster, and creates an uncertainty estimate to account for systemic polling errors.

In other words, it’s a brilliant model created by a genius.

A lot of nutrition research is similar. Take a look at this study that made it into the New England Journal of Medicine: Association of Nut Consumption with Total and Cause-Specific Mortality.

The authors analyze data from tens of thousands of patients over decades, they use complex statistical adjustments to account for confounding variables, and update their data on nut consumption every 2-4 years.

Incredibly thoughtful modeling created by some of the brightest minds in research.

Unfortunately, political polls and nutritional science both suffer from the “garbage in, garbage out” problem

When you’re creating a model that will influence the way that people make important decisions in their lives, it’s vital that the model makes sense and that the math isn’t wrong.

But any model is only as good as the data that goes into it.

No statistical analysis can draw useful conclusions from useless information.

And unfortunately, with both polling and nutritional epidemiology, the inputs to these fancy models aren’t accurate enough for us to make important decisions based on.

Polls get it wrong all the time.

Nutritional epidemiology is a field in need of radical reform.

These problems aren’t solved by doing better models.

Just take a look at how nutrition data is collected

For all of these nutrition studies, the majority of data is based on food frequency questionnaires.

Have you ever taken one of those?

Seriously, take a look at it. Here’s an example of a few questions.

How accurately do you think the average person can answer these questions? How accurately do you think that you could answer these questions?

The researchers then do all kinds of fancy statistical adjustments based on individual patient information to draw conclusions like “the frequency of nut consumption was inversely associated with total and cause-specific mortality.”

Polling isn’t much better than nutritional research

You can read a lot about how the polls weren’t as bad as they’re being made out to be or how the forecasts were actually fairly accurate when thinking in probabilistic terms.

Just like with nutrition research, I would argue that this misses the point.

Pollsters made all kinds of changes after their polls were inaccurate in 2016. The 2020 polls incorporated these adjustments. And a similar level of inaccuracy persisted.

I get that there can be disagreement about election forecasting depending on how you model for uncertainty.

But the consistency of polling errors (even if it’s unpredictable which direction the error is going to be), makes me think that forecasting suffers from the same data quality problem that nutrition science does.

The more meta-problem: the constant need for content creation

Political polls and political forecasting is a content factory. Look how much is written each day on any election forecasting site.

Think about how many people “checked the polls” every day during the weeks leading up to the election.

So much wasted time and energy on cable news, Twitter,¹ and in the newspaper pontificating on the polls, why they’re changing, and what it all means.

It’s not really different than the latest Today Show segment about how 4 cups of coffee a day are bad for men.

We’re amusing ourselves to death.

With election forecasting, we’re learning nothing almost new about what might happen in the election despite the constant stream of new polls. And with nutrition, we’re learning nothing new about what’s healthy despite the constant influx of new research.

Garbage in, garbage out.

1. I am not ready to call it X.