After This Election, Americans Have Lost Faith in Polling. Should Jews?
Image by Steve Pope
By Janet Krasner Aronson, Leonard Saxe, Matthew BoxerNovember 14, 2016
One casualty of the election: people’s faith in surveys.
We relied on polls to help us understand the election and they seemingly failed us. Although pre-election polls came within 1-2% of predicting the national popular vote, state polls that were the basis for Electoral College predictions were far less accurate. Whether and how to use polls is important not only in general but also for the Jewish community, which has relied on survey research to aid decision-making about communal programming, planning and resource allocation.
Despite the simplicity of their questions and the straightforward outcomes that they seek to predict, polls are complex studies that rely on sophisticated statistical modeling. It’s amazing that one can interview a small number of individuals – 1,000 or less across the country – and predict the behavior of 100,000,000 or more people. Key to the success of this prediction is how one selects those 1,000 and how one adjusts for inevitable bias.
The underlying problem is that statistically, virtually the same number of individuals – often no fewer than 1,000 – are required regardless of whether the population that you are interested in assessing is ten thousand or one hundred million. Therefore, the cost and complexity of a survey of a single state or voting district are nearly the same as that of surveying the whole nation. Surveys of single geographic areas are even more difficult, particularly when conducting cellphone or internet surveys. The study of population subgroups, therefore, is one of the critical factors in ensuring the results are valid and useful.
Surveys of the Jewish community face these challenges and more. Because Jews are only about 2% of the population, contacting a random set of households requires, on average, that you reach 50 households to find a single Jewish household. And just as with local election polls, one needs the target number of 1,000 to survey any single subgroup within the Jewish population: those who reside in a local community, those of a certain age (e.g., millennials), or specific demographic groups (e.g., intermarried parents).
Election polls adjust for the bias in surveys by statistically correcting the results to match demographics derived from census data and voter registration. For studies of the Jewish community, census and voter registration data are not available. Proxies can be developed using data from hundreds of high-quality surveys that ask religious identification, in addition to benchmark data such as children’s enrollment in Jewish education and synagogue membership, but it is even more difficult than doing election polls.
In the wake of the election, commentators have suggested that the polling enterprise is suspect and should be abandoned. So too, in the Jewish community, doubts have been cast on the capability of surveys to provide reliable data. Abandoning survey research, however, would preclude one of the fundamental tools for understanding the Jewish community and guiding policy. What, then, should be done to ensure that survey data are accurate and valuable to communal policymakers?
Managing Chutzpah
The primary issue is not whether a poll should be conducted, but how we use and interpret the findings. It is chutzpadik to assume that surveys will yield perfectly accurate measurements of the behavior and attitudes of a population. Surveys are not censuses or actual counts. They yield estimates that are affected by the random and unmeasurable differences between respondents and the population they are intended to represent. Although for political polls the difference of a percentage point alters the outcome, in Jewish community surveys, small differences rarely affect policy.
Focus on Understanding
Instead of focusing on imprecise numbers produced by surveys, the emphasis should be on understanding relationships among variables and groups; for example, understanding which groups are more or less engaged in an activity. Even if the levels of engagement are not precisely measured, they provide insight about how to reach each group. Whether conducting an election campaign or designing programs to foster Jewish engagement, tailoring outreach to specific groups is essential.
Methodology Matters
Quality studies – studies that have sufficient numbers of respondents selected in an unbiased way – are costly. Not every study must meet a gold standard, but users have to be aware of a study’s limitations. Both for electoral polling and Jewish community surveys, studies should be designed with a focus on the highest priority subgroups in order to produce valid information about those groups.
Scientific surveys, whether used to understand the electorate or assess the size and character of Jewish communities, are a tool. Designed and executed properly, they provide valuable information and help correct for our personal experience and biases. The election polls sound an alarm about drawing inappropriate inferences from limited information.
We live in an information age over-saturated with factoids. Yet, the Jewish community needs reliable data to ensure its future. Quantitative information has a sense of concreteness that makes our problems seem deceptively simple. We need chochmah (wisdom) not chutzpah if we are to use polling data to ensure the community’s future.
A message from our CEO & publisher Rachel Fishman Feddersen
I hope you appreciated this article. Before you go, I’d like to ask you to please support the Forward’s award-winning, nonprofit journalism during this critical time.
We’ve set a goal to raise $260,000 by December 31. That’s an ambitious goal, but one that will give us the resources we need to invest in the high quality news, opinion, analysis and cultural coverage that isn’t available anywhere else.
If you feel inspired to make an impact, now is the time to give something back. Join us as a member at your most generous level.
— Rachel Fishman Feddersen, Publisher and CEO
