Data is squishy, and other lessons from the Forward’s poll of American Jews
You can make the same survey results say opposite things — but that doesn’t mean they aren’t a valuable tool
Photo by iStock
By Arno Rosenfeld October 22, 2024
“Antisemitism Notebook” is a weekly email newsletter from the Forward, sign-up here to receive the full newsletter in your inbox each Tuesday
The very first article I wrote for the Forward was about a poll. In the four years since, I’ve done dozens more about Jewish public opinion: how they demonstrated lower vaccine hesitancy than other religious groups; were divided between the Orthodox and the less observant; and may or may not believe that Israel is an apartheid state.
I’ve also written about the challenges of such studies: a flawed one claiming that 25% of hiring managers discriminated against Jews, for example, and ADL’s shifting methodology for counting antisemitic incidents.
But the Forward-CHIP50 poll released last week was the first time I had a hand in shaping the questions being asked. We published two stories of our topline findings: Jews say former President Donald Trump is more supportive of Israelis but Vice President Kamala Harris is better on the Gaza war; Jews are divided over the use of police and censorship to control campus protests. Now I want to take you behind the scenes to shed a little light on the complexities of polling Jews about antisemitism.
It’s hard, and expensive, to conduct a good poll. Ours was paid for by the Knight Foundation’s “Election Hub” and conducted by scholars from Northeastern and the University of Rochester. But we had to decide what to ask.
There have been a plethora of polls asking Jews how they feel about antisemitism — concerned, unsafe, pessimistic. But most don’t ask what these people think should be done about it. That’s partly because open-ended questions do not generally work well in polls attempting to take the pulse of a large population (versus focus groups or individual interviews). You don’t see most political pollsters asking voters, for example, “What should we do to help the economy?” Generally, they’re asked whether they support raising tariffs on foreign goods or giving a tax credit to people with young children.
But for all the consternation about antisemitism in recent years, there are not that many concrete proposals out there that people know enough about to poll-test. I picked two, both focused on campus protests: censorship — prohibiting certain slogans, chants or signs — and using law enforcement to control them.
Both ideas had been implemented by some universities with the support of major Jewish groups, and they both seemed controversial enough to yield interesting results. And they did, though the answers also showed the limitation of polling.
Consider these seemingly contradictory findings from our survey of 907 American Jewish adults:
- 55% support banning offensive speech on campus, and 73% support arresting student protesters.
- 61% oppose censorship and a majority do not support arresting student protesters.
How can this be?
Let’s start with free speech. First we asked respondents, “When it comes to campus protests related to Israel, do you support or oppose prohibiting certain political speech?”
Around 30% said they supported doing so, while 47% were opposed and 23% were unsure. If you take out the people who said they were “unsure,” a common but not universal practice in surveys, those numbers shift to 39% in favor and 61% opposed.
Then we drilled down a bit, asking if people would support banning speech that opposes Israel’s existence as a Jewish state (40% said yes, 50% if you remove the “unsures”). What about banning speech that praises Hamas? That had 44% support (55% without the unsures).
A strong majority, 58% (or 73%), support having law enforcement involved in controlling campus protests. But a minority — 45% — said police officers should arrest peaceful demonstrators who refuse an order to leave, with 35% opposed to such a move. Yet when you remove the “unsures,” the share in favor rises to a comfortable 56% majority.
This underscores how even accurate survey results can be manipulated by advocacy groups that commission them to push a particular agenda. I hope it also conveys how hard it is for people operating in good faith — in this case, me and my colleagues — to faithfully convey our findings.
What does it mean that 30% of Jews say they support banning “certain political speech,” but when you specify anti-Zionist or pro-Hamas speech, those figures jump 10-15%? If nothing else, it’s a reminder that you can only fit so much into a question. You can avoid inflammatory wording — I’m sure even fewer Jews would have said they supported “college administrators abolishing free speech on campus,” and probably more would have backed bans on “incitement or threats against Jewish students.” But it’s hard to find neutral language that fully captures what you really want to find out.
Plus, people are full of contradictions; it’s certainly possible that 15% of American Jews dislike Hamas more than they dislike censorship.
None of this should lead you to give up on polls. The data is often valuable, even if it can be hard to parse. But it does underscore the importance of vetting, which I try to do on a regular basis in this newsletter — looking beyond the topline results at how the questions were worded, what the sample sizes are, and other tricky details.
Working on this survey will make me better at analyzing polls for the Forward, but it also showed me how valuable it is for news organizations to be able to conduct their own polls free from the influence of an advocacy agenda.
A message from our Publisher & CEO Rachel Fishman Feddersen
I hope you appreciated this article. Before you go, I’d like to ask you to please support the Forward’s award-winning, nonprofit journalism during this critical time.
We’ve set a goal to raise $260,000 by December 31. That’s an ambitious goal, but one that will give us the resources we need to invest in the high quality news, opinion, analysis and cultural coverage that isn’t available anywhere else.
If you feel inspired to make an impact, now is the time to give something back. Join us as a member at your most generous level.
— Rachel Fishman Feddersen, Publisher and CEO
