National Population Survey Postponed — Again
The National Jewish Population Survey, the $6 million study whose release was postponed last November so that its methods could be reviewed, has been postponed again for a second review.
The new delay means that the survey, commissioned by United Jewish Communities and originally set for completion in 2000, will not be published for at least two more months, according to investigators called in to review the survey.
The goal of the new review is “not to point fingers” but to determine if “the estimates are as good as they can be,” said the head of the new review panel, Marc Schulman, president of the research firm Schulman, Ronca & Bucuvalas Inc. “Our committee is just a fresh pair of eyes.”
Schulman is also president of the American Association of Public Opinion Research. He was called in after the head of the initial review, sociologist Bernard Shapiro, vice chancellor of McGill University, reportedly recommended a higher level of technical expertise than his panel was able to offer.
The repeated delays appear to reflect the intense polarization of views among Jewish demographic researchers over the validity of methods used by the survey and its 1990 predecessor. The 1990 survey drew worldwide attention with its finding that American Jews were marrying outside the faith at a rate of 52% per year. Critics said the number was inflated as a result of methodological flaws, and that some flaws have been replicated in the new survey, whose early version showed a 4% Jewish population drop. Some critics said that by inflating rates of assimilation, the surveys caused unwarranted panic in parts of the Jewish community.
“If those people weren’t out there, the Marc Schulman group wouldn’t be coming in now,” said a source close to the survey.
Defenders of the survey, for their part, said the criticism was unwarranted and damaging.
“The review simply injects another note of unnecessary concern that all may not be well with this [survey],” said sociologist Egon Mayer, a member of the survey’s National Technical Advisory Committee.
“At the very least it makes the results of decreasing relevance, because of the delay,” said Mayer who objected to the survey being pulled in the first place. Mayer is co-director of the North American Jewish Data Bank, which is co-sponsored by UJC and the City University of New York.
The survey was partially rolled out last October at a press conference where it was announced that America’s Jewish population had declined by some 4% or 300,000 in the previous decade. A second set of findings was to have been released at a UJC assembly in November. However, the second release was canceled after UJC officials disclosed that some of the survey’s backup data had been lost by the polling company, Roper Audits & Surveys Worldwide, making if difficult to verify the survey’s findings. Sources close to the survey told the Forward at the time that the lost data, while relatively minor, were part of a larger pattern of suspected errors.
A six-member investigative committee was assembled under Shapiro to study the survey’s methods, and a new release date was tentatively set for Passover.
Shapiro’s team reported to UJC in January with a list of glitches and “programming errors” to be addressed before the survey could be released. Among them were missing information in the survey’s “screener,” or opening questionnaire, which may have resulted in an undercounting of the overall population by about 1%.
Last week, however, Shapiro called for a second review by a new panel.
The second review is “part of a process of further rebuilding the confidence of the various publics that are awaiting the results of the study,” Shapiro wrote in a memo obtained by the Forward.
Shapiro said the independent review team, headed by Schulman, would examine his own group’s technical investigations. It would check the study’s statistical sample and the weights used to calibrate the results, and would compare the data with similar studies.
Other members of the new panel include Stanley Presser of the University of Maryland, Eugene Ericksen of Temple University and Tom Smith of the National Opinion Research Center in Chicago.
The review will also explore the “strengths and limitations” of the survey data and, for the benefit of future studies, make recommendations about the methods used to gather the data, Schulman said.
Ira Sheskin, director of the Jewish Demography Project at the University of Miami’s Miller Center for Contemporary Jewish Studies, said it was necessary for top researchers in the field to conduct an independent review because Shapiro’s team was not charged with or equipped to perform an in-depth technical analysis of the survey. Sheskin added that a delay was inevitable even without the second review because the firm that conducted the survey and was blamed for many of its glitches, Roper Audits & Surveys Worldwide, took longer implementing the weighting changes than originally anticipated.
When the survey was initially pulled, several members on the advisory committee decried the action. They said that the missing data was relatively minor, and argued that the delay would do more to damage the standing of the already controversial study. Leaders of UJC and a few members of the advisory committee, however, supported the move as a way to free the study from uncertainty. The same debate seems to be reemerging now.
The co-chair of the technical advisory committee, Vivian Klaff, told the Forward that he is not against the latest review but that the study’s results are sound enough to be released before the review is completed.
Even after the re-weighting is completed, “some things are not fixable,” Shapiro said. He said there are irreparable errors in the logging of some questions surrounding Jewish education and he criticized the decision to alter some 2000 survey questions from their 1990 versions, saying it prevented comparisons on important trend issues in the Jewish community.
“It’s not going to be perfect in the end,” Shapiro told the Forward. “And it’s not going to be totally uncontroversial in the end.”
— The JTA contributed to this report.
A message from our CEO & publisher Rachel Fishman Feddersen
I hope you appreciated this article. Before you go, I’d like to ask you to please support the Forward’s award-winning, nonprofit journalism during this critical time.
We’ve set a goal to raise $260,000 by December 31. That’s an ambitious goal, but one that will give us the resources we need to invest in the high quality news, opinion, analysis and cultural coverage that isn’t available anywhere else.
If you feel inspired to make an impact, now is the time to give something back. Join us as a member at your most generous level.
— Rachel Fishman Feddersen, Publisher and CEO
