Tim Shanahan is a social scientist and consultant who works with the Statistical Training and Techniques (STAT) team of the Fors Marsh Group in Arlington, VA. Tim has several years of survey and opinion research primarily in the educational sector. Tim’s research has been used to support organizational membership engagement and explore novel recruiting strategies for graduate and undergraduate students. Tim’s research has also evaluated the effectiveness of graduate admissions advising as well as the role that gender plays in occupational decision making.
What ever happened to President-elect Thomas E. Dewey that early readers of the Chicago Daily Tribune read about on the morning of November 3, 1948? The mood must have been festive in New York City’s Roosevelt Hotel where Dewey and his campaign gathered to await the results of the 1948 presidential election the night before. Thomas Dewey, the Governor of New York and the Republican presidential nominee, had been so far ahead of incumbent President Harry S. Truman that many polling organizations stopped fielding polls by late October. On election day, some papers even went to press early with headlines declaring Dewey the victor. However, the Dewey administration would never come to pass, as Truman was re-elected with a landslide Electoral College victory. In the aftermath of this high-profile polling failure—exacerbated by the iconic photo—the Social Science Research Council appointed the Committee on Analysis of Pre-election Polls and Forecasts to understand what went wrong. The committee’s eventual findings helped modernize survey practice by identifying numerous features of public opinion surveys that should be changed to reduce survey error.
What is Survey Error?
Surveys are used to collect data about numerous measures, such as the number of rooms in people’s homes as collected by the American Housing Survey, or political beliefs in common election polls. In survey research, error is the extent to which an estimate of something being measured is different from the unobserved true value. Numerous aspects of a survey can introduce error, and quantifying and reducing survey error is critical to modern survey methodology. As any professional survey researcher will tell you, a survey is only as good as its methodology.
The Business Impact of Poor Survey Methods
Poor survey quality can lead to errant judgements that can have disastrous consequences for businesses. In 1997, the United Parcel Service (UPS) suffered a long and expensive delivery driver strike due to their poor read on workforce morale that internal workplace satisfaction surveys produced. In the run up to the strike, UPS executives were dismissive of the threat and toed a hard line in negotiations with the Teamsters Union, because internal workplace surveys had shown high employee satisfaction and company loyalty. However, the Teamsters had fielded their own surveys of UPS workers, which found widespread dissatisfaction about pay, full-time job opportunities, and a strong willingness to strike. The data that the Teamsters collected emboldened them to continue mobilization efforts, aggressive negotiations, and to authorize a strike if demands were not met. The Teamsters went on to strike for 15 days, and UPS lost hundreds of millions of dollars. Although the relationship between UPS and the Teamsters had been tense for some time, a well-designed workplace satisfaction survey would not have produced such disparate understandings of UPS’s unionized workforce and would have shown that the company’s negotiating position was softer than it actually was. Thus, a better survey design might have provided more accurate information to UPS executives about the state of their workforce and could have helped avert a costly mistake.
What You Can Do About Survey Error
Error is present in every survey effort; however, a high-quality survey methodology can reduce the impact of bias on your results. Many professional survey researchers now plan surveys using the total survey error (TSE) framework to make the appropriate methodological decisions that will lead to the lowest possible error while balancing competing trade-offs such as survey precision. Four important sources of survey error that your organization should be aware of for your next survey are:
- Coverage error occurs in two different forms that are traditionally referred to as “undercoverage” and “overcoverage.” Undercoverage results when not all of the members of the target population have an opportunity to be selected for the sample. This can occur if a population member is missing from the population list. Overcoverage results when an entity that is sampled is not a member of the population of interest. Overcoverage is a clear indication that the population list includes list members that are not population members, such as non-voters in an election forecasting poll.
- Sampling error occurs when measurements are not obtained from the entire population. A sample is a subset of individuals who are selected from the sampling frame to receive the survey. The sample serves as a proxy for the population, and the sample’s responses are adjusted through weighting to create survey estimates for the entire population from the sample results.
- Nonresponse error is a form of error that arises when individuals who are sampled do not respond. Nonresponse bias occurs when those who do not respond to the survey are systematically different in some way than those who did respond. Weights and other post-survey adjustments can reduce the impact of nonresponse and nonresponse bias and ensure that survey estimates are both precise and generalizable.
- Measurement error occurs when there is a difference between the response that was obtained in a survey and the true value of what was intended to be measured. For instance, a poorly worded question about the frequency of a certain behavior might produce responses that differ from the actual frequency of that behavior, resulting in measurement error, because the data that was collected was different than what the researcher was actually looking to understand.
Jang, D., Sukasih, A., Lin, X., Kang, K. H., & Cohen, S. H. (2019)
When Fors Marsh Group (FMG) conducts survey research, a number of expert teams collectively work to ensure the quality and accuracy of our survey research projects. For example, for statistical surveys, the sampling and weighting team designs a sampling and weighting plan to minimize the combination of bias and variance and considers other factors that may contribute to total survey error. Weighting is designed to account for the sampling design and nonresponse and can include calibration/benchmarking to the target population characteristics. Our quantitative methods researchers conduct statistical power analyses and design and perform data analyses that account for the sampling and weighting design. Our survey research teams also use cognitive interviews to test questionnaires to ensure that survey questions are clear, easy to understand, and are ultimately collecting data as intended by the researcher.
Learn more about FMG’s survey and data collection capabilities.