Why Invest in Quality Research?
- Share
Many organizations understand the important role research plays in contributing to their bottom line. Conducting focus groups prior to the launch of a new advertising campaign or the release of a new product is mandatory in many industries, and soliciting input from actual consumers on an ongoing basis is taken as a matter of course.
While it is true that research efforts can ultimately save organizations significant time and resource investment by providing critical insight into their given market, poor research methods and the desire for a fast turnaround often lead to results that cause more problems than profit.
If the appropriate level of preparation is not taken and well-established research methods are not employed, the resulting data may not provide any valuable insight into the market and, at worst, may provide a faulty view of that market, which could lead to costly mistakes.
What Can Go Wrong?
All too often, poorly designed research efforts produce more error than substance. To avoid these costly mistakes, researchers must prevent errors from occurring by using rigorous methods and a scientific approach to design and execute research. Below we describe two of the most common survey errors, provide examples, and discuss Fors Marsh Group’s approach to minimizing these errors.
Sampling Error
Prior to engaging in any research, it is important to establish the population of interest. Typically, once this population has been defined, quantitative research designs allow inferences about that population to be made based on information provided by a subset (or sample) of that population. Thus, the quality of the data depends greatly on the quality of the sample from which the data is collected. Sampling error refers to any inadequacies in choosing and successfully reaching an unbiased, representative sample of the target population. These errors stem from improper attention given to designing a sampling strategy (i.e., what groups do you want to reach, how many, how will you identify and contact them). Choices in the initial design phase (e.g., developing a pool or “frame” of contact information from which you will draw your sample, technique for sampling within the frame, contact method and timing, response mode, etc.) must all be carefully considered in light of cost implications, data quality, and the desired precision of the sample’s data.
All too often, the sampling stage of a survey project gets cursory attention. A group of individuals is often contacted without considering the harmful consequences collecting data from a nonrepresentative sample may have. A classic example of a sampling error occurred in the presidential election of 1948. The night of the election, the Chicago Tribune printed “DEWEY DEFEATS TRUMAN” as the headline for its post-election morning edition. However, that headline was wrong, and the picture of President Truman holding the paper bearing this headline is now famous. It turns out that the Tribune’s historic mistake was a result of sampling error.
The editor trusted the results of a telephone survey at a time when telephone users were not representative of the population. Telephones were not common in many households then and those with them typically had higher income and more stable addresses. As a result, the voting population was not accurately represented by the sampling method and the consequence is now a historical black mark on the record of survey research. Interestingly, the same problem with telephone surveys is now re-emerging with the increasing predominance of cell-phone usage. Fors Marsh Group is tracking this phenomenon and actively experimenting with and implementing other modes of contact (e.g., mail, Web) to ensure new strategies are readily available to seamlessly transition clients who have historically relied on telephone polls to new modes without affecting the validity or representativeness of their data.
Measurement Error
Similar to identifying the population of interest and ensuring that they are accurately represented in the study, it’s important to identify what you want your survey to measure and ensure the instrument consists of questions that accurately measure the desired information. Measurement error commonly occurs when questions:
- Fail to cover the intended domain
- Measure concepts different from the ones they were designed to measure
- Are poorly worded or otherwise confusing
Too much measurement error can result in worthless survey results since little can be done after the fact to correct for poor measurement. Measurement mistakes often involve poorly constructed questionnaire items. These pitfalls severely undermine the validity of conclusions based on the data collected by questions such as:
.png)
Fors Marsh Group Approach
Fors Marsh Group approaches research design as a science that requires thoughtful collaboration and investigation to ensure the customer’s goals are addressed and information is validly gathered with precision. Fors Marsh Group has developed and refined a rigorous approach to survey research - from sampling methodology, to measurement design, through implementation. We have the requisite knowledge, skills, and abilities to prevent and mitigate errors that commonly plague survey research. Our expertise and approach has served as the basis for a variety of large-scale government survey efforts. Our results have served as a basis for successful changes in many recruiting strategies, advertising efforts, and have lead to improvements in call center customer service strategies. We pride ourselves on conducting scientifically based research that provides our clients with unbiased results to answer their research questions, and providing action-oriented recommendations to help them reach their strategic goals.