Printer-friendly versionPrinter-friendly version

Section 1.2. Chapter 3

Overview

Suppose you desire to do a study or administer a survey. As an investigator, the most challenging task that you will confront is to decide what questions to ask and/or what measurements to obtain. In this chapter you will be introduced to some key definitions associated with obtaining measurements. You will also learn about possible pitfalls found with survey questions.

It's All in the Wording

Chapter 3 lists seven possible pitfalls that can lead to bias when asking questions in a survey or study. In thinkinking about the veracity of results from a sample survey ask yourself if any of these pitfalls are likely to be a problem.  It is also possible that more than one type of pitfall can happen at the same time. Examine the following examples.

Example 2.4. Deliberate Bias such as One-Sided Statements

People who use a form of deliberate bias often desire to gather support for a specific cause or opinion.  Consider two different wordings for a particular question:

Wording 1: It is hard for today's college graduates to have a bright future with the way things are today in the world.

a. agree
b. disagree

Wording 2: Today's college graduates will have a bright future.

a. agree
b. disagree

Although Wording 1 and Wording 2 are contradictory statements, when both questions are used in the same survey, it is not uncommon to find that people answer "agree" to both questions. This is because respondents tend to agree to one-sided statements. Listed below are revised wordings for these two questions. These choices are preferred because the statements are now at least two-sided.

Revised Wording 1: Do you agree or disagree that it is hard for today's college graduates to have a bright future with the way things are today in the world?

Revised Wording 2: Do you agree or disagree that today's college graduates will have a bright future?

Example 2.5. Filtering

Consider two different choices of answers for a particular question:

Choice 1: What is your opinion of our current president?

a. favorable
b. unfavorable

Choice 2: What is your opinion of our current president?

a. favorable
b. unfavorable
c. undecided

This example illustrates the problem of "filtering." Filtering exists when certain choices such as "undecided" or "don't know" are not included in the list of possible answers. People tend to provide an answer of "undecided" or "don't know" only when these choices are included in the list of possible answers.

Example 2.6. Importance of Order

Consider two different wordings for a particular question:

Wording 1: Pick a color: red or blue?

Wording 2: Pick a color: blue or red?

The results in Table 2.1 are from a study conducted in a Statistics class.  As you can see the results vary somewhat based on the order in which the colors are presented. Even though many people probably have a preference for one color over the other, if order does not matter, the percents should be same with each wording.

Table 2.1. Bias due to Order of Comparisons

Color Choice Wording 1 Wording 2
Red 59% 45%
Blue 41% 55%

Example 2.7. Anchoring

Consider two different wordings for a particular question:

Wording 1: Knowing that the population of the U.S. is 316 million, what is the population of Canada?

Wording 2: Knowing that the population of Australia is 23 million, what is the population of Canada?

This survey was conducted in Stat 100 classes where both wordings of the question were randomly distributed.  The students did not know that there were two versions of this question so each only answered the question that they received. The results for this survey are found in Figure 2.4.

The stacked dotplots show the answers to two questions about Canada's population in the survey. Most data points for the wording #1 are located at the right part of the graph, and most points for the wording #2 at the left part.

Figure 2.4. STAT 100 Survey Results

As you can see, the students were influenced by the wording of the question that they were asked to answer. People's perceptions can be severely distorted when they are provided with a reference point or an anchor. People tend to say close to the anchor because of either having limited knowledge about the topic or being distracted by the anchor. You should also consider the following three points:

  • The sample sizes were large enough to detect a difference in the two groups (recall the point made in Chapter 1)
  • Canada's population is about 35 million
  • The anchor might be less distracting if the following wording were used: "What is the population of Canada, when knowing that the population of the U.S. is 316 million?"

Example 2.8. Unintentional Bias

Consider two different wordings for a particular question:

Wording 1: Do you favor or oppose an ordinance that forbids surveillance cameras to be placed on Beaver Avenue?

Wording 2: Do you favor or oppose an ordinance that does not allow surveillance cameras to be placed on Beaver Avenue?

People will tend to answer "oppose" or "no" to a question that contains words such as forbid, control, ban, outlaw, and restrain regardless of what question is actually being asked. People do not like to be told that they can't do something. So the responses to the two questions would not provide similar results. Wording 2 would be preferred over Wording 1.

Example 2.9. Unnecessary Complexity ("Double-Barreled" Problem)

Consider the following question.

Question: Do you think that health care workers and military personnel should be the first to receive the smallpox vaccination?

The problem with this question is that the respondent must consider both health care workers and military personnel at the same time. The following rewording is much better.

Revised Question: Who should first receive the smallpox vaccination?

a. health care workers
b. military personnel
c. both Health care workers and Military Personnel
d. other

Example 2.10. Asking the Uninformed and Unnecessary Complexity (Double Negative Problem and List Problem)

Consider the following question.

Question: Do you agree or disagree that children who have a Body Mass Index (BMI) at or above the 95th percentile should not be allowed to spend a lot of time watching television, playing computer games, and listening to music?

The first concern with this question is that many people may not clearly understand what the Body Mass Index (BMI) represents. BMI is a measure that is used to identify obesity and is calculated by dividing a person's weight (in kilograms) by the square of their height (in meters). (Note: many Web sites have BMI calculators.) In children and adolescents, obesity is defined as a BMI for age and gender at or above the 95th percentile. This definition should be included prior to the listing of the question on a survey.

This question can also cause problems because of a possible "double negative". Specifically, the problem is with the "disagree" choice. This choice produces a double negative because "disagree" and "should not" are both in the statement. Many respondents will not understand what they are really saying. (It is easy to make the mistake of the double negative).

Revised Question-First Revision: Do you agree or disagree that children who have a Body Mass Index (BMI) at or above the 95th percentile should spend less time watching television, playing computer games, and listening to music?

As you examine this revised question you should also note that there still is a list of three choices embedded in the questions. Since respondents sometimes can get hung upon the list of choices; the second revision would be preferred.

Revised Question-Second Revision: Do you agree or disagree that children who have a Body Mass Index (BMI) at or above the 95th percentile should spend less time in sedentary activities?

A follow-up question could be asked to clarify which sedentary activities should be reduced.