Research

A lesson in avoiding questionnaire bias

One of the thorniest components of effective questionnaire design is determining when, and how, to introduce information to respondents. These decisions need to be made very carefully, striking a balance between simplicity and detachment.

In a recent nationwide survey in the United States, we needed to find out four different factors about an incident: whether people were aware of it, whether race played a role in it, whether charges should be sought in connection with it, and whether the law that was apparently used as a justification for it was supported by Americans.

Here is how we decided to ask these questions:

Q1. As you may know, the Florida Legislature passed a law in 2005, which allows residents to use deadly force away from their homes if they have ‰”reasonable fear‰” an assailant could seriously harm them or someone else. This law asserts that people have the right to ‰”stand their ground‰” and ‰”meet force with force‰” when attacked as long as they are in a place they have a right to be, are not engaged in unlawful activity and believe that their life and safety was in danger.

All things considered, do you support or oppose this law?

[SINGLE CHOICE]

Strongly support
Moderately support
Moderately oppose
Strongly oppose
Not sure

The question about ‰”stand your ground‰” legislation was the first one in this block. If the question about the ‰”stand your ground‰” legislation had appeared after respondents were informed about the shooting, it may have led them down a path of opposition‰ – as they have just been reminded of an unfortunate incident that may be directly connected with a law they should be judging on its own merits. The notion here is to take the pulse of respondents solely on the basis of what the law entails, and not to introduce any bias through a specific incident.

Q2. Over the past month, how closely have you followed stories related to Trayvon Martin?

[SINGLE CHOICE]
Very closely
Moderately closely
Not too closely
Not closely at all

The question about whether people are following the case was the second one. We were careful not to ask about ‰”the shooting in Florida‰” or to describe the events at this stage. The idea was to gauge the level of attention that Americans were playing by relying solely on the name of the victim. People may have seen, heard or read about ‰”a shooting in Florida‰”, and that would conceivably provide a superficial uptick.

Q3. It has been reported in the media that George Zimmerman, who was patrolling his neighborhood, saw Trayvon Martin walking inside a gated community. Zimmerman called 911 and reported what he described as a ‰”suspicious‰” person. Zimmerman shot Martin, who was unarmed, and claimed self-defense. Police in Florida did not arrest Zimmerman, but the U.S. Department of Justice is now investigating the case. All things considered, which one of the following statements comes closest to your own point of view?

[SINGLE CHOICE]
Criminal charges should be filed against Zimmerman‰ – the justice system needs to become involved
Criminal charges should not be filed against Zimmerman‰ – this was a tragic accident but not a crime
Not sure

The question about charges explains what had transpired in the media at the time the survey was deployed. It offers a description of events, and then asks respondents about the possibility of criminal charges being filed. The intention was to rely on the media reports for a brief description of events, in the same way most Americans originally learned about them. A second sentence outlines verifiable content: the fact that no charges were filed against the alleged shooter, and the U.S. Department of Justice‰’s decision to investigate the case.

Q4. From what you have seen, read or heard about the Trayvon Martin case, do you think race played a factor in the way the events unfolded?

[SINGLE CHOICE]
Definitely played a role
Probably played a role
Probably did not play a role
Definitely did not play a role
Not sure

The final question seeks to understand whether respondents think race played a role in the way the events unfolded. The ethnicity of the two main participants is not disclosed, so as not to predispose the respondent to choose one answer or another. If the respondent is aware of the story (only one-in-four had previously said that they had not followed it closely at all), then he or she will be qualified to make that call based on the information at their disposal.



Subscribe to the Vision Critical blog

Get free customer intelligence tips and resources delivered weekly to your inbox.

By completing this form you consent to receive emails from Vision Critical. You can unsubscribe at any time. Learn more in our privacy policy.