I am working with Vision Critical University at the moment to create a set of Research 101 tools and I wanted to share my thoughts on the rules for writing survey questions. These rules do not relate to specific questions types (such as a Liker item), or to specific common mistakes (such as the leading question) but at the higher design level.
I would suggest that there are four key rules for writing the wording of surveys questions, three relating to the respondent and one to the researcher. In addition, the survey questions must collectively answer the research and business questions that the study was commissioned to deal with, in a fair and non-biased way.
The four rules are:
2. The respondent must be willing to answer. Not everybody is willing to say how much they earn; even fewer are willing to say if they have recently cheated on their partner. Making it clear that the data is anonymous and indicating why the question is being asked will normally increase the number of people who are willing to answer a question. Adding a 'Prefer not to answer' can help avoid some respondents leaving the survey when they reach a question they are unwilling to answer.
3.The respondent must be able to answer the question. There are several different reasons why a respondent might not be able to answer a question, and all of these reasons should be kept in mind when designing questions. One category is when the respondent knows they don't know. For example, "What is the speed of the internet connection to your home?" is a question that many people do not know the answer to. Some questions are inconvenient to answer, for example the question "How many tins of food do you have in your home?" would normally require that people be at home whilst they do the survey and be willing and able to go to the relevant room(s) to find out how many tins they had. A key category is where the respondent does not know their own motivations, for example "How important is taste versus convenience versus price to you when buying tinned fruit?" - this sort of question usually requires an indirect method of questioning such as conjoint analysis or regression to estimate the respondents motivational structures.
4.The researcher must be able to interpret the meaning behind the answer. For example, the question "Was the train clean and on time?" is fairly easy for the respondent to answer. If the answer is Yes, the researcher knows what is being conveyed. However, if the respondent answers with a No, their train might have been late, or dirty, or both.
1. The respondent must be able to understand what the question is asking.This means using language that is clear and intuitive. It means asking questions that are sufficiently specific. For example, don't ask "How much do you earn?, be specific, "What is you annual income before taxes and stoppages?"
What are your thoughts about these four rules? Are they necessary and sufficient? Can you reduce them or do they need expanding?