How You Ask Questions Matters to Hispanics… and We’ve Got The Proof!
As we have been raising our children, one of the lessons that my wife tries to impart to the kids is, “It’s not just what you say… but how you say it.”
That’s pretty good advice… on the playground, in the classroom and as it turns out, in the world of market research, too.
This past summer, during one of our Hispanic omnibus studies on what’s important to Hispanics in retail establishments, we decided to test different types of questioning methodologies to see if how questions were asked really do make a difference in the results. Guess what? It does.
First a list of the different scale types we were interested in measuring and a little glossary of terms:
Likert scales. Are a structured response format often used to measure agreement with a statement or concept. Respondents are shown an evenly distributed response array with a neutral response in the center and an equal number of negative and positive choices on either side of the neutral center with the far ends being the most negative or positive response. E.g. “On a scale of 1 to 5, with 5 being ‘very happy’ and 1 being ‘very unhappy’ please rate…” This is the most common question structure for surveys.
Branched Likert Scales. Are similar to Likert scales in that they measure levels of agreement or interest, but instead of showing respondents all of the possible responses at once, they present the respondent with a truncated version of the scale with only three options: negative, neutral and positive. Respondents who chose the end responses are asked a follow-up question asking them to indicate their level of agreement, usually by asking respondents to choose between “somewhat” or “very much” options.
Ranked Preference. Unlike the Likert scales above which ask for each statement in the survey to be ‘rated,’ this methodology asks for multiple statements to be ‘ranked,’ relative to each other. E.g. “Of the following five retail store attributes, rank them in order of ‘most important’ to ‘least important’…” Since Hispanics often display a positivity bias, it’s important to have them choose between competing concepts in order to gain a better understanding of their feelings through ranking since all concepts can’t be chosen as #1.
Back to the study…
Hispanic Research Scales Study
To better understand how question design influences ratings of importance, consumers were asked a short series of questions about what is important to them when shopping in a retail store. The five factors in the question we asked were:
- Customer service
- Low prices
- Quality items
- Store cleanliness
- Wide selection
First, the sample was divided randomly into two groups.
- Group A was asked to rate five factors on a traditional five point Likert scale ranging from “not at all important” to “very important.”
- Group B was asked to rate the same factors using a branched Likert scale.
Respondents were then asked Ranked Preference questions on the same five factors, ranking them from “most important” to “least important.” - Finally, a group of non-Hispanics was used as a ‘control group’ and put through the same surveying process as the Hispanic group.
Note, that while the content of their responses is interesting and enlightening, the real focus of our study was to see how scale structure affected the responses.
Assuming that ranked results are the most accurate in teasing out the relationship of concepts to one another – we were interested in determining whether a standard Likert or Branched Likert scale would yield results most similar to the ranking results.
The result? Unequivocally Branching Likert scales yields results most similar to rankers. In fact, for Hispanics and the non-Hispanic control group alike, the relative ranking of all five attributes was consistent across the Branched Likert and Ranked Preference methodologies. Straight, unbranched Likert scales yielded different results for both Hispanics and non-Hispanic groups but appear to help Hispanics more than non-Hispanics achieve consistency in their responses.
So – Should you always branch your Likert scales? As with everything else in business (and in life)… it depends. Branching takes time and therefore, fewer questions can be asked of a respondent within a limited survey window using this method. Often times, straight Likert scales are sufficient to the task if measured concepts are not competing against each other. The best advice we can give is to talk to your research agency team, ask them about the different scales available to you and get their input on which is best for your particular study.
So, “It’s not just what you say… but how you say it.” As usual… it looks like my wife was right!
We are pleased to offer free of charge the full contents of the report referenced in this blog entry. Please click the link below to get your copy now.