Mustard has conducted research in more than 70 countries world-wide in the last few years. We often conduct multi-country studies, where there is a need to compare results by market to evaluate perceptions and assess performance, and ultimately inform client understanding and help them to make better decisions.

When analysing and interpreting data and feedback across markets, we always have to be conscious that results can be heavily influenced by cultural bias. There have been numerous academic papers written which highlight how responses to questions can be affected by people’s beliefs, values, ambitions and level of individualism, amongst other things, all of which are driven by where they have grown up and live.

These cultural differences have been shown to have a big impact on how people use rating scales when completing interviews or surveys. Messick and Frederiksen identified that different countries have different response styles depending on their culture. The three most recognised responses are:

• Extreme Response: This is the tendency for a respondent to select the extreme ratings on a scale. For a 1-10 scale, this generally means 1-4 and 9-10
• Mid-Point Response: This is the tendency for a respondent to select the middle ratings on a scale. For a 1-10 scale, this generally means 5-8
• Acquiescence: This is the inclination to agree with the interviewer or survey sponsor, or to respond in such a way that they think will “please” them

Those who lean towards a strong Extreme Response will typically deliver higher ratings (for example, when evaluating new concepts or propositions giving overall satisfaction or Net Promoter Scores), primarily because positive feedback in surveys usually outweighs negative feedback. Therefore, countries which tend towards an Extreme Response are often seen at the high end of country-ranked results.

Conversely, a Mid-Point Response style usually leads to lower mean scores and indexes, pushing countries more towards the lower end of results ranked by market.

Studies have shown that Latin American markets such as Brazil and Mexico are more likely to adopt Extreme Response behaviour, whereas European markets tend to be more restrained and show Mid-Point Response behaviour.

Knowing that this happens, what can we do to take account of these cultural differences?

Some have argued using different point scales to try and combat the issue; however, no strong conclusions have been drawn from studies in this area. Our preference has been to use a smaller point scale (usually a 5-point scale), with descriptors for each point that are translated for each market so that the respondent is clear on how they are answering when giving a rating.

Others have suggested adjusting for the variations when analysing results. Some marketers, for example, have proposed a European version of Net Promoter Score, where an 8 rating classifies as a ‘promoter’ and a 6 rating as ‘passive’. These adjustments, however, can become complicated, especially where you’re looking at multiple markets and cultures.

Alternatively, there is the potential to analyse ratings given alongside responses to open-ended questions, using the additional comments to contextualise and standardise the results. This has some merits, but is potentially a laborious exercise, particularly for large-scale surveys, and is still open to interpretation – strict guidelines need to be set and followed if attempting this.

Standardisation has also been used as a solution to provide a more robust sample of scores for cross-country comparisons. The primary aim of standardisation is to reduce or eliminate unwanted cross-cultural differences that are not due to variables of interest, but rather response sets and methodology factors. There are a number of standardisation approaches available which make adjustment of the means, adjustment for dispersion (e.g. standard deviation), adjustments using means and dispersions, and using covariate analysis. We won’t bore you with the detail and technicalities of each of these approaches. Again, there are those that argue for and against using standardisation procedures, and no common view has been established on how best to go about evaluating and interpreting cross-country data.

Ultimately you need to appreciate that cultural bias exists, and also be aware of your own biases from your culture which could affect interpretation. If in any doubt you should work with experts in this area, who will have the knowledge of how different markets behave, as well as the experience to analyse and interpret the data.

To conclude, there is no single set of guidelines or solutions that can be applied across all projects. The need for flexibility is key, and sometimes the best option is to focus on benchmarking and tracking results within specific markets.

Colin is Mustard’s MD, and has worked on multi-country studies for leading brands across a range of consumer and b2b sectors for almost 30 years.