A recent NewMR lecture by Jeffrey Henning (@JHenning) focussed on improving representativeness amongst online respondents. It was during the lecture I noticed a quote: “Panellists find surveys worthwhile”.
My first thoughts on this were sceptical, and I tweeted saying so. Little did I know the reaction this tweet was about to cause on Twitter (and how much time that weekend I would spend replying to tweets and mulling over opinions, others’ as well as my own!)
Annie Pettit (@LoveStats) was first to reply, saying that she agreed that panellists found surveys worthwhile and that it wasn’t all about the financial incentives, that these could be viewed as an added bonus. She also added that if you ask respondents why they have taken part in a survey, “wanting to help” is always within the top two answers. Unfortunately I haven’t had the chance to heed her advice yet, but have no doubt that it will be happening soon – I’m too curious to let things go without delving into them first!
I have no doubt that some respondents take part in surveys because they want to help. I too have been in these shoes; when a company I am familiar with invites me to help with some research – I know there is a chance that my answers will affect the future of that company, and I’m keen to get these across.
The key difference between me in this situation and the people that caused me to be sceptical where panels are concerned is the amount of invested interest in the research. I definitely had invested interest in the research I took part in – it was a company that I used often, and I would maybe even benefit from the outcomes of the research. Again, I have absolutely no doubt that panellists often do research which they enjoy, or find interesting – checking off that need for invested interest.
However, I also have no doubt that some panellists are sent upwards of 30 emails a day inviting them to take part in research. I would guess the likelihood of them having a genuine ‘want to help’ with all of these is very slim – especially given that respondents aren’t always told at the start of the survey who the research is for or what it is about. My theory is that the need for financial incentives amongst respondents is negatively correlated to their interest in the research. It is when their interest in the research is lower than their interest in the incentives that we may come across some problems.
A quick Google search for terms like “online research” and “online surveys” really highlights where my concerns lie. Search results throw out phrases like “earn £10/hour” and “make cash in your spare time”. Panels like these are screaming out for panellists who are in it for the money, and not the satisfaction of helping. Perhaps this problem will be that easy to fix – stay away from panels that advertise themselves in this way and you stay away from those problem panellists.
In my opinion, incentive-hungry respondents will always be a problem in market research. They lie their way through screening questionnaires and speed their way through online surveys with their eyes fixed on the prize. This article from New York Magazine shows how much of a reality this is. I am just aware that doing research online for incentives is perhaps easier than focus groups, or telephone surveys. There is an abundance of email invitations and lack of human interaction to ‘catch you out’.
So perhaps my first tweet about that quote was a little rushed, a mild case of think-before-you-tweet. But I’m still not convinced that this isn’t an issue we need to consider.
No, I’m not saying panels are useless. No, I’m not saying we should stop doing online research. All I’m saying is that we should take the responsibility for engaging respondents to increase that interest in the research, and not let the incentives do that for us.
Bethan Turner is a senior research executive at Mustard, and is more than happy to hear any feedback or to continue the debate – on Twitter @MustardBethan.
