Baroness (Shirley) Williams was unequivocal. Sitting in the Newsnight hot-seat, she explained to presenter Jeremy Paxman that a coalition government was what people had voted for. How did she know? There was no box on anyone's voting paper marked 'Coalition'.
What Williams was offering, although she didn't put it this way, was an interpretation of quantitative research data - in this case, general election votes. In her opinion, although she never used that word, the raw accumulations of 36% Conservative, 29% Labour and 23% Liberal indicated that the electorate as a whole had 'wanted' some kind of melange.
This makes as much sense as observing the crowd at a local-derby football match, noting that it divides into two halves, each as passionate as the other in its die-hard support, and drawing the conclusion that the stadium as a whole 'wants' a draw.
Williams' take on 650 local derbies is emblematic of two great sins to which researchers debriefing quantitative and qualitative market research are prey: dubious interpretation and godlike certainty of pronouncement.
Academics who specialise in consumer research have a great deal to say on these sins, usually in tones of haughty disdain for the willingness of marketers to draw too much significance from too little data. In the academic view, all interpretation is biased, no matter how impeccably neutral the researcher might seem, and it is better to acknowledge that upfront. Nor is there any such thing, they insist, as research-derived truth, so all conclusions should be framed equivocally.
In the academic world, the level of evidence required to propose even the slenderest advance in theoretical knowledge is extraordinarily high.
Well that's fine for them, you might think: they don't have real-world marketing decisions to make and a cold-hearted board to convince. In the end, though, that is all the more reason to exercise greater caution in both the commissioning and the debriefing of market research. The stakes are high. If there is one area of commercial practice in which it is worth striving to achieve the academic ideal, this is it. So what can you do to ensure that you guard against those 'Shirley Williams' sins?
First, you can interweave different methodologies to shed light on the same question, so that emerging themes can be cross-corroborated. Academics call this 'triangulation'. It will cost more money and take more time, but avoid a great deal of risk in the end, because the burden placed on flawed human interpretation is dramatically reduced.
Second, when using qualitative methods, you should insist on a word-by-word transcription, explicit coding of themes and slow, methodical sifting. This should be standard practice, but in reality is rare. More common is the junior researcher taking notes at the back of the groups and filtering their impressions on the hoof. Be warned: this is a disastrous way to get to anything approximating the truth.
Third, marketers could try treating researchers with a bit less reverence. Certainly, it is worth pushing back on anyone, including those in your own team, who claims to 'know from research' what consumers want. Probe harder; be tougher. Ask them to show you how they can be so sure, and ask again if you feel the underlying data is open to other interpretations.
Scepticism won't make you popular, but it will help you uncover false assertions. Just ask Paxman.
- Helen Edwards has a PhD in marketing, an MBA from London Business School and is a partner at Passionbrand, where she works with some of the world's biggest brands
30 SECONDS ON ... RESEARCH WISDOM
- 'Advertising research is one-half frustration, one-half exclamation point and one-half question-mark. If this adds up to more than 100%, it proves that mathematics and research sometimes give confusing results.' - Michael P Ryan, Allied Chemical Corp
- 'If we tested innovations every time with a posse of consumers, we would lose the initiative. We prefer to back our own judgement.' - Tim Clark, president, Emirates
- 'The way to do research is to attack the facts at the point of greatest astonishment.' - Celia Green, philosopher and psychologist
- 'First get your facts; then you can distort them at your leisure.' - Mark Twain
- 'I have spent too long being able to manipulate the answers I want from research to rely upon its findings any more than I do weather forecasts.' - Charles Saatchi
- 'Interpretations are always incomplete. One never achieves the understanding; one achieves an understanding.' - Professor Norman Denzin.