The Numbers Aren’t Right

Research shows that 25% of Southern Cross consumers are wondering why the newspaper does not conduct a reader survey. Admittedly, the research base is low. One person raised that question, and I asked three others if they cared one way or another. They didn’t.

Of course, reader surveys can provide useful clues about those who respond: a rough demographic profile, their reading preferences, and so on. The Southern Cross last conducted one in late 2000. It was a very helpful exercise, inspiring innovations which I hope have served to improve the newspaper.

The trick with surveys is to be aware of the diverse elements which might skew the statistics they reveal. As such, the results of a survey can never be truly accurate because the variables tend to unmeasurable, unknowable and unstable. Indeed, anecdotal feedback, in letters and conversations, can give an editor a far better indication as to what works, and what doesn’t.

Some years ago I researched a now defunct weekly newspaper called South, which had been founded in 1987 as an anti-apartheid organ to mobilise the grassroots struggle in Cape Town, especially in the Coloured community. It was influential, and it produced some fine journalists. By the early ’90s, however, it had outgrown its purpose. Apartheid was inexorably vanishing, and South struggled to find its post-struggle feet. Looking for a new direction, it ran a readers’ survey. The results of that showed that its readers mostly were not Coloured after all, but white academics who wanted South to resemble the more sophisticated Weekly Mail. So South revamped itself in a bid to cater to those needs. Soon the paper folded.

What those who analysed the survey’s data failed to acknowledge, among other things, was that academics are more motivated to fill out a survey (research being the scholar’s lifeblood), and more likely to mail it through the faculty secretary. Those who used to be South’s target market  activists, teachers, nurses, students, parents  might well have filled out the survey, but probably never got around to posting because they are busy and have no faculty secretary. The results were misleading, and that should have been easy to spot. Total trust in numbers can be lethal.

Statistics can be entirely meaningless. Take the claims in toothpaste ads. ‘Nine out of ten dentists recommend Sparkledent’, the commercial might boast. Perhaps that is even true; but what were these dentists asked? ‘Doctor, given the choice of Sparkledent and all its competitors, would you recommend Sparkledent?’ Here, 90% is such a massive vote of confidence that I would cheerfully eat that toothpaste for lunch.

‘Doctor, do you recommend that people should brush their teeth using a toothpaste such as Sparkledent?’ Here 10% of dentists would prefer that you used another brand, which inspires less confidence in the product. ‘Doctor, do you recommend that people brush their teeth using Sparkledent instead of a blend of sugar, septic acid and wet cement?’ Here 10% – oh, you know where I’m going with that.

It is all about methodology and context. Which makes M-Net’s game show Power of 10 such a ridiculous contest of chance, more arbitrary even than bingo. Are there really teams of researchers trawling through every South African demographic to establish what percentage can burp the national anthem or how many people paint their spinach red because they don’t like green vegetables? If I was a contestant, I might well react to a ‘wrong’ answer by demanding a comprehensive report detailing research sample, methodology, and error margins. I would insist that every social group of South Africans be proportionally represented. Not just because I’m a bad loser, but because unreliable statistics stink up the laboratories of social studies.

‘I’m sorry, Mark,’ I might say, ‘I cannot accept your answer as scientifically valid. You claim that only 38% of South Africans believe that ‘the rings of Saturn are celestial disco lights’. But your methodology excludes samples of the population groups that have a cultural tradition which holds that the nocturnal sky represents the College of Saints getting groovy to Kool and the Gang. I can therefore not accept your answer.’

And exactly 87,2% of viewers would agree with me.