Look beyond numbers: we need to know why change happens

Sonia Whitehead, Head of Research Programmes at BBC Media Action gave an insightful presentation at an ESOMAR Foundation session recently held in Amsterdam during the ESOMAR Congress. The session aimed at debating some ways forward to help tackle societal issues.

Working in the development sector I am aware that, particularly over the last few years, donors and others expect project results to be quantifiable. Numbers talk.

This was apparent at the UN World Data Forum in Cape Town in January when we looked at how the SDGs (Sustainable Development Goals) will be measured, and it was discussed again at the Esomar World Research Congress in Amsterdam this week. Donors want statistics to demonstrate impact and show a project is value for money, but aren’t as confident using insights from qualitative research.

In my team we crunch a lot of numbers. In 2016, for example, we surveyed more than 20,000 people, which yielded strong results. For example, a statistical analysis of governance data across seven countries (which controlled for confounders such as people’s age, education levels and existing interests) shows that people who listened to or watched our programmes know more about politics and get more involved in civic life. In our health work, we used quantitative data to understand how change happens.

But we have increasingly come to appreciate the need for qualitative research to contextualise our data and explain why we are having an impact. We’re not alone.

At the Esomar conference I took part in a panel with colleagues from My Choices Foundation, StreetInvest, and Save the Children to showcase how qualitative research can provide greater value to the development sector. I summarised what we had learned from our recent projects.

Story behind the numbers

Qualitative research helps us to explain the story behind quantitative data.

Our radio health shows in Ethiopia, Jember (Maternal Light) and Biiftuu Jireenya (Dawn of Life) aim to help women have safer pregnancies and deliveries, and improve the health of newborn babies. Our surveys showed they were reaching a huge audience – around 21 million people across three states – more than a third of the adult population.

However, analysis of our data showed it was difficult to isolate the impact of our programmes. The health sector in Ethiopia was changing fast the government and donor investment leading to local health workers being trained across the country. This investment was reaping benefits; data from the Demographic and Health Survey showed the percentage of women going for antenatal care had jumped from 27 % to 62 % between 2011 and 2016. But what impact were we having?

To understand, we designed qualitative research, whereby we spoke to women and those they felt influenced their decisions (such as their local health extension worker, their husbands and close family members).

Women told us how health services and social norms had changed since they had given birth to their older children. Things that had been unthinkable, such as giving birth at a health facility, were now possible. They were also able to identify storylines and items in the shows that made them think differently about what was “normal” during pregnancy and birth. The shows were reinforcing the advice that health workers were giving – and health workers also felt women trusted them more because they were saying the same information that they had heard on the radio.

Reaching vulnerable people

A limitation of quantitative research is that it’s hard to reach vulnerable groups such as refugees or survivors of gender-based violence. They are less likely to agree to a face-to-face survey or might not have a fixed address. In this situation, it helps to use qualitative research techniques that put respondents at ease.

Last year, we spoke to Syrian refugees travelling from Greece to Germany to understand how humanitarian agencies could best communicate with them. We employed Arabic-speaking researchers and trained them to use a narrative approach, where they sat with a refugee for at least an hour, hearing about their journey and the part communication had played in it. Being listened to, rather than asked a series of fixed questions, helped the respondents feel comfortable and more likely to share information.

Qualitative research can also help us understand sensitive issues. In Syria, we assessed our radio drama Hay el Matar. To understand issues around the conflict we asked a trusted, local NGO to use projective techniques. This involved asking respondents to discuss the feelings of others – either people in their community or an “extremist” character in the drama. People felt more comfortable talking about how others view violence, or explaining the motives of a fictional character, than talking about themselves.

We discovered that Syrians interpreted people’s accents as a strong signal for which side of the conflict they were on. We relayed this to our radio producers who took greater care with the accents of the actors. We could not have grasped these nuances with quantitative data alone.

Insights on a budget

There’s another cold, hard fact to take into account. Conducting quantitative research, on the scale needed to measure media interventions, is expensive. Project budgets often don’t stretch to this so we need to gain qualitative insights in the absence of large-scale, representative surveys.

At the end of the panel discussion in Amsterdam this week, the Esomar Foundation’s Phyllis MacFarlane asked a crucial question: “What does the development sector need to do for qualitative research to be taken more seriously? Steve Kretschmer from Surgo Foundation summed it up nicely. Quantitative data needs to be used to explain the “what” he said, but when it comes to the “why”, well-designed qualitative research is invaluable. He added that the development sector could work with the commercial sector to adopt practices such as using video to capture emotions (rather than relying on what people say) to make the most of audience insight and – most importantly – to make a difference.