ICTworks

With the enormous growth of mobile connectivity in developing countries and the proliferation of apps enabling easy collection of data via mobile technologies, governments and development practitioners have been racing to complement face-to-face surveys with mobile data collection using phone interviews, SMS text messages, and Interactive Voice Response (IVR).

Remote surveys using mobile technologies promise more cost-effective data collection, enabling increased frequency of data collection from more people and in remote and insecure locations.

Mobile technologies have been successfully used to collect survey data for diverse applications including:

• household economic welfare in Malawi
• maize farming in Cameroon
• monitoring food security in a conflict zone in DRC

Despite these many examples, there’s relatively little systematic evidence about how the mode of data collection might affect the resulting data and the conclusions we can draw from those data.

Within this context, we tested whether different modes of data collection (traditional face-to-face interviews (F2F) and computer assisted telephone interviews (CATI) over mobile phones) resulted in different estimates of dietary diversity and quality among women and young children in rural Kenya.

We evaluated the mode using two widely used nutrition indicators:

• Minimum Dietary Diversity-Women (MDD-W)
• Minimum Acceptable Diet (MAD) for infants and young children.

While both indicators capture diet quality, MDD-W asks adult women about what they ate the day before the survey and MAD asks mothers what they fed their children the previous day.

We asked nearly 2,000 Kenyan women across two counties (Kitui and Baringo) the same questions twice (Round 1 and Round 2), once using CATI and once in a F2F survey, randomizing which survey mode they got first.

Here are four key lessons we learned from our study:

Lesson 1: Similarity of CATI and F2F depends on the indicator

When women were surveyed about their diet, we found no difference in their scored dietary diversity, or the estimated prevalence of MDD-W between the two modes.

However, when women were asked about the diet of infants and young children in their care, the results from phone interviews and face-to-face interviews were different. Women reported that their infants ate on average one more meal and one more food group per day over the phone compared to in-person interviews.

Because of this change, our estimates of the prevalence of infants consuming an adequate nutritious diet (MAD) in Kenya was 17% higher via CATI than F2F.

But why should the mode affect women and young children’s diet data so differently? Collecting survey data is a social process and changing the way people interact in a survey can change the answers you get.

Different survey modes change the comfort level of participants, potentially resulting in different results. Particularly for socially-sensitive questions (things that might be embarrassing like sexual health), using the mode that makes participants more comfortable may result in more honest responses.

Lesson 2: Mode can create a sampling bias

Although the rate of mobile phone ownership and access among women in Kenya is high (up to 80% in our study area), we found statistically significant differences between women who reported having access to a mobile phone and those who did not. Women without access to mobile phones were younger, less wealthy, had completed less education, and were more likely to live in households without a formally employed household head than women who did have mobile phones.

Despite these differences, we found no difference in dietary diversity between women with and without access to mobile phones.

Thus, in deploying remote surveys, it’s crucial to understand the extent of technology gap within the population of interest and incorporate methodologies to mitigate potential sampling biases.

Lesson 3: Enumerator gender matters

When men and women collect survey data, they sometimes report different results due to social sensitivities or biases. We also know that there is a strong gender gap in technology and that men and women may interact with technology in different ways. In a survey setting, these two factors may interact in unexpected ways.

We found that male enumerators reported poorer dietary diversity scores on average than female enumerators. However, male and female enumerators also differed in how much their scores differed between the CATI and F2F modes. By the second round of the survey, male enumerators were scoring infants’ diets the same regardless of the mode, while female enumerators were still tending to give higher scores via CATI.

This observed interaction between gender and mode suggests a need for more research to better understand how these dynamics impact survey findings.

Lesson 4: Remote data collection is safe and cost-efficient

Compared to traditional F2F surveys, CATI was much more cost-efficient for collecting nutrition data. While each successful F2F survey costed about US$16 (including enumerator salaries, transportation and lodging), each successful CATI survey costed just US$5 (including enumerator salaries and call center operation). In one of the locations, CATI also enabled us to collect data from participants despite an outbreak of insecurity, which prevented field teams from conducting scheduled F2F surveys.

Data a la mode

Mobile data collection technologies, including CATI, SMS, IVR and other modes, are rapidly becoming the modes du jour to collect data remotely. Perhaps rightfully so as they can enable rapid, cost-efficient data collection of critical information for humanitarian and development operations. However, practitioners must be aware that the way they collect their data may affect the answers they get; particularly when data may be socially sensitive.

By: Christine Lamanna (ICRAF), Kusum Hachhethu (WFP), Sabrina Chesterman (ICRAF), Suneetha Kadiyala (LSHTM), and Todd Rosenstock (ICRAF).

This work is a collaboration between the World Agroforestry Centre, United Nations World Food Programme Mobile Vulnerability, Analysis, and Mapping, and the London School of Hygiene and Tropical Medicine. Support for this work was provided by IMMANA, USAID, and CCAFS.