ICTworks

Sensors promise rapid insights into development programs. Exciting and quickly evolving technologies are expanding the range of what can be measured, while the precision, accuracy and frequency of measurement are continually improving as well. An evaluator’s dream!

But what have we learned from our use of sensors to date? What measurement challenges can sensors overcome and how can they be incorporated into mixed methods evaluation? What are the real and perceived opportunities and limitations, the costs and benefits?

One breakout session at the recent MERL Tech conference in Washington, DC tackled these questions head-on. A series of three case studies demonstrated the benefits of sensors in monitoring & evaluation scenarios and described implementation (i.e., required resources, challenges, and data analysis).

Case #1: Temperature sensors to measure stove use behaviors in Ulaanbaatar, Mongolia

Olga Rostapshova from Social Impact presented the application of sensors in an impact evaluation of the MCC Energy-Efficient Stove Subsidy Program in Ulaanbaatar, Mongolia, which aimed to reduce air pollution and decrease fuel expenditures through the subsidized distribution of more fuel-efficient heating stoves. Data on household stove operation is crucial to estimating fuel use, and surveys are typically used to collect this data.

However, self-reported estimates can be imprecise or subject to courtesy or recall bias. To gather unbiased and precise measurements of stove use behavior, small temperature sensors (stove use monitors or SUMs) were placed in a sub-set of surveyed households: 1) on the foot of the stove (to measure for stove temperature), and 2) on the dwelling wall (to measure home temperature).

The sensor data provided important insights into stove use behavior of households with and without the MCC stoves. While the program reduced emissions and air pollution, households with MCC stoves did not reduce their coal consumption. Sensor data helped explain this result.

While the MCC stoves do use fuel more efficiently when properly used, households were often not operating the stoves correctly. The SUMs data showed that people were adding coal to the fire, although MCC stoves must be “cold-started” to unlock fuel savings – an option that is not too attractive during the cold winter months in Mongolia!

The SUMs data on ambient temperature also indicated that homes with MCC stoves were kept warmer than those with traditional stoves, although the fuel used was the same on average. This suggests that households were utilizing fuel efficiency to increase home temperatures, whether knowingly or not. Ultimately, inexpensive temperature sensors were critical for collecting accurate outcome data on and explaining unexpected results.

Case #2: Pressure loggers to measure piped water supply intermittency in urban Tanzania

John Feighery of mWater discussed the use of pressure logging devices to measure intermittent urban water supply in Social Impact’s impact evaluation of the Millennium Challenge Corporation (MCC) Tanzania Water Sector Project. Household meters don’t track supply information and survey data can be imprecise, biased, or “lumpy”, while water pressure loggers can provide accurate measurement of pressure at the tap.

Each device connects to the end of a yard tap and logs pressure continuously (every 10 minutes) for months. Pressure loggers were installed in a sub-set of households that were surveyed. Logger data can be used to analyze service patterns, such as the number and duration of outages, and average hours of water service per day.

The data collection required substantial planning and logistical coordination, as well as flexibility to respond to challenges. Sufficient well-trained staff had to be allocated to the activity to maintain the desired pace of device installation and rotation.

The sampling strategy had to be flexible – an efficient replacement strategy was put in place to avoid delays when circumstances at a selected household were not conducive to installation. Data was manually downloaded from each device and then uploaded to the mWater server, so ensuring Internet connectivity was important (ideally data would be transmitted wirelessly from the device, but the hardware did not offer this functionality).

Installations were trickier that initially envisioned, with more disassembly of taps required compared to the team’s initial expectations, but staff were trained to handle these contingencies. Security of the devices was another concern: lock-boxes were used to minimize risk of theft or damage.

The sensors were rather costly, at about $500 per unit. This type of sensor could also be used for utility network monitoring to reduce non-revenue losses, for output-based aid or performance-based contracting verification, and to help monitor water access and reliability indicators.

Case #3: Motion sensors to measure monitor hand-pump functionality in Rwanda

Dexter Gauntlett of SweetSense showed how the CellPump project in Rwanda used sensors to monitor the functionality of hand-pumps. Better and more cost-effective ways are needed to monitor functionality of water points, which have a notoriously high failure rate in the developing world.

In the featured pilot project in Rwanda, more than 200 censors were installed at water pumps, with data from each sensor transmitted wirelessly through the cellular network to a dashboard for the operations and maintenance team. The dashboard displays the real-time status of each pump equipped with a sensor. This enables operations and maintenance teams to employ an “ambulance” model, dispatching teams only to water points flagged for repair or check-up.

This sensor-driven model increased pump uptime from 50 percent to 91 percent. The mean reported non-functioning days dropped from 214 days to 20 days, nearly three times better than the current “best practice” circuit rider model. Importantly, the reduction in down-time at the water points with sensors means that even including the cost of setting up the sensor monitoring system, the sensor-driven ambulance model could be more cost-effective than other operations and maintenance approaches such as the circuit-rider model.

What can we learn from these cases?

1. Thinking more broadly about sensor use: While each case demonstrated that sensors provided substantial advantages in increase data quality for hard-to-measure indicators, the sensors were still embedded in a traditional data collection approach, with discrete phases of data collection at various points during project implementation.

   Depending on context and information needs of a particular project, there could be substantial benefits to collecting sensor data over longer time horizons. Increased availability of long-term data would enable more rigorous and timely monitoring, and open a range of possibilities for evaluation, such as long-term time-series approaches. (One example in the U.S. is a recent initiative in the Chicago area to measure a broad range of indicators from traffic congestion to air pollution.)

2. Cost-effectiveness and evaluation as intervention: The use of sensors for MERL often blurs the line between implementation and M&E – the sensors themselves can become part of the intervention, and can promote cost-effectiveness, as described in case #3. This suggests that along with considerations about cost-effectiveness, sensors should be included in initial program planning as tools that have the potential to improve program outcomes.

   Allocating sufficient funding for such outcome-monitoring and possibly outcome-improving technology from project inception would increase effectiveness. Ultimately, monitoring with sensors can help stretch program dollars.

3. Fostering local demand: With increased attention on user-centered design thinking in development, it’s important to consider how local stakeholders can make good use of data collected with sensors. Local institutions could benefit enormously from such data (e.g., water utilities from case #1). How can we think about integrating sensors into public or private service delivery at local level? Can we do more to foster local demand and advocate for the upfront investment?
4. The human element: While sensors hold promise for data collection, management, and even processing when automation is possible, the human element continues to be critical – to ensure the quality of data, valid interpretations, and strategic data-informed decision-making. Substantial time investment is necessary to set-up effective systems, analyze data, interpret the trends and patterns, and to use the results to learn and adapt program management effectively.
5. No substitute for good planning: Budgeting the time, resources (money and people), and systems needed to efficiently integrate the sensor data into the larger context of an evaluation or regular program monitoring cannot be understated.

A lively discussion followed the presentation, focusing on how sensors could be used effectively to gain greater understanding of development impact. Questions included:

• What do we not know about sensors? What other sectors are ripe for sensor use? Could we expand the use of sensors to apply them to sectors that are not traditionally targets for this technology (e.g., school attendance)? We must think creatively about leveraging existing technologies and expand the range of what’s possible to apply this kind of insight to other sectors. Examples include infrared sensors to track IDP movements, satellite data to track leaf health, and soil moisture sensors.
• How can we increase the value of sensor data beyond individual projects, for the benefit of other users, programs, donors, etc.? We should consider donor/funder requirements and appropriately guarding private and identifiable data, while striving to make the data more available for others. (Note: data from MCC-funded impact evaluations are posted for free download to the MCC Open Data Catalogue).
• To what degree are we involving and working with local communities to design these activities? Are we thinking about what they need to know, and what technological solution may work best for them? Similarly, how can we ensure that the data collected reaches beneficiaries (either after or during monitoring or evaluation, with information going directly to beneficiaries if feasible)? For example, the benefit in terms of behavior change or other economic outcomes of consumers having real-time data on water point functionality or water quality would likely be substantial.
• When thinking about the “human element” component, how can we make sure to build capacity to analyze and interpret data, and not just provide a technology or piece of equipment?
• Depending on the technology used in the context of expanding sensor technology to output-based aid or pay-for-performance, are we considering the risks of “doctoring” the devices? Is this a risk? How much of a risk? How could it be mitigated?
• How can we make sure that necessary resources are mobilized to address non-functionality identified by the sensors that might otherwise go undiagnosed? While these are “good problems” to be creating, who will bear the cost and responsibility for failures “uncovered” by increasingly rigorous monitoring using sensors? How will the presence of a sensor for monitoring functionality change community attitudes toward maintaining infrastructure (e.g., water point)?
• How might we think about a business model to create demand from local partners and extend the operational scale and deployment of sensors?

The cases featured in this breakout session are just a small selection of the many possible use cases. There are many other examples out there, and we encourage other users to discuss their own experience in the comments.

By Danae Roumis, Social Impact; Olga Rostapshova, Social Impact