ICTworks

Despite the growing recognition that quality, timely, and accessible data are essential to every country’s ability to deliver vaccines effectively to its population, few data use interventions have been rigorously studied or evaluated.

There is limited evidence of how data can be effectively used to support data-driven action and decision-making. We found more evidence on the intermediate outcomes of data use interventions on data quality, availability, analysis, synthesis, interpretation, and review.

The information and evidence we collected in A Realist Review of What Works to Improve Data Use for Immunization permitted us to develop stronger evidence-informed theories about what works to improve the quality and use of data, for whom, and under what circumstances. We reached the following conclusions.

Multicomponent interventions were more effective

Nearly all the interventions we reviewed leveraged more than one data use strategy. The more comprehensive the set of strategies, and the more they addressed barriers at various stages of data use (e.g., data availability, data quality, and data use skills) and touched upon multiple mechanisms driving data use behaviors and actions, the more likely they were to achieve results.

By addressing different facilitators of data use, the multicomponent interventions employed interconnected, mutually reinforcing strategies that appeared to have a greater collective effect than a single intervention.

Notably, successful intervention packages included strategies that addressed:

• skill sets and capacity of data users;
• gaps in feedback mechanisms;
• data use within existing systems, workflows, and workloads;
• user-centered design principles;
• interaction between data producers and data users, and structured problem-solving;
• data use culture and motivation to use data; and
• long-term commitment of financial and human resources.

Health systems approach to data use were more sustainable

Interventions were more successful over the long term when they focused on systematizing data use at all levels of the health system and as part of decision-making processes. This occurred by:

• Routinely conducting data review meetings at all levels,
• Making national guidelines and protocols on data use available to frontline staff,
• Creating dedicated staff positions at all levels of the health system to oversee data management and use activities,
• Incorporating training in data use in national in-service and pre-service training curricula.

Limited effectiveness of HMIS on data use

Transitioning from paper to computerized health management information systems across all levels of the health system seems to have made higher-quality data more available to decision-makers and may have contributed to better data use at the district level when complemented by activities that reinforce data use.

The effect on data use at the facility level, however, remains less conclusive. In many countries, the significant operational challenges, extended time required for a return on investment, and absence of complementary data use activities have contributed to the mixed results presented in the research literature.

Full transition to computerized systems may be more successful when they are incrementally phased in only once a reliable foundation of data use infrastructure, human resource capacity, and skill base has been established.

LMIS make higher-quality data more available

Computerized logistics management information systems that were implemented at district levels and higher seem to have had more success than similar efforts to digitize routine service-delivery data at a facility level.

There were often fewer operational challenges when they were implemented at district and higher levels, where Internet connectivity, electricity, and information technology support were more reliable.

In addition, we hypothesize that data users may have greater knowledge of how to use supply chain data to take action directly compared with routine service delivery data, which are more commonly collected for reporting by frontline health workers who feel little connection to or agency over the data.

Although implementing computerized LMISs as a single intervention improves data quality and use, there were even greater gains in data use and supply chain performance when LMISs was complemented by other data use activities

Dynamic relationship between data quality and data use

Although poor data quality was an important barrier to data use, we found limited evidence that single-component interventions to improve data quality led to improvements in data use.

Conversely, we found stronger evidence that data quality improved through the use of data. As decision-makers started using their data more and identifying inconsistencies with data quality, they took more corrective actions to improve data quality.

This is an excerpt from A Realist Review of What Works to Improve Data Use for Immunization