ICTworks

As DFID aims to harness the Data Revolution, ensuring that data – both quantitative and qualitative information – drive decision-making, public accountability, and the achievement of the Sustainable Development Goals (SDGs), ensuring that systems, processes, and skills for data are aligned with these objectives is paramount.

Across sector policy teams, country offices, and various analytical and technical cadres, different strengths and weaknesses, as well as needs and ambitions exist. To inform a strategic approach to data, as framed in its forthcoming Data Roadmap, DFID collaborated with Development Gateway to perform a Decision and Data Use Landscaping study.

This report details lessons learned from approximately 60 interviews across 4 DFID country offices, all sector policy teams, senior managers, and various analytical and technical cadres and offices.

Key Decision-Making Processes

Each interviewee was asked to list 2-3 of the most important decisions that they take (for roles with direct decision- making authority) or inform (for those with advisory or technical roles), and then to describe the role of data in these decision processes.

Through these interviews, clear patterns emerged, including that data use is highest during design (e.g. Business Case and Portfolio Strategy Refresh) processes, before declining throughout the programme/portfolio implementation and monitoring cycle. The table below provides an overview of data use across these processes.

Use of Internal and External Data

DFID staff report higher levels of satisfaction with their use of internal data (typically rated 7/10) compared to external data (typically rated 4/10). Internal data use centres on spend and high-level results (e.g. portfolio quality scores – PQIs – or headline indicators), together with overall programme risk ratings. Users expressed enthusiasm for improvements to the Aid Management Platform (AMP) and management information tools in recent years, and a desire for additional training to understand which tools to use for which purposes.

Ambition for increased use of Value for Money, results frameworks, and disaggregated risk profiles at both programme and portfolio level were frequently voiced. However, DFID staff reported challenges in using Value for Money data. As DFID increasingly works in Fragile and Conflict-Affected States, the ability of DFID staff to verify implementing partner results data is limited, requiring continued innovation in the use of beneficiary feedback and third-party monitoring.

External data use is largely restricted to the design and planning stages, and centres on trusted international data sources, particularly in sector policy teams. In country offices, official statistics and government administrative data are also used, although the sourcing and quality of these data present significant challenges to stats advisors and programme staff. Data on the activities of other funders is also frequently sought, typically through OECD channels for benchmarking of DFID funding levels.

However, interest in having more real-time information on which programmes are taking place in a given sector/country suggest potential for increased use of IATI data.

DFID Staff Data for Decision Making Needs

Specific data for decision making needs voiced by DFID staff in this study centred on:

• Data trainings, with an emphasis on understanding how and where to find data, and to assess the quality of external data.
• Increased communication around MI, as users struggle to stay abreast of the latest tools and features.
• Creation of an internal data dictionary to ensure a common understanding of data fields
• Creation of an external data catalog to crowdsource information on data availability, coverage, and quality.
• Tools to analyse performance at a portfolio level (see below), and to easily identify key lessons learned from annual review processes.
• Automation of routine analysis in order to free time of analytical cadres to support advanced analysis, which is frequently outsourced due to overwhelmed internal resources

Senior Management Data Preferences

While Senior Managers are often interested in data and what it tells them, most do not want to be presented with raw data when making decisions. Instead, they look for synthesised analysis described by one DFID advisor as “tweets with links” – succinct summaries of major takeaways from programme implementation, evaluation, external research or other sources, backed with appropriate references and caveats. Senior Managers can then dig deeper and assess the reliability of the evidence and data presented to them, if needed.

Visualizations that clearly illustrate the larger data narrative are much more helpful to SMs than spreadsheets of raw data, although many Senior Managers suggest that “league tables” displaying relative rankings (e.g. of programmes, or of DFID spend vs other agencies, etc.) are valuable and easily interpretable. By the time the information reaches the Senior Managers’ desk, data have been largely processed and generalised, and most prefer it that way. As one Senior Manager put it, they want their advisors to “tell me the key takeaway, don’t ask me to interpret raw data.”

This is not only because they are pressed for time, but also because they often need to be able to communicate their policy positions to audiences that are not necessarily concerned with such details. One drawback from this process is that analysts repeatedly mentioned that Senior Managers have limited knowledge of available data sources and are challenged to interpret statistics independently. Analysts feel that this sometimes limits their ability to communicate their analyses with Senior Managers effectively.

Data Needs for Portfolio Analysis

DFID staff at all levels voiced a need for more strategic use of data to monitor portfolio-level performance, and to ensure alignment with portfolio strategy. Several specific use cases for portfolio analysis using automated dashboards emerged, including portfolio stock-takes, on-demand information on spending levels and allocation, learning across programmes, strategy formulation, and responding to data requests.

Specific data types needed to meet these uses include:

• Financial data,
• Results information,
• Value for Money,
• Project and Program risk,
• Implementing partner performance
• Constituent needs,
• Other funder activity.

Some of these data (e.g. financial and risk) are readily available, others are being expanded by the AMP and Management Information team but require updated reporting processes (e.g. risk, results, implementing partner), while others require further analysis and investment (Value for Money, external data).

Data for Decision Making Report Conclusion

DFID is appropriately considered as a leader in data for decision-making. However, the Landscaping Study uncovered a number of gaps and areas where deeper investments could create a leap ahead in the use of results data, Value for Money, and external data to develop a deeper understanding of DFID’s portfolio and to drive more effective targeting and implementation. Deepened collaboration between analytical cadres and the MI team can create this leap through the strategic deployment of tools, training, processes, and innovation to match the needs uncovered through this study.

The study was developed by Josh Powell, Sarah Orton-Vipond, Vinisha Bhatia, and Annie Kilroy of Development Gateway