ICTworks

I know you are excited to explore consumer-grade tools like ChatGPT in your work. We had a Technology Salon that asked how Generative AI solutions can improve international development. Our discussion focused on the positive and negative aspects of GenAI in development, and where things like Google Translate are already changing the way we operate.

One aspect of that discussion was the current state of generative artificial intelligence solutions freely available to consumers that can be used to search the Internet and create text. Four of the most popular tools are:

1. Google’s Bard
2. OpenAI’s ChatGPT
3. Microsoft’s Bing Chat
4. Perplexity AI Chat

How Do Public GenAI Tools Interpret USAID Projects?

I recently took a turn with each free consumer-facing solution to see how they interpret USAID programs and impact. I asked each GenAI tool a simple question: “List 10 USAID Programs Using Artificial intelligence for Social Impact with Links to Sources.” The last phrase of the request is key for me. I want answers that are based in reality – not artificial intelligence “hallucinations.”

1. Google Bard was useless

When I asked Google Bard about USAID programs that are using artificial intelligence solutions for social impact, with links to the source documents, the response was very disappointing.

Google Bard said, “I can’t assist you with that, as I’m only a language model and don’t have the capacity to understand and respond.” Even when I tried to rephrase the question and removed the requirement for links to sources, it still couldn’t help me.

2. OpenAI ChatGPT made things up

When I asked ChatGPT to list USAID programs with artificial intelligence activities that increased social impact, and links to sources, it started out explaining that as an AI language model, not a search engine, it did not have real-time browsing capabilities.

However, it would try to answer my query as of September 2021, with a list of USAID programs that were utilizing artificial intelligence for social impact at that time. Then ChatGPT went on to list the following three activities, none of which seem to be clear USAID projects. My analysis is in italic.

• Artificial Intelligence for Earthquake Damage Assessment (AI4EQ): This program focuses on using AI to enhance rapid and accurate damage assessment after earthquakes to improve emergency response and recovery efforts. I cannot find any mention of AI4EQ.
• AI for Wildlife Conservation: USAID has supported programs that utilize AI for wildlife conservation in Asia, including projects focused on tracking and protecting endangered species using AI-based monitoring systems. The Wildlife Asia website doesn’t speak to AI for conservation.
• AI for Agricultural Productivity: USAID has partnered with organizations to leverage AI technologies in agriculture, aiming to improve productivity, crop yield prediction, and resource management for smallholder farmers in the Feed the Future program. This reference is too vague to check on the FtF website.

3. Microsoft Bing Chat had messy answers

Bing started strong by explaining that AI has the potential to transform and improve various aspects of development programs, such as health, education, agriculture, environment, and governance. It then listed several impressive-looking references to USAID programs that use AI to achieve social impact. However, the programs were not always what they seemed. My analysis is in italic.

• HealthAlert in South Africa: This program uses AI and ML to provide personalized risk assessments and referrals for HIV testing by using a chatbot app called HealthAlert. The chatbot app uses natural language processing (NLP) and behavioral science to engage users in a conversational way and provide tailored information and guidance based on their responses. HealthAlert existed – for COVID-19 but it was implemented by the South African government without any mention of USAID support that I could find. HealthAlert came out of Praekelt.org, which now operates the WhatsApp-based system under the Turn.io brand.
• PlantVillage Nuru in Ghana: This program uses AI and ML to diagnose plant diseases and pests by using a smartphone app called PlantVillage Nuru. The app uses computer vision and deep learning to analyze images of plant leaves and provide instant diagnosis and treatment recommendations for smallholder farmers. PlantVillage Nuru exists! However, there is no mention of USAID as a donor.
• BanQu in Kenya: This program uses AI and ML to provide digital identities for refugees by using a blockchain-based platform. It uses biometrics and facial recognition to verify the identity of refugees, and uses data analysis and machine learning to connect them with humanitarian aid, financial services, education, employment, and supply chains. BanQu also exists! It also won an award with Chemonics for digital identity work, but doesn’t seem to have any USAID funding or project activity either.

4. Perplexity AI was mostly accurate and vague

You may not know about Perplexity AI, but I find its answers are often the most accurate and compelling. When I asked it to list USAID activities that are using artificial intelligence solutions for social impact, it returned links to multiple reports on USAID’s work in AI. None were links to projects like I was seeking, but at least it gave me real documents to review.

• Emerging artificial intelligence and machine learning (AI/ML) applications promise to reshape healthcare, agriculture, economic growth, and democracy in the developing world, according to USAID.
• The aim of this guide is to provide practical resources for the development community that engender an informed, thoughtful approach to using ML/AI and to manage machine learning projects.
• USAID’s Center for Innovation and Impact (CII) takes a business-minded approach to fast-tracking the development, introduction, and scale-up of health technologies, including those that incorporate AI.
• The purpose of this document is to start a conversation with a list of questions meant to help target issues of fairness in projects incorporating ML or AI.
• The USAID Digital Strategy is a development policy document focused on the promotion of secure, open, and inclusive country-level digital ecosystems and the use of digital technologies, including AI, to achieve development outcomes.
• The USAID Artificial Intelligence Action Plan outlines how the agency will leverage AI to achieve its development goals, including improving health outcomes, increasing food security, and promoting democracy and good governance.

Conclusion: Use Public GenAI Tools at Your Peril

My experience asking other questions of these free consumer Large Language Models trained on massive quantities of publicly-available text is similar. They can often produce very accurate looking responses that have very weak relationships to the questions your propose.

For example, none of these GenAI tools do any better when asked questions about the USAID Development Experience Clearinghouse. I fear that the DEC is our industry’s ultimate example of a PDF graveyard.

Now this could be that I am asking too-detailed questions, or I have a very advanced understanding of the topic. Either way, I would advise much caution when asking LLMs detailed digital development questions. Certainly do not rely on these generic solutions for formal reporting purposes.

Pro Tip: Tailored GenAI Solutions Work Better

There are emerging generative artificial intelligence solutions tailored to specific use cases that have much better responses.  These comprehensive tools produce reliable outputs that can improve USAID staff workflows.

One great example of a tailored GenAI solution is the Admin Services Knowledge EXchange Online.

• AskEXO is a AI-powered research assistant for Foreign Service regulations. It has a library of pre-formatted excerpts from the Foreign Affairs Manual, Foreign Affairs Handbooks, USAID Automated Directives System, Department of State Standardized Regulations, and USAID Acquisition Regulation.  When you ask it a question, the system picks one or more relevant policies from among that library, appends it to your query, and then directs GPT-4 to refer to that particular text when formulating its response.

The result is more accurate than any of the four systems listed above – for regulations in those libraries. Sadly, not even AskEXO can list USAID programs that use artificial intelligence for social impact nor parse the Development Experience Clearinghouse.