ICTworks

A few months ago, I was in Pakistan working with my colleagues on a rapid landscaping of AI in Agriculture. We decided to put ChatGPT (v3.5) to a simple test. We asked it “when is the best time to plant wheat in Faisalabad?” in both English and several local languages.

After reading the ICTworks post that pitted ChatGPT vs. Claude on the FAR, I decided to ask both ChatGPT and Claude the same Faisalabad wheat challenge to see if either could be a Pakistani agricultural extension agent. Here are the results.

ChatGPT vs Claude in English

I asked both ChatGPT and Claude in English when is the best time for farmers to plant wheat in Faisalabad? Both answered the question generally correctly.

ChatGPT’s Response

The best time to plant wheat in Faisalabad, Pakistan, typically falls between late October and mid-November. This time frame aligns with the Rabi season, which is the winter cropping season in South Asia. During this period, the weather conditions are favorable for wheat cultivation, with cooler temperatures and sufficient moisture in the soil.

It’s important to note that agricultural practices can be influenced by local climate variations and specific conditions, so it’s advisable to consult with local agricultural extension services or experts for the most accurate and up-to-date information regarding the optimal planting time for wheat in Faisalabad. Additionally, farmers often consider factors such as soil moisture, temperature, and weather forecasts when deciding on the exact timing for planting.

Claude’s Response

The best time to plant wheat in Faisalabad, Pakistan is from mid-October to mid-December. More specifically:

• Early to mid-October: Planting should start in early October and continue until mid-October. This timing allows the wheat to become well-established before the winter months.
• Mid November: Another good planting window is in mid-November, around 2-3 weeks after the first window. This later planting can produce good yields if the weather cooperates.
• Stop planting by mid-December: Wheat planted much later than mid-December is at high risk of having low yields. Temperatures start warming up and there may not be enough chill hours for the wheat to properly vernalize.

So in the Faisalabad region, the prime wheat planting window spans from about October 15th through December 15th. Adjust exact planting dates each year based on monsoon rains and short-term weather forecasts. But in general, aim to get wheat seeded from mid-October to mid-December for best results. Let me know if you need any other details!

Follow On Question on Wheat Varieties

I asked a follow up question after the first one to see if either Large Language Model chatbot had a deeper understanding of available wheat varieties: “Which wheat variety is the best to plant in Faisalabad for maximum yields?” Here’s where a real difference emerged.

ChatGPT gave a fairly noncommittal response that began with, “The choice of wheat variety for maximum yields in Faisalabad, or any other region, depends on various factors such as climate, soil type, water availability, and local agricultural practices. As of my last knowledge update in January 2022, I don’t have specific information about the latest wheat varieties released after that date.“

Claude, on the other hand, provided recommendations of four wheat varieties, along with their potential yield and benefits. When I shared those with my colleague in Pakistan, he noted that its response was accurate and detailed, but that it excluded some newer varieties.

I then asked Claude, “I hear that there are some newer and better wheat varieties now available that are good for Faisalabad. Do you know about those?” Claude won the battle between ChatGPT and Claude in English handily with this response as follows:

Yes, there are some promising newer wheat varieties being promoted for the Faisalabad region of Pakistan by agricultural researchers:

1. Pakistan-18 (Pak-18): This variety was released by AARI in 2018 specifically for cultivation in Punjab province. Field testing shows it to have 5-8% higher grain yield compared to older popular varieties in the region.
2. Markaz-19: Released by MARC in 2019, Markaz-19 has shown exceptionally high yield potential under irrigation, over 6 tonnes per hectare. It also has better time to maturation. However, it has lower drought tolerance than some varieties.
3. TDK-20: Released from the Faisalabad district itself, results show TDK-20 giving yields roughly 10% higher than older check varieties. It has performed very well under variable temperature and water conditions.
4. NARC-Potohar: Created for northern Punjab, this variety has done quite well during recent trials in Faisalabad also. It has excellent resistance to yellow rust disease and wide temperature adaptation.

In addition to yield improvements, some of these newer varieties have better disease package, grain quality, and climate resilient traits. I would recommend checking with local agricultural experts and universities each season for the very latest variety recommendations for the Faisalabad area to make the most informed wheat planting decisions. Let me know if you need any other details!

ChatGPT vs Claude in Local Languages

So Claude won the first round, but how about when we put them to the test in Urdu, Pashto, Sindhi, and Punjabi? When it came to Urdu, both recognized Urdu and responded to the question back in Urdu similarly to how they did in English.

Both also recognized and were able to respond in Pashto. Here, Claude gave a generally correct response, suggesting that wheat can be sown any time in November or December, but that early November may result in higher yields. ChatGPT, on the other hand, gave a bit of a whimsical response, suggesting that the best time to plant is in the season of the ‘lion’ and the ‘whale’.

Likewise, both recognized Sindhi and were able to respond in Sindhi. However, their responses were both far from accurate. ChatGPT gave a very broad window of October through March, while Claude suggested that March is the best time to sow wheat.

Punjabi was an interesting challenge, that I was sure both would fail, since in Pakistan, Punjabi is written in Latin characters (referred to as Roman Punjabi) and not in the Punjabi script that is still used in India.

As I had suspected, Claude recognized that I was trying to ask it a question in a language other than English, but it said it could not recognize the language. ChatGPT, to my surprise, recognized it as Punjabi and responded back in Roman Punjabi, although its response was fairly comedic. It interpreted ‘time’ very literally to mean ‘time of day’ and noted that the best time to sow wheat is from midnight until noon, without mentioning a specific month or time of year.

Neither GenAI Understands Local Languages

In the battle between ChatGPT and Claude in local Pakistani languages, it’s a draw in Urdu and there are no winners in the other languages. This is about what I expected, as Urdu is better represented online and therefore more likely to have been included in their LLMs. On Wikipedia, for instance, there are just under 200,000 articles in Urdu, but only 51,554 in Punjabi (using the Punjabi script), 18,506 in Pashto, and 15,488 in Sindhi.

Can GenAI Chatbots be Ag Extension Agents?

In some ways, I was surprised that both ChatGPT and Claude’s broadly generalized models did as well as they did when answering questions specific to a single crop in a specific location in Pakistan.

Are they ready to be deployed as agricultural extension agents directly to farmers, especially those living in areas that are underrepresented in terms of data and content in local languages? Certainly not. In fact, I would say that it would be irresponsible to encourage these farmers to use these generalized models for advice and decision making at the moment.

However, it does demonstrate how quickly generative AI models are advancing, and highlights the potential opportunity for customized agriculture advisory chatbots built off of these LLMs that are trained on localized content, such as Farmer.Chat and Agri1. Even those, though, are still in the (relative) early days of development and are going to require a lot more training, testing, and iteration, before they are accurate and dependable enough to be used directly by farmers as a primary source of advice.

In addition to the above, there is a lot more that we need to consider if we want to increase the likelihood that generative AI tools are optimized for inclusive benefit, and mitigated for risk and climate costs. For those who want to dig deeper into this issue, a great place to start is the Inclusively Advancing Agri-Food Systems Through AI and Automation report that USAID and the Bill & Melinda Gates Foundation commissioned earlier this year.

The views and opinions expressed in this blog post are those of the author and not necessarily the views and opinions of his employer, which at the time of writing is the United States Agency for International Development.