ICTworks

OpenAI released ChatGPT 3.0 to the world exactly one year ago and we’ve all been on a wild Generative AI ride since then.

Thankfully, as loyal ICTworks readers, you’ve been on the chatbot journey since 2016 exploring how these tools can be used to advance international development objectives. You’ve also already understood ChatGPT’s skills when we explored if chatbots could fool you in 2020.

OpenAI has experienced leadership challenges recently and Anthropic, a rival GenAI company, released Claude 2.0 to compete with the newly-released ChatGPT 4.0. It’s time to test chatbot LLMs again!

ChatGPT vs Claude on FAR Analysis

ChatGPT 4.0 and Claude 2.0 are very capable large language models (LLMs) that can produce realistic text when asked questions on pretty much any topic. However, how does each model work when focused on a specific text that is very relevant to international development?

Sign Up Now to see more GenAI explorations!

I decided to have a contest between ChatGPT and Claude using the Federal Acquisition Regulation, the official guide for US federal government contracting. I chose the FAR because its a real challenge for any human to comprehend, yet the FAR is a public set of text-based rules. Therefore, it should be a perfect challenge for Generative AI.

The FAR spans thousands of pages of policies, clauses, provisions, and procedures related to government acquisition, and is over 4x the size of the Bible:

• The King James version of the Bible contains around 783,000 words and if you read it at 250 words per minute for 8 hours per day, it would take you 6.5 days to read the entire King James Bible.
• Printed copies of the complete FAR can span 10,000+ pages, and reading it at 250 words per minute for 8 hours per day would take you over 25 days to get through the full FAR documentation.

GenAI Prompt for FAR Analysis

I am not a GenAI prompt engineer (yes, prompt engineering is a defined job now!) but here is my simplistic prompt for both ChatGPT and Claude that I used to confine their analysis to the FAR and generate factual responses with links back to the underlying FAR clause.

You are a factual authority on the FAR. You will contain your analysis to the FAR website. You will answer in the format of answer, FAR Clause, URL.

I would love improvements on this prompt to get better results from either LLM. Please add your prompt ideas in the comments to help LLMs better answer questions about US government regulations.

ChatGPT vs Claude: Which One is Better?

As you’ll see below, ChatGPT and Claude answers were good and different, reflecting the different training models for their learning and software algorithms that guide their responses.

• ChatGPT 4 had more verbose answers that were closer to FAR language
• Claude 2 gave responses that could be better understood by non-FAR experts.

Either GenAI tool could be utilized to better understand the Byzantine intricacies of the FAR. However, neither are better than a FAR expert who knows the regulations as a profession, as you will see below.

FAR experts spend years immersed in government acquisitions policy and countless hours in training and practice to be effective. ChatGPT and Claude can be a first attempt at refining an eventual request of a FAR expert.

Q1: What Are The Responsibilities of a COR?

My first question to both ChatGPT and Claude on the Federal Acquisition Regulation was a simple one. “What are the responsibilities of a contracting officers representative according to the FAR?” Here are their unedited responses:

ChatGPT 4: What are COR responsibilities?

The responsibilities of a Contracting Officer’s Representative (COR) are outlined in various parts of the Federal Acquisition Regulations (FAR). Here are the key responsibilities:

1. Technical Monitoring and Administration (FAR 1.604): The COR assists in technical monitoring or administration of a contract. They are required to maintain a file for each assigned contract, which must include:

• A copy of the contracting officer’s letter of designation and documents describing the COR’s duties and responsibilities.
• A copy of the contract administration functions delegated to a contract administration office, which may not be delegated to the COR.
• Documentation of COR actions taken in accordance with the delegation of authority .

2. Contracting Officers’ Responsibilities (FAR 1.602-2 ): While not directly describing COR duties, this section provides context to their role. Contracting officers are responsible for effective contracting, compliance with contract terms, and safeguarding U.S. interests. They designate and authorize CORs in writing, specifying:

• The extent of the COR’s authority.
• Limitations on the COR’s authority.
• The period covered by the designation.
• That the authority is not redelegable.
• Potential personal liability for unauthorized acts.

The COR must be a government employee, certified and qualified for their responsibilities, and cannot make any commitments or changes affecting contract terms.

3. General Authority and Selection of Contracting Officers (FAR 1.603-1): This section highlights the broader framework within which CORs operate. It mandates agency heads to maintain a procurement management program and system for appointing contracting officers. The selections and appointments must align with the Office of Federal Procurement Policy’s standards for skill-based training in contracting and purchasing duties.

Claude 2: What are COR responsibilities?

According to FAR Part 1.602-2, the responsibilities of a contracting officer’s representative (COR) include:

1. Technical monitoring of the contractor’s performance and progress under the contract. This includes inspecting and accepting deliverables, assessing reports, and monitoring technical progress.
2. Performing inspections necessary in connection with contract performance. This includes inspection of things like materials, components, work in process, etc.
3. Making recommendations for acceptance or rejection of contractor deliverables. The COR analyzes deliverables and recommends acceptance/rejection to the contracting officer.
4. Assisting in contract administration, which involves things like reviewing invoices and monitoring government-furnished property.
5. Making sure the contractor complies with reporting requirements in areas like progress, performance, and status.

So in essence, the COR serves as the “eyes and ears” for the contracting officer when it comes to overseeing the contractor’s work and contract compliance. But the COR does not have the authority to make contractual commitments or direct changes on behalf of the government.

Q2: Does FAR Prohibit Certain ICT Purchases?

For my next question to ChatGPT and Claude, I wanted to get specific to digital development issues that impact our ability to design new ICT4D solutions. I asked each chatbot, “Are USG programs prohibited from purchasing certain ICT hardware or software?”

ChatGPT 4: FAR prohibitions on ICT purchases

According to the Federal Acquisition Regulations (FAR), United States Government (USG) programs are indeed prohibited from purchasing certain Information and Communications Technology (ICT) hardware and software, primarily for reasons of national security and protection of governmental interests. Here are the key prohibitions outlined in the FAR:

1. Prohibition on Kaspersky Lab Products (FAR 52.204-23): This clause specifically prohibits contracting for hardware, software, and services developed or provided by Kaspersky Lab and other covered entities. The prohibition, which came into effect following the National Defense Authorization Act for Fiscal Year 2018, applies to any hardware, software, or service developed or provided, in whole or in part, by Kaspersky Lab or related entities. Contractors are prohibited from providing any covered article that the government will use on or after October 1, 2018, and from using any covered article in the development of data or deliverables produced under the contract.

2. Prohibition on Certain Telecommunications and Video Surveillance Services or Equipment (FAR 4.2102): As of August 13, 2019, agencies are barred from procuring or obtaining, or extending or renewing contracts to procure or obtain, any equipment, system, or service that uses covered telecommunications equipment or services as a substantial or essential component of any system, or as critical technology as part of any system. This prohibition extends to contracts with entities that use such covered telecommunications equipment or services. There are exceptions for certain services and equipment that cannot route, redirect, or permit visibility into user data traffic.

3. General Prohibition on Kaspersky Lab Products (FAR 4.2002): Similar to FAR 52.204-23, this part emphasizes the prohibition on government use of any hardware, software, or services developed or provided, in whole or in part, by Kaspersky Lab or related entities. This prohibition applies both to the provision of covered articles to the government and to their use in contract performance.

4. Prohibition Related to Information Technology (FAR 39): Contracting officers are instructed not to purchase any hardware, software, or services developed or provided by Kaspersky Lab that the government will use on or after October 1, 2018.

These regulations reflect the government’s focus on ensuring the security and integrity of its technological infrastructure and operations.

Claude 2: FAR prohibitions on ICT purchases

Yes, certain federal agencies and programs are prohibited from purchasing certain information and communications technology (ICT) hardware and software under section 889 of the FY2019 National Defense Authorization Act (NDAA).

Specifically, section 889(a)(1)(A) prohibits executive agencies from procuring or obtaining any equipment, system, or service that uses covered telecommunications equipment or services as a substantial or essential component or as critical technology as part of any system. Covered telecommunications equipment includes telecommunications equipment produced by Huawei Technologies Company or ZTE Corporation, or any subsidiaries/affiliates.

Additionally, section 889(a)(1)(B) prohibits agencies from entering into contracts (or extending contracts) with entities that use any equipment, system, or service that uses covered telecommunications equipment, as defined above.

There are some exceptions and implementation timing nuances outlined in FAR Subpart 4.21 and the associated clause 52.204-25, but in general certain federal agency programs are prohibited from buying ICT hardware/software from Huawei, ZTE, or affiliates.

Q3: Can the US Government Purchase Live Goats?

My last question to ChatGPT and Claude is meant to be both a funny question to ask of the FAR and a serious test of the LLMs knowledge of how government regulations can be used and applied.

Skilled procurement officers know that the USG can purchase anything as long as it is not specifically prohibited by the FAR. Let’s see if the LLMs can look deep into government regulations to see if there is a rule against a certain purchase and if not, allow it.

ChatGPT 4: Can USG buy live goats?

The Federal Acquisition Regulations (FAR) do not explicitly prohibit the U.S. government from purchasing live animals, such as goats. However, there are specific regulations and guidelines that govern the acquisition and use of live animals, especially in the context of research, development, test, evaluation, or training activities.

According to the Defense Federal Acquisition Regulation Supplement (DFARS) 252.235-7002, which is part of the FAR System:

1. Registration and Approval Requirements: Contractors who use animals in research, development, test, evaluation, or training must register their facilities with the Secretary of Agriculture as per the applicable U.S. code and regulations. Additionally, the proposed use of animals must be approved in accordance with Department of Defense Instruction (DoDI) 3216.01, “Use of Animals in DoD Programs.” Contractors are required to provide evidence of such registration and approval to the Contracting Officer before beginning work under the contract​

2. Animal Acquisition: Contractors must acquire animals in accordance with the guidelines set forth in DoDI 3216.01​

3. Compliance with Laws and Regulations: The care and use of animals must conform to the pertinent laws of the United States, the regulations of the Department of Agriculture, and the policies and procedures of the Department of Defense. This includes compliance with specific U.S. codes and the Code of Federal Regulations (CFR) pertaining to animal welfare.​

These regulations suggest that while the purchase of live animals, such as goats, is not outright prohibited, it is subject to stringent regulations and oversight, particularly in the context of their use in government-related activities. The primary focus of these regulations is to ensure animal welfare and compliance with relevant laws and guidelines.

Claude 2: Can USG buy live goats?

Yes, the US government can buy live goats. There is nothing in the Federal Acquisition Regulation (FAR) that would prohibit the purchase of live goats, as long as proper procedures are followed. Some key points:

• Agencies must first check available inventories, excess supplies from other agencies, Federal Prison Industries, and AbilityOne sources before purchasing from commercial sources (FAR 8 – Required Sources of Supplies and Services).
• For commercial purchases, agencies must use full and open competition to the maximum extent practicable (FAR 6 – Competition Requirements).
• For purchases over the micro-purchase threshold, agencies must publicize contract actions to increase competition, transparency, and fairness (FAR Part 5).
• Acquisitions must comply with small business set-asides where applicable (FAR 19 – Small Business Programs).

So in summary – yes agencies can purchase live goats if needed, provided they adhere to regular federal contracting procedures for pricing, competition, transparency, fairness, and small business participation as outlined in the FAR.

We Can Do Better on FAR

Both public LLMs have their advantages and issues. They are general purpose tools we are trying to use on a very specific data set. We can do better than either using a targeted approach specific to the FAR.

I helped build a FAR Bot (and FAM/FAH Bot) at my last employer that used an open source LLM trained just on the FAR (and FAM/FAH) to answer specific procurement questions with more accurate responses and better detail.

I’m sure that those systems are even more advanced now and can be trained on any large body of knowledge set – be it procurement rules (FAR), human resource policy (FAM/FAH), legislation (imagine a Federal Law Bot!), etc. Since you’ve read this far, tell me what you’d like to see analyzed by GenAI and I’ll explore it in an upcoming post.