Think Before You Prompt: Why ChatGPT Makes a Lousy Attorney
Written by Andrew Foreman
Attorney, Foulston Siefkin LLP
AI is rapidly reshaping the world around us, and businesses are using AI for advertising, sales, customer service, coding, accounting and more. It is difficult to think of anything AI cannot be used for. I’ve used AI for everything from designing my daughter’s birthday invitations to researching the best evergreen hedge for my backyard (Manhattan Euonymus is a top contender).
It makes sense, then, that business owners would wonder: Why not use AI for legal advice and drafting contracts? Last year, Deloitte Australia, the Australian member firm of the well-known international accounting firm Deloitte, answered this question the hard way — through firsthand experience. The firm used AI to produce a report for the Australian government. Unfortunately, the report was filled with hallucinated legal citations and bogus conclusions. In the end, all the firm got for this embarrassing failure was lost revenue and a damaged reputation. Here are three reasons business owners should hesitate to rely on AI for legal advice or other legal services.
AI chatbots not only make mistakes; they make subtle mistakes.
When an AI chatbot cannot provide an answer to a user query, it will frequently hallucinate information in an effort to provide what it “thinks” the user wants to know. This is sometimes referred to as the AI sycophancy problem. Despite how frequently these basic mistakes go undetected, they are often easy to catch with a simple Google search.But legal mistakes are not always so easy to spot. It is much harder to determine when an AI chatbot misreads a real case or fails to include an important term in a contract. Even when AI chatbots technically provide correct answers, they can still fail to identify important exceptions, provide complete answers, or ask essential follow-up questions — in other words, do the job attorneys are hired to do
These subtle mistakes necessitate expert review.
Precisely because AI chatbots often make mistakes, reliance on AI-generated legal content without expert verification carries significant risk for the user. I recently asked an AI chatbot to draft a simple will. The AI chatbot failed to include witness signature blocks in the draft or even explain that witness signatures are required for the will to be enforceable in Kansas. The AI draft also granted all powers under a fictitious law, the “Kansas Nebraska Uniform Powers of Trustees Act,” and it failed to provide for contingent trusts for the decedent’s children. Even without a background in estate planning, some of these errors may still be caught, but any that evade detection risk devastating consequences.
Of course, this issue is far from unique to the practice of law. For example, a company that uses AI to develop a software application cannot simply ask the AI to confirm the application securely stores customer data. Likewise, businesses should not simply hope that an AI-drafted contract does not impose or otherwise create liability for their company. The inherent need for professional verification means many of the best use cases for AI will be by professionals supplementing and enhancing their own services, rather than by others attempting to replicate those same services.
ChatGPT doesn’t care about your business.
As AI models continue to improve in nearly every way imaginable, one striking limitation persists. Despite billions of dollars and years of research, programming moral values such as empathy, honesty or integrity has proven to be an elusive goal. Instead of making ethical machines, we’ve built masters of mimicry.
In contrast, the legal profession is built on a code of professional ethics developed over centuries. We old-fashioned attorneys care about our clients’ success, and we want to help them make good choices as they navigate difficult situations. Until we imbue AI with a conscience, ChatGPT won’t lose any “sleep” if it gives you bad legal advice that costs your business. Though it may sound like your friend, it doesn’t care what happens after you exit the chat and take its advice into the real world.
Relying on AI to provide legal advice or draft contracts is not dangerous because its output is obviously wrong, but rather, because it can be deceptively good. There is no profession or industry immune to the effects of AI, and the practice of law is no exception. In fact, you should hope your attorneys are learning to use AI (wisely) to create value for their clients. In 2023, Foulston Siefkin formed an internal committee to identify promising new AI tools and technologies that our firm can leverage to create value for our clients.To quote Spider-Man’s uncle: “With great power comes great responsibility.” AI is a powerful tool with tremendous potential. We are living in a world awash with AI-generated content, so remember: Think before you prompt.
This information is not a legal opinion, and it does not provide legal advice for any purpose. If you have questions or want more information, contact your legal counsel.
