10 Questions to Ask AI Legal Software Vendors Before You Sign a Contract

What’s at stake when you choose a legal AI vendor

When you adopt AI, you hand an outside company your most sensitive client files. Most firms accept that for the time savings, and whether it pays off comes down to what you locked in before you signed. After that, a bad choice is hard to walk back.

Firms that skip the vetting tend to follow the same pattern. Nothing goes wrong at first. Then, months later, you read the fine print and find out the contract let the vendor train on your files all along. The accountability still lands on you. Under law society guidance across the provinces, your duty of confidentiality covers client information no matter where it’s stored or who handles it, and under PIPEDA you remain accountable for personal information you hand to a service provider.

None of this is a reason to avoid AI. It’s a reason to be deliberate about the tools you choose. Clio’s Legal Trends for Canadian Law Firms found Canadian legal professionals are among the most AI-forward in the world—95% of firms are using AI and 66% of legal professionals say it’s increased their revenue. Yet much of that work still runs through generic, consumer-grade tools that were never built for client files. Knowing how to evaluate AI legal software means asking the right questions, and asking them early. 

10 questions for AI legal software vendors

These are the questions a demo often won’t answer on its own. Write down each vendor’s response so you can compare them side by side later.

1. Where is our data stored, and is it used to train your AI models?

Ask this one first, because a weak answer here ends the evaluation no matter how good the product looks. AI is safe to use with confidential client data when the vendor stores that data in a jurisdiction you have agreed to, encrypts it in transit and at rest, and commits in the contract never to use your files for model training.

From there, press for specifics. Find out which country holds the data, and where the backups live. Since most AI runs on a third-party model underneath, ask which subprocessors are involved and where they operate. Then confirm whether the vendor has zero-data-retention agreements with those providers. Because your firm stays accountable for personal information wherever a service provider processes it—a core PIPEDA principle—confirming data storage location first is the best practice.

Storing data elsewhere isn’t automatically a problem, but it can create an obligation of your own. Alberta firms have to notify clients before transferring their personal information to a service provider outside Canada. Quebec firms must assess the transfer before information leaves the province. Either way, you can’t meet the requirement if the vendor won’t tell you where the data goes.

2. What accuracy and testing standards can you demonstrate?

Once you know how the vendor handles your data, the next question is whether the tool is any good. Every vendor claims accuracy. The more important question is how they got the number. A good vendor can talk you through how they tested the product, how it catches hallucinations, and how it did on the type of work you actually do. If they can’t tell you what they tested or on whose law, the number doesn’t mean much.

A demo isn’t a test either. The only way to really know is to run a pilot. Build a few fabricated matters that look like the work you actually do, and give every vendor on your shortlist the same documents and tasks. See how accurate the outputs are, and how long you spend checking them. 

3. Are outputs explainable and backed by verifiable sources?

Canadian courts have started sanctioning lawyers whose filings contained AI-generated citations to cases that don’t exist. In Zhang v Chen in British Columbia, counsel was ordered to personally pay costs for the extra work fabricated cases created. In Ko v Li in Ontario, fake citations triggered contempt proceedings, and the count of similar incidents has climbed since. No tool takes the work of verifying AI outputs off your hands, but a strong one makes it far quicker.

Have the vendor trace a single claim back to its source while you watch. Either the click lands on the passage that supports it, or it hands you a case name to track down yourself, which is the research you bought the tool to avoid.

4. Does it have access to Canadian case law and legislation?

AI legal software works with Canadian law only when the vendor builds it on Canadian sources and keeps them current. Most legal AI is built on top of a general-purpose frontier model, and those models have absorbed far more American legal material than Canadian, simply because there is more of it. Left uncorrected, that imbalance pulls answers toward U.S. authority, and none of it is visible from the outside. Ask the vendor to document the depth and breadth of its Canadian coverage in the areas you practice, including which courts and statutes, how far back, and how often it’s updated.

Put a question to it that turns on federal law, another on a provincial limitation period, and another that turns on a statute amended in the past year. The failure you’re screening for is the tool that answers a Canadian question with American law, or one province’s rule with another’s, in the same confident tone it uses when it’s right.

5. How does the tool integrate with our existing tech stack?

A tool that sits outside your document management, practice management, and billing systems asks people to change how they work, and that’s usually where it fails. A file has to leave one system, run through the tool, and come back. That holds up for a few weeks, until a busy stretch hits, the extra steps get skipped, and a tool you’re paying for goes unused.

So ask which of your systems it connects to today, and what each connection does. It’s not enough that a tool can read a document you upload. You want one that works with the matters and files already in your practice management system, pulls in the right billing and deadline information, and puts its work back where your team already is.

What you’re testing for is context. AI that can see the matter behind a request needs less correcting than AI working from a single file you uploaded, and that context only exists when the tool sits inside the system holding your matters. That’s the design behind the AI in Clio Manage, and it’s a fair standard to hold any vendor to. Whichever vendor you’re looking at, ask them to show you the integration, not just describe it. Comparing how other legal AI tools available to Canadian firms handle this tells you what’s standard and what a vendor is dressing up as a selling point.

6. What security certifications and safeguards do you have?

Good security means encryption in transit and at rest, independent validation such as a SOC 2 Type II audit, limits on who inside the company can see your files, and a written incident response plan. Independent review carries weight because an outside auditor has checked the vendor’s practices instead of the vendor vouching for itself. Some vendors publish their security documentation openly. If Clio is on your shortlist, you can start your due diligence in our Trust Centre.

Then ask for the report itself. A vendor with a current SOC 2 will usually share it under an NDA, and the exceptions it lists tell you more than the badge on the website does. That’s where the auditor recorded the moments a control didn’t work as designed. From there, find out who can reach client data and when—a reputable vendor will show you the access controls and logging that keep that group small. Last, ask whether they’ve had a breach and what changed afterward. The way they walk you through it is how they would handle the same conversation with your firm.

7. How do you support human oversight and review?

Across the provinces, you stay responsible for AI-assisted work. You have to supervise the work product and check it before it goes out. What the rules don’t tell you is how a busy firm keeps that up. A tool that counts on someone catching every problem afterward won’t survive a heavy week, so the controls need to be built into the product. It’s a basic part of AI legal compliance for Canadian firms.

So ask what the tool does to document accountability. Does it log who used it, on what, and when, so you can show how a document was produced instead of reconstructing it later? A vendor built for legal work will have these answers.

8. What happens to our data if we switch vendors?

Settle the exit terms before you sign, because that’s when you have the most leverage. Once your files are in the system and your team is trained on it, switching gets harder.

Review the terms of service, and confirm which formats you can export in. Many vendors support common formats like .csv. Then ask whether that covers the work the tool produced or only the documents you put in. Find out how long an export takes, what it costs, and how soon your data is deleted afterward, backups included. If not already included in the ToS, ask for the deletion timeline in writing, with a specific number attached.

The ease of the answer tells you a lot. A vendor confident in its product won’t mind the question. Vague timelines, export formats only they can open, and fees that show up only when you leave all tell you the opposite, and that cost is part of what you sign up for now.

9. What ongoing training, support, and updates are included?

Everything you have tested so far describes the product as it is today, and AI products change under their users. A model update can move output quality in either direction without anyone at the firm noticing, which isn’t how conventional software behaves. The bigger question behind training and support is who tells you when the product changes, and what they do about it.

Find out what onboarding comes with the licence, whether you get a named contact or a support queue, and what the documentation looks like for someone who wasn’t in the room for the pilot. Then get to the part firms miss. Ask whether the vendor gives advance notice of significant changes, whether they retest after an update or let the version you validated quietly become one you didn’t, and whether they can tell you which model produced a given output six months ago.

10. Does the vendor have real legal-industry expertise?

Purpose-built legal AI and a general-purpose model with a legal logo on it are different products. By this point in the conversation, you’ll have a good sense of which one you’re looking at. A vendor with legal depth has been answering the previous nine questions easily all along. Still, put it to them directly, and treat “built for legal” as a claim that needs evidence.

Ask for source coverage documented by jurisdiction, testing tied to actual legal tasks, and practicing lawyers involved in building the product. Ask, too, what the tool doesn’t do well, because a vendor that knows its own limits and writes them down is one that understands the work.

How to calculate the real value of legal AI

The clearest way to price a tool is to start with a concrete outcome. “Efficiency” is hard to measure, but “the AI-drafted NDA needs one round of redlines instead of three” isn’t, and naming that outcome first is what lets the pilot show whether the tool delivers.

Then look at time. Vendors talk about how fast the tool works, whether that’s a draft, a summary, or a case analysis. But you still have to read what it produced, check it against the file, and fix what’s wrong. That’s the part that decides whether you saved any time at all. During the pilot, time the whole job from start to finish, review included, and do it task by task. Some work, like a discovery summary, may come back nearly clean. Other work, like first-draft research, may take longer to finalize.

Cost works the same way. The licence fee is what’s in the proposal. The true cost also includes setting the tool up, connecting it to your other systems, training your team, paying more as more people use it, and getting your data back out when you leave.

Mistakes to avoid when choosing an AI legal software vendor

Firms that get this wrong are rarely careless. They’re usually just short on time, working against a sales process that moves faster than a proper evaluation does. The same few missteps come up again and again.

• Rushing the decision. It usually starts with a competitor’s announcement or a client asking about AI, and suddenly a decision that deserves a few weeks gets made in a couple of days. Nothing about the technology requires that pace. The pressure comes from the sales process, and you don’t have to take it on.
• Bringing in IT, security, and leadership last. When the timeline is tight, the people who will run the integration and answer for a breach often don’t see the contract until terms are settled. At that point they can either approve something they’ve never read or reopen a deal everyone thought was done. Bring them in at the shortlist stage instead.
• Skipping the trial. The rush also squeezes out the trial, which is the step that matters most. A demo shows the product on documents the vendor picked. Your practice runs on disorganized productions, matters with twelve amended pleadings, and clients who photograph paper with a phone. Build your test files to look like that, because it’s the only way to find out how a tool handles the work you actually do.
• Pricing on generation speed. Every tool looks fast in a demo, which is exactly why speed doesn’t tell you much. What you’re buying is the time saved on finished work, once review and corrections are counted. And the costs that decide whether the purchase was worth it, such as training and added seats, all show up after signing.
• Treating the evaluation as finished. Even a careful process ends too early if it stops at the contract. The tool you tested early on can work differently a few months later, because the model underneath changed. Write down what you were promised, ideally in your firm’s AI policy, and check it after major releases and before each renewal.

Bringing secure AI to Canadian law firms

None of this is a bar that vendors can’t clear. Plenty can, and that’s what makes the questions worth asking. Some general-purpose AI tools do train on whatever users put into them, and their terms say so. But that’s a decision about how those products were built, not a limit of the technology.

Take the data question. Encryption in transit and at rest, data stored in your region, and a commitment not to train on client files are all on the market today. That’s how Clio handles data for Canadian firms. Your data stays within Clio’s Canadian infrastructure, and it’s never used to train third-party models. Clio completes annual SOC 2 Type II audits, and those reports are in our Trust Centre alongside the rest of our security documentation.

Purpose-built Canadian law coverage exists too. Clio Work, Clio’s AI for legal analysis, case strategy, drafting and research, is available in Canada and built on Canadian case law and legislation—through its purchase of Jurisage and its database of more than 470,000 Canadian cases across more than 40 courts. Tools built on Canadian law already exist, so you don’t have to accept “our product is global” as an answer.

The right AI partner makes the difference

With most software, a bad choice is something you can just work around. Legal AI is different. The wrong choice attaches to your client data and to duties you can’t hand off, and it stays with the firm long after you’ve forgotten the decision that caused it.

That’s why the questions to ask AI legal software vendors are critical to getting the decision right, not something to set aside once signing is done. These questions get at what actually separates one vendor from another: where your data lives, how rigorously the tool was tested, whether it genuinely knows Canadian law, and what happens if you ever need to walk away. Press on the answers, run a pilot on files built to look like your own matters, and work out the full cost beyond the licence price before you sign. Take the list into your next vendor call. The answers you get, and how quickly you get them, will tell you most of what you need to know.

If you want to see how Clio measures up, book a demo and put us through the same questions as everyone else.