A marketing provider sends you a screenshot. An AI tool has included your firm in a list of lawyers. The message sounds encouraging: your firm is getting noticed.
It is a useful observation. But before treating it as a business result, ask what the screenshot actually establishes.
It shows that your firm appeared in that answer. By itself, it does not show how often prospective clients encounter your name, what other users see, or whether anyone contacted the firm.
In a November 9 discussion about examining AI answers, James Dooley and his guests considered the limits of drawing conclusions from a single response. For lawyers assessing their marketing, the practical lesson is straightforward: ask for the context behind the example before accepting the conclusion attached to it.
Start with the question that produced the answer
The question or instruction typed into an AI tool is often called a prompt. Read the full prompt before evaluating the response.
Imagine a fictional firm called Example Family Law. Compare these two questions:
- “What can you tell me about Example Family Law?”
- “How should I choose a lawyer for a contested divorce in Chicago?”
The first question supplies the firm’s name. The second does not. An answer to the first can help you examine how the tool describes a known firm. It does not demonstrate that someone who has never heard of the firm would discover it.
Neither question is inherently wrong. They answer different business questions.
The same distinction applies when someone asks an AI tool to compare your firm with a named competitor. Being discussed in a comparison you requested is different from being suggested without your name in the question.
Previous messages also matter when interpreting an example. Ask to see the conversation that led to the screenshot, rather than a cropped answer alone.
Make sure you know which experience was tested
“We checked AI” is too broad to describe a useful review. Different products, modes and account settings can involve different conditions.
Google explains that AI Overviews and AI Mode may use different models and techniques, with responses and supporting links that vary. Its AI Mode help page also describes personalization for eligible users based on previous searches and saved activity.
Microsoft’s documentation for Copilot Chat and agents provides another example: access to web search can depend on account type, license and organization settings. A response using current web information is a different testing condition from one produced with web search turned off.
These facts do not explain why a particular firm appeared in a particular answer. They explain why the testing conditions belong in the report.
Ask which product and mode were used, when the check occurred, and what relevant settings were visible. If a detail was not available, the report should say so. A confident guess does not fill the gap.
Ask for the pattern behind the screenshot
A screenshot can be genuine and still support a much narrower conclusion than the accompanying sales pitch.
Suppose a hypothetical report records your firm’s name in six of twenty answers. The defensible statement is that the firm appeared in six of those twenty recorded answers, under the conditions used for those checks.
That does not establish that 30 percent of prospective clients see your firm. The questions may have been chosen by the provider. The answers may come from one account or one product. No actual client behavior has been measured in this example.
Twenty is not a recommended universal sample size. The point is that the number of checks and the way they were chosen affect what a result can tell you. A larger collection does not automatically represent your market.
A useful report should make these questions easy to answer:
- What exact questions were asked, and why were those questions relevant to the firm’s practice?
- Were the firm’s name or competitors’ names supplied in advance?
- How many checks were performed, and were unsuccessful examples retained?
- Were the same conditions used when comparing one period with another?
- Did the response merely mention the firm, suggest it as an option, or link to its website?
Repeating selected questions can help you see whether a pattern persists within the tests. It still does not reveal every prospective client’s experience or prove what caused a change.
Read what the answer actually says
An appearance is not automatically a favorable recommendation. A response might mention your firm while describing a service you do not offer, confusing two attorneys, or linking to an outdated page.
Check the substance before celebrating the mention. Does the answer describe the correct practice, lawyers and location? Does a linked page support the statement beside it?
If the answer contains a factual error, review the public information describing your firm. That is a more concrete starting point than assuming the firm needs more mentions everywhere.
Also be cautious about asking the AI tool to explain exactly why it chose your firm. The November 9 panel questioned how much insight an answering model has into the process used to select its sources. Treat an explanation as something to investigate, rather than an independent audit of that process.
Keep the conclusion proportional to the evidence
One favorable answer does not establish that your firm consistently reaches the right prospective clients. One omission does not establish that your firm is invisible to them.
Use the observation to ask a better question. Is the firm’s information accurate? Are the test questions relevant to the matters it handles? Is a claimed improvement visible across a documented set of checks, or only in the example selected for presentation?
Those distinctions can help you assess a marketing recommendation without needing to understand the technical machinery behind every AI tool.
Find out what the AI examples really show
If you have been shown an impressive AI screenshot, or are concerned that your firm is being overlooked, start by examining what was actually tested.
Schedule a detailed review of your firm’s online presence with BSPE Legal Marketing. We can examine your website and public profiles alongside the AI answers being reported, helping you distinguish inaccurate information from conclusions the available evidence does not support.
