The current narrative surrounding ChatGPT goes something like this:
Keywords are dead. Users are engaging in long, complex, conversational back-and-forths with AI to find exactly what they want. Then, they hop over to Google to verify everything.
It sounds plausible. But is it true?
We wanted to move past the anecdotes and theory.
We did a 24-hour observational study where we sat down with 52 everyday people and tasked them with finding local healthcare services, resulting in 177 ChatGPT sessions.
We didn’t force them to stay on ChatGPT, but we started them there and observed where their habits took them.
The results were in direct contradiction to what is currently believed by our industry.
It turns out that old habits die hard, and when your customers use AI to find local services, it looks a lot like Google keyword searches from 2003.
Here is what we found.
Background
We observed 54 people across 177 prompts to see how they used ChatGPT, Google, and websites to choose a provider of the following services:
- Find a new dentist
- Find a place to get botox
- Find a dermatologist to check a mole
- Find a surgeon to get a hair transplant
- Find a chiropractor
- Find a plastic surgeon to get a facelift
We gave each participant a scenario. For example:
You woke up this morning and noticed a mole that concerns you. You need to find a dermatologist to get it checked out. You’ve decided to start your search on ChatGPT, but please act as if you were not being observed. You are free to visit the website, check social media, look at review websites, whatever you would normally do.
Participants lived throughout the US and Canada.
Age ranges:
| <30 | 25 |
| 30-50 | 22 |
| 50-65 | 5 |
| 65+ | 2 |
75% of Sessions Included Traditional Keyword Searches
I was so bought into the idea of “conversational search” that we didn’t even set out to measure this initially. But after observing users searching by “keywords” instead of natural language in our Google AI Mode study, we set out to see if the same where true.
What did we find?
75% of user sessions included at least one prompt that would be categorized as a “keyword search.”
We are talking about users providing prompts like:
- “dentist near Boston”
- “best affordable dentist Manhattan”
- “Petersburg Virginia.”
You can view the entire list of prompts here.
The reality is simple: it requires significantly less effort to type “dentist 11214” than it does to type “Find me 5 good dentists according to online recommendations near India Street, Brooklyn, New York.”
(Both of those examples came from our study).
The user who typed the keyword got exactly what they were looking for; If “good plastic surgeons in brooklyn 11214 area” gets the job done, users see no reason to write more.
45% of Sessions Were One-Shot Prompts (And 34% of Follow-ups Just Wanted “More Results”)
The industry narrative asserts that users are having long, detailed sessions with AI to refine their choices.
Our data paints a different picture.
45% of the sessions—almost half—consisted of a single prompt. The user asked for a business, got a list, and moved on.
Even among the users who did ask a follow-up question, 34% of those second prompts were just asking for “more results” rather than refining their criteria conversationally.
We broke down the average number of prompts per session based on the task, and the numbers are consistently low:
| Task Name | Avg # of Prompts |
| Find a new dentist | 2.41 |
| Find a chiropractor | 2.33 |
| Decided to get a facelift | 2.00 |
| Find a place to get botox | 1.96 |
| Find a dermatologist | 1.71 |
| Hair transplant | 1.33 |
This suggests that people are largely satisfied with the results they get with their first prompts, though we did observe follow-up questions that asked for help refining the results they received.
The Average User Considered 3.7 Businesses (Only 27% Looked at Just One)
In traditional SEO, the #1 spot takes the lion’s share of clicks. In the world of ChatGPT, the behavior appears slightly different.
Our average user checked out 3.7 different businesses from the results provided.
Only 27% of our participants looked at just one business.
That means three out of four users are going to explore multiple options.
This is critical for search marketers: in ChatGPT, you don’t necessarily need to be the absolute first recommendation to win the business.
You just need to be part of the “consideration set.” If you appear in the top 3-5 results, you still have a strong chance of winning the click.
39.2% Of Users Visited the Website from ChatGPT
The most salient difference in behavior between Google AI Mode and ChatGPT is the small number of users that clicked through to the website.
In our study only 41.5% of the participants visited the website.
40% felt “ready to call and make an appointment” based entirely on what they found through ChatGPT.
21% Switched to Google to Verify (And 17.6% Clicked Through from Google)
There is a belief that people research on ChatGPT and then strictly switch to Google to “verify” the information.
We found this to be true 20.5% of the time.
17.6% of our participants clicked to the business website from a Google Search AFTER finding the brand on ChatGPT.
This is an attribution nightmare.
If a user finds you on ChatGPT, opens a new tab, and Googles your brand name to find your site, that traffic shows up in your analytics as “Google Organic” or “Direct.”
In reality, the credit may belong to ChatGPT, but you won’t know unless you ask.
As AI search grows, your organic traffic reports may start claiming credit for leads that actually originated in an LLM.
