PageSpeed Insights now flags a missing llms.txt, and plenty of people sell it as the key to AI search. We checked what Google says on the record and what crawl logs actually show.
Key takeaways
- No major AI provider or search engine has committed to using llms.txt as a ranking or citation signal. Google's John Mueller: "no AI system currently uses llms.txt."
- In Ahrefs' study of 137,210 domains, 97% of llms.txt files received zero requests in a month. In EZY.ai's logs, OpenAI fetched robots.txt 3,990 times and llms.txt 7 times.
- A PageSpeed Insights warning means Lighthouse checks for the file, not that Google uses it. The audit's category is marked as still under development.
- The fundamentals still decide who gets cited in AI answers: crawlable HTML, clear answers, structured data and real authority.
“Do we need an llms.txt file? Someone told us it’s the new SEO.”
We hear this more and more. And then it landed on our own desk: we ran diligently.pl through PageSpeed Insights and found a warning about it. So we checked. Not what SEO blogs say, but what the search engines have said on the record, and what crawl logs show.
The short answer
No major AI provider or search engine has committed to using llms.txt as a ranking or citation signal. Google has said the opposite.
First, a quick refresher. llms.txt is a plain Markdown file in the root of your site (yoursite.com/llms.txt). It lists your most important pages in a format that is easy for a language model to read. Jeremy Howard proposed it in 2024. The idea is simple and sounds sensible: give AI a tidy summary so it doesn’t have to dig through your HTML.
The problem is that it’s a proposal, not a standard. Nobody has to read it, and it looks like hardly anyone does.
Why PageSpeed Insights flagged us
On mobile, diligently.pl scores 97 for performance, 96 for accessibility, 96 for best practices and 100 for SEO. But there is a fifth score next to them: Agentic Browsing, 2 out of 3.
Lighthouse, the engine behind PageSpeed Insights, now has an Agentic Browsing category. One of its audits looks for an llms.txt file. Ours reported: “llms.txt does not follow recommendations. Fetch of llms.txt failed: Timed out fetching resource.” The simple reason: we don’t have one.
Does that mean Google wants one? Not quite. Look at how the category describes itself: “This category is still under development and subject to change.” The audit text is also careful: without a good file, models “may not be able to understand” how you want your site crawled. A tool that checks whether a file exists is not a search engine that uses it. Which brings us to what Google actually says.
What Google says
John Mueller, Search Advocate at Google, wrote on Bluesky on 17 June 2025:
“FWIW no AI system currently uses llms.txt.”
John Mueller, Google, via Search Engine Roundtable
He added that the chatbots do fetch your pages, but none of them fetch the llms.txt file.
A year later nothing had changed. In a Reddit thread reported by Search Engine Journal in June 2026, he called it “purely speculative for now.” His advice: if an AI platform that brings you clients says it needs the file, create one. Until then, there’s no reason to.
What the crawl logs show
Statements are one thing. Server logs are another. Three independent studies all point the same way:
- Ahrefs looked at 137,210 domains in May 2026. About 28% publish an llms.txt, and 97% of those files received zero requests that month. Their own conclusion: “If you publish an llms.txt file today, the most likely outcome by far is that nothing ever fetches it.” (Ahrefs)
- EZY.ai compared how often AI crawlers fetched robots.txt and llms.txt on 83 sites over 12 weeks (EZY.ai):
| Crawler | robots.txt fetches | llms.txt fetches |
|---|---|---|
| OpenAI | 3,990 | 7 |
| Anthropic (ClaudeBot) | 3,120 | 9 |
| PerplexityBot | 775 | 0 |
| Googlebot | 5,125 | 67 |
| Meta-ExternalAgent | 172 | 193 |
- SE Ranking analysed about 300,000 domains and found no relationship between having an llms.txt and how often a domain is cited in AI answers. As reported by Search Engine Journal, their model got more accurate when the llms.txt signal was removed.
A fair caveat: the EZY sample is small and skews toward small-business sites, and several of these studies come from companies that sell AI-visibility tools. SE Ranking’s result is a correlation, so it can’t prove the file has no effect. But there is no study showing that it works, either.
Who does read it
Not nobody. In Ahrefs’ data, the AI agents that fetched llms.txt most were GPTBot and Claude Code, ahead of every AI search bot. Coding agents are the natural audience: they read documentation, and a clean index of docs helps them.
That explains why Anthropic publishes an llms.txt for its developer documentation. It’s useful for people who build with an API. It’s a different thing from a signal for ranking a marketing website in ChatGPT or Google.
So if you run a developer platform with public docs, an llms.txt is reasonable. If you run a B2B company website, it’s a nice-to-have at best.
What to do instead
The honest answer is boring: the things that get you cited in AI answers are mostly the things that already got you ranked.
- Make sure AI can reach your content. Check that your robots.txt doesn’t block the crawlers you want, and that your key pages render as plain HTML, not only after JavaScript runs.
- Answer real questions directly. Clear headings, short answers up front, concrete details. AI summarises what is easy to summarise.
- Mark up what you are. Structured data (schema.org) tells machines what a page is about without guessing.
- Build real authority. Being mentioned and linked from credible places still matters, for Google and for AI.
- Measure it. Check whether your brand shows up in AI answers for the questions your clients ask, and track it over time.
None of this is a trick. That’s the point.
How we approach SEO for LLMs at Diligently
We don’t sell “GEO” as a separate magic product. AI search optimization is part of our SEO work, and it’s included in our monthly SEO plans (Startup, Advanced, Enterprise).
In practice, it looks like this:
- We start with your business, not a tool. We learn what you offer, who buys and what they ask before we touch a keyword.
- We audit the technical side. Crawlability, speed, mobile usability and structured data, because an AI can’t cite what it can’t read.
- We write for two readers. Content is optimized for Google rankings and for LLM visibility, including your existing pages and articles, not only new ones.
- We report monthly and adjust. SEO takes time, typically 3–6 months to show results, and we’d rather tell you that up front than promise a quick win.
And our own llms.txt? We don’t have one on diligently.pl, which is exactly why PageSpeed Insights flagged us. If you want one anyway, it’s a small job and we’re happy to set it up. Just don’t expect it to move your numbers.
Bottom line
llms.txt isn’t harmful. It just isn’t proven, and no one who matters has said they use it. Spend your time on the fundamentals first.
Not sure where your site stands in AI search? Schedule a call, and we’ll tell you honestly what’s worth doing.
Sources: John Mueller on Bluesky via Search Engine Roundtable (17 June 2025) and Search Engine Journal (June 2026). Studies: Ahrefs (June 2026), EZY.ai (April–July 2026), SE Ranking via Search Engine Journal. PageSpeed Insights report for diligently.pl, 8 October 2026. Figures as of October 2026.