LLMs.txt: Hype vs. Reality
LLMs.txt is a proposed Markdown standard for AI bot crawling that works similarly to robots.txt but targets LLMs like GPTBot, ClaudeBot, and PerplexityBot.
Despite the hype around greater LLM visibility and referral traffic, data analysis of 296K domains shows adoption remains minimal and provides no measurable benefit.
When segmented by organic traffic levels, the share of sites using LLMs.txt stayed consistent across all segments, revealing no correlation with improved visibility or citations.
The takeaway is clear: LLMs.txt isn't where SEO energy should be invested right now, and focusing on technically sound robots.txt configuration remains the smarter priority.
LLMs.txt - a proposed standard for website crawling by AI bots… or just hype with no official adoption yet? We dug into the data to find out. Long story short, LLMs.txt is a Markdown file that popped up last year. In theory, it works like robots.txt - but this time, built specifically for LLMs. You add it on your site to give AI crawlers like GPTBot, ClaudeBot, and PerplexityBot - a curated list of content to read and cite. The hype comes from the promise of more control over LLM visibility and maybe even referral traffic. But our data says otherwise. Out of 296K domains analyzed, only about 10% had an LLMs.txt file - 90% didn’t. And when we segmented sites by organic traffic - from under 100 visits to over 100K - the share stayed the same across all segments. This distribution lets us test whether LLMs.txt makes a difference. And
the result you see from our machine learning data analysis - it isn’t giving sites better visibility or more citations. This insight is just one piece of our full deep dive on what really drives LLM rankings – dropping soon on the blog. Stay tuned! As of today, it’s safe to say LLMs.txt isn’t where you should put your energy asap. The smarter move is to focus on what already impacts visibility: robots.txt. SE Ranking’s Website Audit makes that part simple. You can check robots.txt for issues, monitor crawler access, and keep your site technically sound. And feel free to subscribe to catch our next findings.