Sitemaps, llms.txt and data files: what's worth adding to your website
There's a lot of advice about files that help search engines and AI tools read your site. Here's what each one does, which ones matter, and which are optional.
Website owners hear a lot of advice about files that help search engines and AI tools: robots.txt, sitemaps, structured data, IndexNow, llms.txt. Some matter a lot. Some are optional. Here's a plain guide.
robots.txt: make sure you aren't blocking anyone
robots.txt is a small file at yourwebsite.com/robots.txt that tells crawlers what they may visit. The main risk is blocking something by mistake: a rule left over from when the site was being built, or a security setting that blocks unfamiliar bots. If you want to appear in AI search tools, check that their search crawlers are allowed. OpenAI, for example, lists its crawlers and what each one does (OpenAI's crawler documentation), so you can allow its search crawler while blocking the one used for training.
Sitemaps: worth having
A sitemap lists the pages you want search engines to find, usually at /sitemap.xml. Most website builders make one for you. Two details matter:
- Include only real, public pages you want found, not drafts or duplicates.
- Keep the last-modified dates accurate. Google uses them when they're consistently accurate, and ignores the priority and change frequency fields (Google's sitemap guide).
Submit your sitemap in Google Search Console and in Bing Webmaster Tools.
Structured data: worth having
Structured data describes your business, products and articles in a standard format, and it must match what's on the page. Our plain-English guide to structured data explains it.
IndexNow: useful for Bing and others
IndexNow lets a site tell participating search engines, including Microsoft Bing, the moment a page is added or changed, instead of waiting for them to visit. Many website platforms support it with a setting or a plugin. Google doesn't use it, so keep your sitemap too.
llms.txt: optional
llms.txt is a proposed file that gives AI tools a short, readable summary of a site with links to its key pages. It's easy to add and may help some tools, but it's a proposal, not something every assistant reads. Google says you don't need new machine-readable files or AI text files to appear in its AI features (Google's guide to AI features). Add one if it's easy, but fix the basics first.
Data files: helpful when the facts are clean
Some sites also publish their key facts as data, such as a JSON or Markdown version of a business profile, so tools can read them without working through a whole page. Every VerifiedHive business page has these files, made from the same details as the page, so the two never disagree.
What matters most, in order
- Your important pages exist, load quickly and state facts in text.
- Nothing blocks the crawlers you want.
- A sitemap lists your real pages with honest dates.
- Structured data matches each page.
- Then the extras: IndexNow and llms.txt.
If you don't have the time to set all this up, a free VerifiedHive business page comes with structured data and data files already in place, and it's listed in our sitemap once it has a launch or the Verified badge.