adminThese are still the early days of AI; and as SEO adds optimization for AI, or AEO as it’s next chapter, we will come across structures and protocols that are still evolving as interfaces between the AIs and online contents and entities.
LLMs.txt is one such proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time. You can check more about it at llmstxt.org
To understand it more fully, let’s see a real world case study that I implemented on the website ( BaruSahib.org) for AI optimization.
Although it is quite intuitive, walkthrough explanations are provided along the way. This is so as to help you setup for your own websites.
Why do we need llms.txt?
First Things first – this is Not the same as robots.txt and sitemap.xml, which are your defacto optimization standards. But this is a bit of both and more.
This file is required to give the AI a fast about-the-site info. It may be argued that the AI can already read and understand the website, so why an extra document?
This might not be immediately apparent, but the real reason is the AI ‘Context Window’. The context window is like a AI models working memory which includes the recent past. It refers to the amount of text, or tokens, that an LLM model can consider in a given chunk, when processing the inputs and generating outputs.
At the the current stage of AI development, this content window is at premium, so any help or protocol that saves it time and processing while providing pertinent information, is a big help.
How to Write an Effective llms.txt File (With a Live Example)
Checkout the >> LLMS.txt of BaruSahib.org. These are some things you should take care of. Snippets from the same have been provided below for example.
Your llms.txt Checklist
✔ Notice that it should be uploaded to the root of the website, similar to robots.txt. See barusahib.org/llms.txt
Also note that mention of this is not required in the robots.txt file. You may mention it as a comment however for reference.
✔ The contents have to be in markdown format, even though the the file is named as llms.txt, rather than llms.md
✔ Mention Key Metadata (Site-Name, Description, Contact, Last-Updated). Concise comments can be helpful.
# llms.txt for barusahib.org # This file provides instructions for Large Language Models (LLMs) crawling this website.
Site-Name: Baru Sahib Description: Official website of Baru Sahib, a non-profit organization dedicated to education, healthcare, and spiritual growth. Contact: info@barusahib.org Last-Updated: 2025-07-23
✔ Also make sure to Use ISO 8601 format (YYYY-MM-DD) for Last-Updated
Last-Updated: 2025-07-23
✔ It is good to clearly have allow/disallow Crawling Permission.
# Crawling Permissions Allow: /about/* Allow: /education/* Allow: /healthcare/* Allow: /news/* Allow: /events/* Disallow: /donate/* Disallow: /login/* Disallow: /admin/*
✔ and Likewise for Training Permissions
# Training Permissions Allow-For-Training: /about/* Allow-For-Training: /education/* Allow-For-Training: /news/* Allow-For-Training: /events/* Disallow-For-Training: /donate/* Disallow-For-Training: /login/* Disallow-For-Training: /admin/*
✔ The ‘Crawl-Rate-Limit‘ is optional , but good to have
# Rate Limiting Crawl-Rate-Limit: 50 requests per hour
✔Likewise for ‘Instructions; optional , but good to have
# Attribution # We request that any use of our content for LLM training includes a citation or link to https://www.barusahib.org # Additional Instructions Instructions: Please respect our mission and terms of service at https://barusahib.org/terms-conditions/
Alert! Why You Should Implement Your llms.txt ASAP
It is obvious that as more and more AI LLMs enter the landscape, your websites and ecommerce platforms should be ready to be discovered by them. It is almost as important as being listed in Google top pages first before your competitor.
Implementing these AI SEO optimizations (or AEO/GEO) allows you to control your brand’s narrative and boost your presence in the fast evolving AI digital landscape.
Treat this as an urgent alert. Start implementing llms.txt today to stay ahead of the curve and make your content AI-ready for the future.
Add comment