SeoSpidie
See your site the way search engines do
Build a valid robots.txt file by choosing what search engines β and AI crawlers β can access. It updates live as you edit; copy it or download it and drop it in your site's root.
A robots.txt file is a small plain-text file that sits at the root of your domain, for example example.com/robots.txt. Before a well-behaved crawler such as Googlebot, Bingbot or GPTBot visits your pages, it reads this file to learn which parts of the site it is allowed to request. Each group of rules starts with a User-agent line naming the crawler, followed by Allow and Disallow lines that list URL paths.
Robots.txt controls crawling, not indexing. If you disallow a page, search engines will usually stop fetching it, but the URL can still appear in results when other sites link to it. To keep a page out of search results, let it be crawled and add a noindex tag instead. Robots.txt is best used to keep crawlers away from areas that waste their time: admin screens, internal search results, cart and checkout pages, and endless filter combinations.
The file also has a second job: a Sitemap line tells every crawler where your XML sitemap lives. In recent years robots.txt has become the main way site owners decide which AI crawlers may read their content, because companies such as OpenAI, Anthropic, Google and Perplexity publish separate user-agent names for their bots.
robots.txt is the first file well-behaved crawlers read. It tells them which areas to skip β admin pages, internal search, duplicate filters β and where your sitemap is. One wrong line can hide an entire site from search, so it pays to build it carefully.
The generated file is plain text. Each User-agent group applies to the crawler it names; a group for "*" applies to every crawler that has no group of its own. Within a group, the most specific matching rule wins, and Allow wins a tie.
After uploading, open yourdomain.com/robots.txt in a browser to confirm it is live, then check the robots.txt report in Google Search Console. If a page you care about later shows "Blocked by robots.txt" in Search Console, find the rule that matches its path and narrow it.
A furniture shop in Manchester noticed in Search Console that Google was crawling thousands of filter URLs β every combination of colour, material and price β while new products took weeks to appear. Its robots.txt contained only a sitemap line.
Using the generator, the owner kept the default "allow" rule, added disallowed paths for /cart/, /checkout/, /account/ and the ?filter= parameter, and added the sitemap URL. They allowed search-related AI crawlers but blocked training-only bots. After uploading the file and checking it at /robots.txt, crawl reports over the following weeks showed far fewer requests to filter pages and faster discovery of new products.
At the root of your domain, so it opens at yourdomain.com/robots.txt. Each subdomain needs its own file. A robots.txt placed in a subfolder is ignored.
Google usually refreshes its copy of robots.txt within about a day. You can check the version Google has read in the robots.txt report in Search Console.
Yes, for crawlers that respect it. AI companies publish user-agent names for their bots, and you can allow or disallow each one. It is a request, not access control, so truly private content needs a login.
