How to Create a Robots.txt File With a Free Generator
Generate a correct robots.txt file without memorising syntax, then upload and test it so you never block pages by accident.
Writing robots.txt by hand is simple until it is not. Group order, trailing slashes and wildcard rules all affect what crawlers do, and a typo is silent: nothing breaks visibly, your pages just stop getting crawled. A generator removes that risk by producing clean, valid syntax from a few choices.
Here is how to create the file with the free SeoSpidie robots.txt generator, upload it and confirm it does what you intended. For background on the rules themselves, see robots.txt explained with examples.
Step 1: Decide what should stay out of search
Write a short list before opening any tool. Typical candidates on a small business or blog site:
- Admin or login areas (
/admin/,/wp-admin/) - Internal site search results (
/searchor?q=URLs) - Cart, checkout and account pages on a shop
- Endless filter or sort combinations that create thousands of near-identical URLs
Things that should not be on the list: your CSS and JavaScript folders, your images (unless you have a reason), and any page you want to rank.
Step 2: Set the default rule for all crawlers
In the generator, start with the group for every bot (User-agent: *). Add a disallow line for each path on your list. Keep paths as short as possible โ /admin/ already covers everything inside it, so there is no need to list sub-folders.
Step 3: Add rules for specific bots if you need them
You might want to treat certain crawlers differently, for example AI training bots. Remember that a bot that has its own group ignores the general * group, so repeat any shared rules inside its group too. If you are unsure which AI crawlers exist and what they do, the AI crawler checker lists the main ones and shows how your site treats them.
Step 4: Include your sitemap
Add your sitemap address so every crawler can find it, e.g. Sitemap: https://example.com/sitemap.xml. It must be a full URL including https://. No sitemap yet? Make one with the free sitemap generator first.
Step 5: Upload to the root folder
Copy the generated text into a plain text file named robots.txt (all lowercase) and upload it to the top folder of your site โ on cPanel hosting that is usually public_html. Save it with UTF-8 encoding and do not use a word processor, which can add hidden formatting. Open https://yourdomain.com/robots.txt to confirm it is live.
Step 6: Test before you relax
Pick five to ten of your most important pages โ homepage, top services, best-selling products, main blog posts โ and run them through the bulk indexability checker. Each should be reported as crawlable. Then test one URL you intentionally blocked to confirm the rule works. In Google Search Console, the robots.txt report shows when Google last fetched the file and whether it found errors.
A sensible starter file
If you run a typical small website and are unsure, a safe, minimal starting point is: one group for all crawlers that blocks only your admin folder and your internal search page, followed by your sitemap address. The generator builds exactly this for you.
Replace the paths with your real ones. It is better to block too little than too much; you can always add rules later when you see crawlers wasting time on low-value URLs.
Related guides
More free tools: browse all our free SEO tools, no sign-up needed.
Frequently asked questions
Will robots.txt hide a page from Google search results?
Not reliably. It stops crawling, but a blocked URL can still appear if other sites link to it. Use a noindex tag for pages that must stay out of results.
Does WordPress already create a robots.txt file?
WordPress serves a virtual one if no real file exists. Uploading your own robots.txt to the root replaces it.
Can a wrong robots.txt hurt my rankings?
Yes. Blocking important pages or the CSS and JavaScript your pages need can lead to pages dropping out of search or being misjudged. Always test after changes.
How often should I review the file?
After every site migration, redesign or CMS change, and otherwise once or twice a year.