How to create robots.txt
- Choose a starting point: allow everything, recommended WordPress rules, block all crawlers (for staging sites), or allow search while blocking AI training crawlers.
- Edit groups: set the user-agent (use
*for all crawlers), add Allow or Disallow rules with paths starting with/, and an optional crawl delay. - Add your sitemap URL.
- Review the generated file and any warnings, then copy or download it and upload it to your site’s root as
/robots.txt.
Understanding robots.txt rules
robots.txt is a plain text file that tells web crawlers which parts of your site they may visit. Rules are grouped by User-agent, the crawler’s name. Disallow asks the crawler not to fetch matching paths, Allow makes an exception inside a disallowed folder, and an empty Disallow: means everything is allowed. The Sitemap line points crawlers to your list of pages.
This robots.txt generator checks that paths start with / or * and that sitemap addresses are full URLs. It also warns loudly if your rules would block all crawlers from the entire site, a common and costly mistake.
Review before publishing. robots.txt is public and is only a request: well-behaved crawlers follow it, but it doesn’t secure private pages. Use passwords or a noindex tag for pages that must not appear in search.
Common rules
- Block admin areas:
Disallow: /wp-admin/ - Block internal search results:
Disallow: /?s= - Block a single bot: a group with
User-agent: GPTBotandDisallow: / - Point to your sitemap:
Sitemap: https://example.com/sitemap.xml
Robots.txt generator FAQ
Where does robots.txt go?
In the root of your domain, so it’s available at https://yourdomain.com/robots.txt. WordPress and many SEO plugins also let you edit it from the dashboard.
Will Disallow remove a page from Google?
Not necessarily. A blocked page can still appear if other sites link to it. Use a noindex tag instead to keep a page out of search results.
Does Google support Crawl-delay?
No, Google ignores it, though some other search engines respect it.