Robots.txt Generator
Pick a default policy, add rules per crawler, and the finished robots.txt updates live below.
No extra rules — only the default policy applies.
Options
User-agent: * Disallow:
What each line means
- User-agent — names the crawler the following rules apply to; * means every crawler without its own block.
- Disallow — asks that crawler not to fetch URLs starting with the given path. An empty value means nothing is blocked.
- Allow — re-permits a path inside a disallowed area, for example one public folder within a blocked section.
- Crawl-delay — asks the crawler to wait this many seconds between requests.
- Sitemap — tells search engines where your XML sitemap lives so pages are discovered faster.
robots.txt is advisory: well-behaved crawlers follow it, but it does not enforce access. Use authentication for anything private.
How it works
- Choose a default policy — allow all crawlers or block everything — or start from a preset such as “Block AI trainers”.
- Add rule rows: pick a crawler like Googlebot or GPTBot, choose Allow or Disallow, and enter the path, for example /admin/.
- Paste your sitemap URL if you have one, then press Copy or Download and upload the file to the root of your site as robots.txt.
Frequently asked questions
- What does a robots.txt file actually do?
- It tells crawlers which parts of your site they may request, using User-agent lines to name a bot and Allow or Disallow lines to set paths. The file must live at the root of your domain, for example example.com/robots.txt. Crawlers fetch it before crawling and follow the matching rules for their own user-agent.
- Does robots.txt guarantee that bots stay out?
- No — it is purely advisory. Reputable crawlers such as Googlebot, Bingbot and GPTBot honor it, but nothing technically prevents a bot from ignoring the file. For content that must stay private, use authentication or server-level blocking; robots.txt is a request, not an access control.
- How do I block AI crawlers from training on my content?
- Add Disallow: / rules for the user-agents used by AI companies: GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended (Google’s AI training signal), ClaudeBot (Anthropic) and Applebot-Extended (Apple). The “Block AI trainers” preset adds all of these in one click while leaving normal search engines untouched.
- Will blocking pages in robots.txt remove them from Google?
- Not reliably. Robots.txt stops crawling, but a page that is linked from elsewhere can still appear in results as a bare URL without a description. To keep a page out of the index, allow crawling and add a noindex robots meta tag instead, or protect the page with a login.
- Is my site information uploaded anywhere by this tool?
- No. The file is assembled entirely by JavaScript running in your browser; no paths, sitemap URLs or other details are sent to any server. You can build the file, copy it, and upload it to your own hosting without anything leaving your machine.
About this tool
This robots.txt generator builds a correctly formatted crawler policy file without you having to memorize the syntax. Choose whether your site is open or closed by default, then stack rule rows: each row pairs a user-agent — the wildcard *, Googlebot, Bingbot, or AI crawlers like GPTBot and CCBot — with an Allow or Disallow directive and a path. A sitemap line and an optional crawl-delay can be added, and the finished file updates live as you edit, ready to copy or download as robots.txt.
Everything runs client-side in your browser. The tool groups your rules by user-agent, writes one block per crawler, appends the Sitemap line at the end, and hands you plain text — no account, no upload, no server ever sees your site structure. That matters when the paths you are hiding are exactly the ones you would rather not paste into a random web form.
Common uses: keeping staging folders, admin panels and internal search results out of crawler traffic; publishing a sitemap location so search engines discover new pages faster; and opting out of AI training crawls. The “Block AI trainers” preset disallows GPTBot, CCBot, Google-Extended, ClaudeBot and Applebot-Extended in one step while leaving Google and Bing search crawling fully intact — a setup many publishers now ship by default.
Two practical notes. First, robots.txt is advisory: well-behaved bots follow it, badly behaved ones do not, so never rely on it to protect sensitive content — use authentication for that. Second, placement and casing matter: the file must be reachable at https://yourdomain.com/robots.txt exactly, paths are case-sensitive, and a stray Disallow: / under User-agent: * will deindex an entire site over time. Test your file in Google Search Console’s robots.txt report after uploading.