Robots.txt Generator
Build your robots.txt visually. Add rules, sitemaps, crawl-delay, and AI crawler opt-outs — then copy or download the file.
Paths start with / (for example /admin/ or /*.pdf$). Rules apply to all crawlers.
Opting out does not affect your Google or Bing search rankings. Compliance is voluntary for each crawler.
Blocking a search engine crawler stops it crawling your site, so your pages can drop out of its results.
Use the full URL, starting with https://
Related Tools
How to Use
- Choose Access & Rules — Choose whether crawlers may crawl the whole site (the default) or whether the whole site is blocked, for example on a staging site. Then add Allow and Disallow rules in the visual builder: each rule has a type dropdown and a path that must start with /. Click "+ Add Rule" for more.
- Set Options — Add the full URL of your XML sitemap, set an optional crawl-delay (1–60 seconds; Google ignores it), and tick any AI crawlers (such as GPTBot, ClaudeBot, CCBot, or Google-Extended) or search engine crawlers you want to keep out of your whole site.
- Copy or Download — The preview updates as you edit, and any rule or sitemap that cannot be used is flagged and left out. Copy the text or download robots.txt and upload it to the root of your website (example.com/robots.txt). Reset restores the default settings.
About the Robots.txt Generator
The Robots.txt Generator writes the small text file that tells crawlers which parts of a site they may fetch. You pick the default (crawl everything, or block the whole site, as on a staging copy), add Allow and Disallow paths, a crawl-delay and sitemap URLs, and tick any AI or search crawlers you want to keep out entirely. Site owners use it to keep crawlers away from admin areas, internal search results and cart pages, and to opt out of AI crawlers, without memorizing the syntax.
How the file is built
- Your rules go into one group for all crawlers (
User-agent: *). When the site is open, Disallow lines come first andAllow: /last; when it is blocked, your Allow exceptions come beforeDisallow: /. That order gives the same result for crawlers that use the longest matching rule (as Google does) and for older ones that use the first match. - Each crawler you tick gets its own group with
Disallow: /. - Paths must start with
/and cannot contain spaces or#; a crawl-delay must be a whole number from 1 to 60; sitemaps must be full URLs. Anything that fails is left out and listed under the preview. Duplicate lines are removed.
Worked example
We added Disallow /admin/, Disallow /search, Allow /admin/help/, a rule typed as checkout/, a crawl-delay of 10, a sitemap at https://example.com/sitemap.xml, a second sitemap typed as sitemap.xml, and ticked GPTBot and CCBot. Both mistyped entries were skipped with a message. The rest produced:
| Line | What it means |
|---|---|
User-agent: * | Rules for every crawler that has no group of its own |
Disallow: /admin/ | Skip every URL whose path starts with /admin/ |
Disallow: /search | A prefix match: also covers /search-tips and /searchable, not only the /search folder |
Allow: /admin/help/ | Longer than /admin/, so Google (which applies the longest matching rule) may crawl the help pages |
Allow: / | Everything else may be crawled (the default anyway) |
Crawl-delay: 10 | Asks for 10 seconds between requests; Google does not support this field |
User-agent: GPTBot + Disallow: / | GPTBot may not crawl anything (the same group follows for CCBot) |
Sitemap: https://example.com/sitemap.xml | Where to find your sitemap; it must be a full URL |
What robots.txt does not do
From Google Search Central, checked on 2026-10-14 (introduction, specification):
- It is not a way to keep a page out of Google. A blocked URL can still appear in results, without a description, if other pages link to it. To keep a page out, let Google crawl it and add a noindex rule, or put it behind a password.
- It is not security. Crawlers decide whether to obey it. The file is also public, so listing a secret folder in it tells everyone where that folder is.
- Groups are not combined. A crawler follows only the most specific group that matches its name, so GPTBot above ignores the
*rules entirely. - Scope is one host. The file must sit at the top level of the site, and each subdomain needs its own. Paths are case-sensitive, and Google reads at most 500 KiB of the file.
Limitations of the generator
- It builds one shared rule group plus whole-site blocks for the crawlers you tick; it cannot give a single crawler its own path rules.
- Wildcards such as
*and$are passed through as typed, without checking. - Crawler names change as companies launch new bots. The list covers well-known ones, and compliance is voluntary; use server or firewall rules if you need enforcement.
Practical tips
- End folder paths with a slash (
/search/) unless you mean to block every path that starts with those letters. - Do not block CSS, JavaScript or images that your pages need; Google says blocking such files makes it harder to analyze the pages that depend on them.
- To remove a page from search results, use a noindex robots meta tag from the Meta Tag Generator and leave that page crawlable here.
- After uploading, open yourdomain.com/robots.txt in a browser to confirm the live file. The robots.txt guide walks through common setups.
Privacy: the file is generated in your browser, and your paths and settings are not sent anywhere until you upload the file to your own server.
Frequently Asked Questions
What is robots.txt?
Robots.txt is a plain text file at the root of your website (example.com/robots.txt) that tells crawlers which paths they may or may not crawl. It uses User-agent (which crawler a group of rules applies to), Allow, and Disallow directives. Robots.txt is advisory: reputable crawlers such as Googlebot and Bingbot follow it, but it is not access control and some crawlers ignore it. It also controls crawling, not indexing: a disallowed URL can still appear in search results if other pages link to it. Protect private content with a password, and use a noindex meta tag on a crawlable page to keep it out of search results.
What does Allow and Disallow mean?
Disallow asks crawlers not to fetch a path: "Disallow: /admin/" covers every URL that starts with /admin/. Allow explicitly permits a path, which is useful for opening a subfolder inside a disallowed folder. Google and Bing apply the most specific (longest) matching rule, so "Allow: /" together with "Disallow: /admin/" still keeps crawlers out of /admin/. The generator also writes specific rules before "Allow: /" so older crawlers that use the first matching rule behave the same way.
What is crawl-delay?
Crawl-delay asks a crawler to wait a number of seconds between requests: "Crawl-delay: 10" means one request every 10 seconds. Bing and some other crawlers respect it; Google ignores it and adjusts its crawl rate automatically based on how your server responds. The generator accepts whole numbers from 1 to 60.
Should I block AI crawlers?
That depends on your preference. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), and meta-externalagent (Meta) collect web content for AI training or AI products. Google-Extended and Applebot-Extended are not separate crawlers but control tokens: they tell Google and Apple whether content their regular crawlers fetch may be used for AI models such as Gemini, without affecting Google Search or Apple search features. PerplexityBot indexes pages for Perplexity's AI search, so blocking it can remove your site from Perplexity answers. Blocking AI crawlers does not affect your Google or Bing search rankings. Compliance is voluntary, so use server or firewall rules if you need enforcement.
Why is Bingbot listed under search engines?
Bingbot is Microsoft's search engine crawler, not an AI training crawler. Blocking it removes your pages from Bing crawling, which also affects search results that rely on Bing's index. That is why search engine crawlers are listed separately and are never blocked by default.
Where do I upload robots.txt?
Upload the robots.txt file to the root directory of your website so it is reachable at yourdomain.com/robots.txt. On most web hosts this is the public_html or www folder. The file must be named exactly "robots.txt" (lowercase) and be plain text. Each subdomain (for example blog.yourdomain.com) needs its own robots.txt.
Is my configuration sent to a server?
No. The robots.txt is generated entirely in your browser using JavaScript. Your configuration and paths are not transmitted anywhere. Download the file and upload it to your own server manually.