Skip to content

Robots.txt Generator

Build your robots.txt visually. Add rules, sitemaps, crawl-delay, and AI crawler opt-outs — then copy or download the file.

CUSTOM RULES

Paths start with / (for example /admin/ or /*.pdf$). Rules apply to all crawlers.

BLOCK AI CRAWLERS

Opting out does not affect your Google or Bing search rankings. Compliance is voluntary for each crawler.

BLOCK SEARCH ENGINE CRAWLERS (rarely needed)

Blocking a search engine crawler stops it crawling your site, so your pages can drop out of its results.

SITEMAP URLs

Use the full URL, starting with https://

Related Tools

Headline Analyzer SEO Content Analyzer JSON Formatter & Validator Meta Tag Generator Schema Markup Generator

How to Use

  1. Choose Access & Rules — Choose whether crawlers may crawl the whole site (the default) or whether the whole site is blocked, for example on a staging site. Then add Allow and Disallow rules in the visual builder: each rule has a type dropdown and a path that must start with /. Click "+ Add Rule" for more.
  2. Set Options — Add the full URL of your XML sitemap, set an optional crawl-delay (1–60 seconds; Google ignores it), and tick any AI crawlers (such as GPTBot, ClaudeBot, CCBot, or Google-Extended) or search engine crawlers you want to keep out of your whole site.
  3. Copy or Download — The preview updates as you edit, and any rule or sitemap that cannot be used is flagged and left out. Copy the text or download robots.txt and upload it to the root of your website (example.com/robots.txt). Reset restores the default settings.

About the Robots.txt Generator

The Robots.txt Generator writes the small text file that tells crawlers which parts of a site they may fetch. You pick the default (crawl everything, or block the whole site, as on a staging copy), add Allow and Disallow paths, a crawl-delay and sitemap URLs, and tick any AI or search crawlers you want to keep out entirely. Site owners use it to keep crawlers away from admin areas, internal search results and cart pages, and to opt out of AI crawlers, without memorizing the syntax.

How the file is built

Worked example

We added Disallow /admin/, Disallow /search, Allow /admin/help/, a rule typed as checkout/, a crawl-delay of 10, a sitemap at https://example.com/sitemap.xml, a second sitemap typed as sitemap.xml, and ticked GPTBot and CCBot. Both mistyped entries were skipped with a message. The rest produced:

LineWhat it means
User-agent: *Rules for every crawler that has no group of its own
Disallow: /admin/Skip every URL whose path starts with /admin/
Disallow: /searchA prefix match: also covers /search-tips and /searchable, not only the /search folder
Allow: /admin/help/Longer than /admin/, so Google (which applies the longest matching rule) may crawl the help pages
Allow: /Everything else may be crawled (the default anyway)
Crawl-delay: 10Asks for 10 seconds between requests; Google does not support this field
User-agent: GPTBot + Disallow: /GPTBot may not crawl anything (the same group follows for CCBot)
Sitemap: https://example.com/sitemap.xmlWhere to find your sitemap; it must be a full URL

What robots.txt does not do

From Google Search Central, checked on 2026-10-14 (introduction, specification):

Limitations of the generator

Practical tips

Privacy: the file is generated in your browser, and your paths and settings are not sent anywhere until you upload the file to your own server.

Frequently Asked Questions

What is robots.txt?

Robots.txt is a plain text file at the root of your website (example.com/robots.txt) that tells crawlers which paths they may or may not crawl. It uses User-agent (which crawler a group of rules applies to), Allow, and Disallow directives. Robots.txt is advisory: reputable crawlers such as Googlebot and Bingbot follow it, but it is not access control and some crawlers ignore it. It also controls crawling, not indexing: a disallowed URL can still appear in search results if other pages link to it. Protect private content with a password, and use a noindex meta tag on a crawlable page to keep it out of search results.

What does Allow and Disallow mean?

Disallow asks crawlers not to fetch a path: "Disallow: /admin/" covers every URL that starts with /admin/. Allow explicitly permits a path, which is useful for opening a subfolder inside a disallowed folder. Google and Bing apply the most specific (longest) matching rule, so "Allow: /" together with "Disallow: /admin/" still keeps crawlers out of /admin/. The generator also writes specific rules before "Allow: /" so older crawlers that use the first matching rule behave the same way.

What is crawl-delay?

Crawl-delay asks a crawler to wait a number of seconds between requests: "Crawl-delay: 10" means one request every 10 seconds. Bing and some other crawlers respect it; Google ignores it and adjusts its crawl rate automatically based on how your server responds. The generator accepts whole numbers from 1 to 60.

Should I block AI crawlers?

That depends on your preference. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), and meta-externalagent (Meta) collect web content for AI training or AI products. Google-Extended and Applebot-Extended are not separate crawlers but control tokens: they tell Google and Apple whether content their regular crawlers fetch may be used for AI models such as Gemini, without affecting Google Search or Apple search features. PerplexityBot indexes pages for Perplexity's AI search, so blocking it can remove your site from Perplexity answers. Blocking AI crawlers does not affect your Google or Bing search rankings. Compliance is voluntary, so use server or firewall rules if you need enforcement.

Why is Bingbot listed under search engines?

Bingbot is Microsoft's search engine crawler, not an AI training crawler. Blocking it removes your pages from Bing crawling, which also affects search results that rely on Bing's index. That is why search engine crawlers are listed separately and are never blocked by default.

Where do I upload robots.txt?

Upload the robots.txt file to the root directory of your website so it is reachable at yourdomain.com/robots.txt. On most web hosts this is the public_html or www folder. The file must be named exactly "robots.txt" (lowercase) and be plain text. Each subdomain (for example blog.yourdomain.com) needs its own robots.txt.

Is my configuration sent to a server?

No. The robots.txt is generated entirely in your browser using JavaScript. Your configuration and paths are not transmitted anywhere. Download the file and upload it to your own server manually.