Robots.txt Generator
Build your robots.txt visually. Add rules, sitemaps, crawl-delay, and AI bot blocking — then download the file.
Related Tools
How to Use
- Configure Rules — Add Allow and Disallow rules using the visual builder. Each rule has a type dropdown (Allow/Disallow) and a path input. Click "+ Add Rule" for more. Toggle "Allow all" to let crawlers access your entire site.
- Set Options — Add sitemap URLs so search engines can find your sitemap. Set an optional crawl-delay. Check the AI bot blocking checkboxes to prevent AI training crawlers (GPTBot, CCBot, Google-Extended) from indexing your content.
- Generate & Download — Click Generate to see the robots.txt output. Copy to clipboard or download as a robots.txt file. Upload the file to the root of your website (example.com/robots.txt).
About the Robots.txt Generator
WritePadPro's Robots.txt Generator provides a visual interface for creating robots.txt files without memorizing the syntax. Add rules with dropdowns and inputs, toggle AI bot blocking with checkboxes, and add sitemap URLs — then download the ready-to-upload file.
What robots.txt Controls
- Crawler access — Allow or disallow specific paths for search engine crawlers
- Sitemap discovery — Point crawlers to your XML sitemap for more efficient indexing
- Crawl rate — Set crawl-delay to prevent server overload from aggressive crawlers
- AI bot blocking — Prevent AI training crawlers from using your content
Common Use Cases
- Block admin pages — Disallow /admin/, /wp-admin/, /dashboard/ to prevent crawling of backend interfaces
- Block search results — Disallow /search/ to prevent crawling of internal search result pages (which create duplicate content)
- Block staging — Disallow / on staging sites to prevent premature indexing
- Block assets — Disallow /private/ or /internal/ for directories that should not appear in search
- Block AI crawlers — Disallow / for GPTBot, CCBot, Google-Extended to opt out of AI training
AI Bot Blocking
Several AI companies send crawlers to websites to gather training data for large language models. This generator includes one-click toggles for the most common AI crawlers:
- GPTBot — OpenAI's crawler for ChatGPT training data
- CCBot — Common Crawl's crawler used by multiple AI companies
- Google-Extended — Google's crawler for Gemini AI training (separate from Googlebot search)
Blocking these crawlers does NOT affect your search engine rankings — they are separate from the search crawlers (Googlebot, Bingbot) that index your pages for search results.
Syntax Reference
The robots.txt format uses these directives: User-agent: specifies which crawler the rules apply to (* means all crawlers), Allow: permits access to a path, Disallow: blocks access to a path, Crawl-delay: sets seconds between requests, and Sitemap: provides the full URL to your XML sitemap. Each directive must be on its own line.
Privacy
The generator runs entirely in your browser. Your website paths and configuration are not sent to any server. Download the generated file and upload it to your own web server.
Frequently Asked Questions
What is robots.txt?
Robots.txt is a plain text file placed at the root of your website (example.com/robots.txt) that tells search engine crawlers which pages they are allowed or not allowed to access. It uses a simple format with User-agent (which crawler the rule applies to), Allow (pages to crawl), and Disallow (pages to skip) directives. While crawlers are not required to obey robots.txt, all major search engines (Google, Bing, Yahoo) respect it.
What does Allow and Disallow mean?
Disallow tells crawlers not to access a specific path. "Disallow: /admin/" means crawlers should not visit any URL starting with /admin/. Allow explicitly permits access to a path, which is useful when a parent directory is disallowed but you want to allow a specific subdirectory. "Allow: /" means the entire site is open for crawling.
What is crawl-delay?
Crawl-delay tells crawlers to wait a specified number of seconds between requests. "Crawl-delay: 10" means wait 10 seconds between each page request. This is useful for servers with limited resources. Google ignores crawl-delay (use Google Search Console instead), but Bing and some other crawlers respect it.
Should I block AI training bots?
That depends on your preference. GPTBot (OpenAI), CCBot (Common Crawl), and Google-Extended (Gemini training) crawl websites to gather training data for AI models. Blocking them prevents your content from being used for AI training but does not affect your search rankings. These are separate crawlers from Googlebot (search indexing), which you should almost never block.
Where do I upload robots.txt?
Upload the robots.txt file to the root directory of your website so it is accessible at yourdomain.com/robots.txt. On most web hosts, this means placing it in the public_html or www directory. The file must be named exactly "robots.txt" (lowercase) and must be a plain text file.
Is my configuration sent to a server?
No. The robots.txt is generated entirely in your browser using JavaScript. Your configuration and paths are not transmitted anywhere. Download the file and upload it to your own server manually.