Skip to content

Robots.txt: What It Does and How to Write One

What Is robots.txt?

robots.txt is a plain text file placed at the root of a website (example.com/robots.txt) that tells search engine crawlers which pages they are allowed to access and which they should skip. It is the first file a well-behaved crawler checks before scanning any other page on your site.

Generate a properly formatted robots.txt with WritePadPro's Robots.txt Generator — select your rules, specify allowed and blocked paths, and get a valid robots.txt file ready to upload to your server.

The Robots Exclusion Protocol was proposed in 1994 by Martijn Koster as a simple convention for website owners to communicate with web crawlers. It was a de facto standard for decades until the IETF published it as RFC 9309 in 2022. Crawlers are still not required to obey it — but all major search engines (Google, Bing, Yahoo, Yandex, Baidu) respect robots.txt directives. Malicious bots and scrapers typically ignore it.

How Search Engine Crawling Works

Understanding robots.txt requires understanding how crawling works.

The Crawl Process

  1. Discovery: The crawler learns about your site from links on other sites, sitemaps, or previous crawls
  2. robots.txt check: Before crawling any page, the crawler fetches your robots.txt to check if the page is allowed
  3. Crawling: If allowed, the crawler downloads the page HTML and processes it
  4. Indexing: If the page has no noindex directive, it enters the search index
  5. Ranking: When a user searches, indexed pages compete for ranking based on relevance and quality

Crawl Budget

Every site has a "crawl budget" — the number of pages Googlebot will crawl in a given time period. For small sites (under 10,000 pages), crawl budget is rarely a concern. For large sites (100,000+ pages), robots.txt helps focus the crawl budget on important pages by blocking crawling of low-value pages (filtered search results, admin areas, duplicate content). For meta-level page control, see our Meta Tags Guide which covers robots meta tags for page-level indexing control.

robots.txt Syntax

The Four Directives

DirectiveSyntaxMeaning
User-agentUser-agent: GooglebotSpecifies which crawler the following rules apply to. Use * for all crawlers.
DisallowDisallow: /admin/Block crawling of the specified path. Empty value means allow all.
AllowAllow: /admin/public/Override a Disallow for a specific sub-path. Supported by all major search engines and part of RFC 9309.
SitemapSitemap: https://example.com/sitemap.xmlPoints crawlers to your XML sitemap. Can appear multiple times.

Additional Directives

DirectiveSupportMeaning
Crawl-delayBing and some other crawlers (NOT Google)Request a delay (in seconds) between successive crawl requests
HostYandex only (legacy; a 301 redirect is the usual way to set the preferred domain)Specify the preferred domain version (with/without www)

Pattern Matching

  • * matches any sequence of characters: Disallow: /*.pdf$ blocks all PDF files
  • $ matches end of URL: Disallow: /page$ blocks /page but not /page/subpage

Common robots.txt Rules

Block Admin Areas

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /dashboard/

Block Search Results and Filters

Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=

Block Duplicate Content

Disallow: /tag/
Disallow: /author/
Disallow: /*?page=

Allow CSS and JavaScript (Important!)

Allow: /assets/css/
Allow: /assets/js/

Google needs to access CSS and JS to render pages correctly. Blocking them causes rendering issues and may hurt rankings.

Block AI Training Bots

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Many website owners block AI training crawlers to prevent their content from being used to train large language models. Others allow them so their pages can appear in AI assistant answers. WritePadPro's own robots.txt allows them, so the choice depends on your goals. For schema markup alongside robots.txt, see our Schema Markup Guide.

Include Sitemap Reference

Sitemap: https://writepadpro.com/sitemap.xml

Always include a Sitemap directive. It helps crawlers discover all pages efficiently, especially new content and deep pages that may not be linked prominently.

robots.txt vs Meta Robots vs X-Robots-Tag

Three mechanisms control search engine crawling and indexing. Each operates at a different level.

MechanismLevelControlsBest For
robots.txtSite-level (file)Crawling (which URLs the crawler visits)Blocking entire directories, saving crawl budget
Meta robotsPage-level (HTML)Indexing (whether the page appears in results)Preventing specific pages from appearing in search
X-Robots-TagHTTP headerIndexing (like meta robots but for non-HTML files)Preventing PDFs, images, or API responses from indexing

The Critical Distinction

robots.txt blocks crawling. If you Disallow a page, the crawler never visits it — but the page CAN still appear in search results if other pages link to it (Google may show the URL with little or no description).

Meta robots blocks indexing. The crawler visits the page but obeys the noindex directive and does not add it to the search index. The page will NOT appear in search results.

To fully prevent a page from appearing in search: use meta robots noindex (not robots.txt Disallow). If you block a page in robots.txt, the crawler cannot see the noindex tag because it never fetches the page.

Common Mistakes

1. Accidentally Blocking the Entire Site

# WRONG: blocks everything!
User-agent: *
Disallow: /

This rule blocks all crawlers from all pages. It happens during site migrations and staging deployments when the staging robots.txt is not updated before going live. Always verify your robots.txt after deployment.

2. Blocking CSS/JS Files

Blocking /assets/ or /wp-content/ prevents Google from rendering your pages correctly. Google needs CSS and JavaScript to understand page layout and content. Allow all asset directories.

3. Using robots.txt for Security

robots.txt is public — anyone can read it at example.com/robots.txt. Listing secret admin URLs in Disallow directives actually reveals their existence. For security, use authentication (login required), not robots.txt. For sensitive directories, use server-level access control (.htaccess Deny, firewall rules).

4. Conflicting Disallow and noindex

If a page is blocked in robots.txt, the crawler cannot see a noindex meta tag on that page. The page may still appear in search results (with limited information from external links). To truly prevent indexing, allow crawling and use noindex.

Using WritePadPro's Robots.txt Generator

WritePadPro's Robots.txt Generator builds a robots.txt file from a visual form.

Step 1: Configure Rules

Open the Robots.txt Generator. Choose the default for all crawlers (User-agent: *): allow crawling, or block the whole site, for example on a staging site. Then add Allow and Disallow rules; each rule has a type dropdown and a path that must start with /. Click + Add Rule for more.

Step 2: Add Sitemap and Options

Enter the full URL of your XML sitemap (click + Add Sitemap for more than one). The generator adds the Sitemap directives at the end of the file. You can also set an optional crawl-delay (1-60 seconds; Google ignores it) and tick AI crawlers such as GPTBot, ClaudeBot, CCBot, or Google-Extended, or search engine crawlers, that you want to keep out of the whole site.

Step 3: Generate

Click Generate robots.txt; the preview also updates as you edit. Any rule or sitemap that cannot be used is flagged and left out of the file.

Step 4: Deploy

Click Copy, or Download robots.txt, and place the file at the root of your domain (accessible at example.com/robots.txt). Upload via FTP, file manager, or your hosting control panel.

Privacy

All generation runs locally in your browser. Your site structure information is never transmitted to any server.

Summary

robots.txt is a text file at your domain root that controls which pages search engine crawlers can access. It uses four directives: User-agent (which crawler), Disallow (block path), Allow (override Disallow), and Sitemap (point to XML sitemap).

Common rules: block admin areas, search/filter pages, and duplicate content paths. Always allow CSS/JS directories. Include a Sitemap reference. Block AI training bots if desired. Remember: robots.txt controls crawling, not indexing — use meta robots noindex to prevent pages from appearing in search results.

Generate your robots.txt with WritePadPro's Robots.txt Generator — select rules, add your sitemap, and get a valid file ready to deploy. For page-level meta tags, see our Meta Tags Guide.

Frequently Asked Questions

Does robots.txt block pages from appearing in search results?

Not reliably. robots.txt blocks crawling — the crawler does not visit the page. But Google can still show the URL in search results if other pages link to it, usually with little or no description. To truly prevent a page from appearing in search results, allow crawling and add a noindex meta tag. This lets the crawler visit the page, see the noindex directive, and exclude it from the index. Blocking in robots.txt AND using noindex is counterproductive because the crawler cannot see the noindex tag on a blocked page.

Where do I put my robots.txt file?

At the root of your domain, accessible at example.com/robots.txt. It must be at the root — robots.txt in a subdirectory (example.com/blog/robots.txt) is ignored by crawlers. For subdomains, each subdomain needs its own robots.txt: blog.example.com/robots.txt is separate from example.com/robots.txt. Upload the file via FTP, your hosting file manager, or your CMS's file management interface. After uploading, verify it is accessible by visiting your-domain.com/robots.txt in a browser.

Should I block AI crawlers like GPTBot?

This is a business decision, not a technical one. Blocking AI crawlers (GPTBot for OpenAI, CCBot for Common Crawl, ClaudeBot for Anthropic) asks them not to collect your content for training large language models; compliance is voluntary. Reasons to block: you want to control how your content is used, you are concerned about AI-generated content competing with your original work. Reasons to allow: AI assistants and AI search tools may cite and link to your content, and blocking their crawlers can reduce that visibility. Blocking AI crawlers does not affect your Google or Bing search rankings. Many publishers block AI training bots while allowing regular search crawlers. WritePadPro's own robots.txt allows AI crawlers, so the site can appear in AI assistant answers.

How do I test my robots.txt?

Three methods: (1) Google Search Console's robots.txt report, under Settings > robots.txt, shows the robots.txt files Google found, when it last fetched them, and any errors. (It replaced the older robots.txt Tester, which Google retired in 2023.) (2) Visit your-domain.com/robots.txt in a browser to verify the file is accessible and contains the correct rules. (3) Use Google's URL Inspection tool in Search Console — it reports whether the page was blocked by robots.txt during crawling. Always test after making changes to catch accidental blocking of important pages.

Try These Tools

Related Articles