How to Write a robots.txt File for Your Website (2026 Guide)

A misconfigured robots.txt file is one of the most common reasons a website loses organic traffic overnight. I've seen it happen on client sites — a well-meaning developer adds a single line that accidentally blocks Googlebot from crawling the entire site, and rankings collapse within weeks.

This guide covers what robots.txt actually does, what to block (and what never to block), how to handle common platforms like WordPress and Shopify, and how to deal with the new wave of AI scrapers.

If you want to skip straight to generating your file, use our free Robots.txt Generator — it handles WordPress, e-commerce, and AI bot blocking with a few clicks.

What is robots.txt?

robots.txt is a plain text file that sits at the root of your domain (at yourdomain.com/robots.txt) and tells web crawlers which pages they are and aren't allowed to visit.

It works through a simple instruction set:

• User-agent: which bot the rule applies to. * means all bots.
• Disallow: a path (or partial path) the bot should not crawl.
• Allow: explicitly permit a path (used when you've disallowed a parent directory but want to allow a subdirectory).
• Sitemap: tells crawlers where your XML sitemap lives.

A minimal, do-nothing robots.txt looks like this:

User-agent: *
Disallow:

Sitemap: https://yourdomain.com/sitemap.xml

Disallow: with nothing after it means "allow everything". The sitemap line tells Google and Bing where to find your full page inventory.

What should you actually block?

The goal of robots.txt is not to hide your site — it's to protect your crawl budget and prevent sensitive or low-value pages from being crawled unnecessarily. Here's what typically makes sense to block:

Admin and login pages

Pages like /admin/, /wp-admin/, and /wp-login.php should always be blocked. They have no value in search results, and exposing them to crawlers is a minor security risk. Google doesn't need to know your login page exists.

Duplicate or low-value content

Search result pages (/search/), tag archives (/tag/), and some author pages generate duplicate or thin content. Blocking them conserves crawl budget for your real content.

Private application areas

For SaaS products or web apps, block /app/, /dashboard/, /account/, and similar paths. These pages require authentication and serve no purpose in search results.

E-commerce transactional pages

Cart (/cart/), checkout (/checkout/), and order confirmation pages are transactional dead ends for crawlers. Block them to keep crawl budget on product and category pages.

What should you never block?

This is where sites get into trouble. These are the paths you should never disallow for Googlebot:

• CSS and JavaScript files — Google needs to render your pages to understand them. Blocking /wp-content/ or /assets/ entirely breaks rendering and can destroy rankings.
• Images in product or content pages — Google Images can drive real traffic. Only block images in private areas.
• Your sitemap path — never block /sitemap.xml or /sitemap_index.xml.
• Your main content pages — pages you want to rank should never appear in Disallow.

Platform-specific recommendations

WordPress

WordPress generates a default robots.txt that blocks /wp-admin/ while allowing /wp-admin/admin-ajax.php (which is needed for some plugins). A solid WordPress robots.txt looks like this:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /xmlrpc.php
Disallow: /wp-login.php
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap.xml

Note: /wp-content/uploads/ and /wp-content/themes/ should remain crawlable — these contain your images and CSS.

Shopify

Shopify generates its own robots.txt automatically and blocks /cart, /checkout, /account, and other transactional paths by default. Since 2021 you can customise it via robots.txt.liquid in your theme — but the defaults are sensible for most stores.

Wix, Squarespace, Webflow

These platforms manage robots.txt at the platform level. Wix and Squarespace give you limited control; Webflow lets you set a custom robots.txt in Site Settings → SEO. All three block admin paths by default.

Should you block AI scrapers?

This is a nuanced question with a real answer. There are two types of AI bots you need to think about separately:

AI training bots

These bots crawl your site to collect training data for large language models. OpenAI's GPTBot, Common Crawl's CCBot, Anthropic's anthropic-ai and Claude-Web, ByteDance's Bytespider, and Cohere's cohere-ai all fall into this category.

Blocking these prevents your content from being used to train AI models without compensation or consent. If you care about that, block them:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

AI answer-engine crawlers

These are different. Perplexity, Google AI Overviews, and similar tools crawl the web in real time to generate answers. Blocking them means you lose the chance of being cited when someone asks an AI a question your content answers.

This is the core trade-off of Answer Engine Optimisation (AEO). If being cited in AI answers is part of your strategy — and for most businesses it should be — you want answer-engine crawlers to access your content. Training bots and answer bots are not the same thing.

Our Robots.txt Generator blocks training bots by default but leaves answer engines open.

The Sitemap line

Always include your sitemap URL at the bottom of robots.txt:

Sitemap: https://yourdomain.com/sitemap.xml

This isn't a crawl instruction — it's a hint to Google, Bing, and other search engines about where to find your full page list. If you have a sitemap index file (common on WordPress with Yoast or RankMath), use that URL instead:

Sitemap: https://yourdomain.com/sitemap_index.xml

Submit this sitemap in Google Search Console too — the robots.txt sitemap line helps, but Search Console submission is the more reliable signal.

Testing your robots.txt

After creating or editing your robots.txt, test it before it causes damage:

1. Google Search Console → Settings → robots.txt — Google's built-in tester shows exactly which URLs are blocked and which are allowed under your current file.
2. Fetch a specific URL with the URL Inspection tool in Search Console to confirm Google can access your important pages.
3. Visit yourdomain.com/robots.txt directly in your browser to confirm the file is live and readable.

Common mistakes to avoid

• Blocking CSS/JS — kills rendering, hurts rankings silently.
• Disallowing / for all bots — blocks your entire site from being crawled. This has shipped to production more than once.
• Using robots.txt as a security measure — it's a polite request, not a lock. Malicious bots ignore it. Use server-side authentication for actual security.
• Blocking pages you want indexed — a surprisingly common issue when using wildcard patterns like Disallow: /*? which can match URLs you didn't intend.
• Forgetting the sitemap line — small omission, meaningful impact on crawl efficiency.

Putting it all together

robots.txt is one piece of a larger technical SEO picture. A well-configured file helps Google focus on your real content, but it needs to work alongside a proper sitemap, clean internal linking, and correct use of noindex where appropriate. See our full SEO checklist for the complete picture.

For schema markup — the other technical layer that helps AI engines understand your content — use the Schema Markup Generator. Together, robots.txt (what to crawl) and schema (how to understand what's crawled) form the technical foundation that both Google and AI answer engines need.