Free Tool

Robots.txt Generator

Generate a robots.txt file for your website in seconds. Control which pages search engines can crawl, block AI scrapers, and declare your sitemap. Free, instant, no sign-up.

Used to build the Sitemap: line — e.g. https://yourdomain.com/sitemap.xml

Choose a template to pre-fill common rules, then customise as needed

Block these bots
Disallow these paths (optional)

Only set a crawl delay if your server struggles under crawl load. Googlebot ignores it — it uses Search Console's crawl rate settings instead.

Your robots.txt

  


  

    
What this file does

    
    
  
    

  
    
    
    
      
    
        
    Where to put your robots.txt
    
        
    Upload the file to the root of your domain — it must be accessible at yourdomain.com/robots.txt (not in a subfolder). Most hosting panels let you upload directly to the public_html or www folder. See our full robots.txt guide for platform-specific steps.
    
      
    
      
    
        
    Does robots.txt affect Google rankings?
    
        
    Indirectly, yes. Blocking unnecessary pages keeps your crawl budget focused on important content. But robots.txt controls crawling, not indexing — pages can still appear in Google if other sites link to them. Use noindex meta tags to block indexing.
    
      
    
      
    
        
    Should I block AI scrapers?
    
        
    Blocking AI training bots (GPTBot, CCBot) prevents your content being used to train AI models without your consent. However, it won't prevent AI answer engines like Perplexity from citing your site in answers — those use different mechanisms. See our AEO guide.
    
      
    
      
    
        
    Robots.txt vs meta robots
    
        
    robots.txt controls whether a bot can visit a page at all. The robots meta tag (noindex, nofollow) controls whether an already-visited page appears in search results. Use both — robots.txt alone can't prevent indexing via external links. Use our SEO checklist to cover both.