Robots.txt Generator
Generate a robots.txt file to control how search engine bots crawl and index your website.
⚙️ Configure Your robots.txt
User-agent: *
Disallow:
What is robots.txt?
A robots.txt file tells search engine bots which pages or sections of your website they can or cannot crawl. It is placed at the root of your domain (e.g., https://yoursite.com/robots.txt).
Click to expand...
Protect Private Pages
Block admin panels, login pages, and private content from being indexed by search engines.
Save Crawl Budget
Search engines have a limited crawl budget per site. Use robots.txt to direct bots to your most important pages.
Block AI Training Bots
New AI crawlers like GPTBot (OpenAI) and ClaudeBot (Anthropic) can be blocked from using your content for AI training.
🔗 Related Tools