← Back to Tools

robots.txt Generator

Create a custom robots.txt file for AI bots in 2 minutes

1. Choose Your Strategy

2. Select AI Bots

4. Domain (Optional)

Used to add sitemap URL to robots.txt

Your robots.txt

# Generated by LLMDiscovery.ai
# https://llmdiscovery.ai/tools/robots-generator

# Allow all bots (including AI crawlers)
User-agent: *
Allow: /

# Explicitly allow GPTBot
User-agent: GPTBot
Allow: /

# Explicitly allow ClaudeBot
User-agent: ClaudeBot
Allow: /

# Explicitly allow Google-Extended
User-agent: Google-Extended
Allow: /

# Explicitly allow PerplexityBot
User-agent: PerplexityBot
Allow: /

📝 Installation Instructions

  1. 1. Copy or download the generated robots.txt
  2. 2. Upload to your website's root directory
  3. 3. Verify at: https://yourdomain.com/robots.txt
  4. 4. Test with Google's robots.txt tester

Track Bot Compliance

Want to see if bots actually respect your robots.txt? Track real bot visits with LLMDiscovery and get instant Chrome notifications.

Start Tracking Free

✨ Includes Chrome extension for real-time alerts