Robots Configuration

Crawler directives & webmaster information for #about

# robots.txt for #about
# Last updated: 2025-01-15

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/v1/internal/
Disallow: /*.pdf$
Allow: /*.css$
Allow: /*.js$
Allow: /*.png$
Allow: /*.jpg$
Allow: /*.svg$

Crawl-delay: 1

Sitemap: https://www.about.com/sitemap.xml
Sitemap: https://www.about.com/sitemap-blog.xml

# Special directives for AI/LLM crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Googlebot-Image
Allow: /
Crawl-delay: 2

📄 XML Sitemap

Complete index of our public pages for efficient crawling.

View sitemap.xml →

🔐 Restricted Areas

Admin panels, private assets, and internal APIs are blocked to preserve performance & security.

🤖 AI/LLM Crawlers

Training data collection is restricted. Please respect our content usage policy.

📧 Webmaster Contact

Questions about our crawling policy? Reach out to our technical team.

webmaster@about.com