Robots.txt Generator

Create robots.txt files to control search engine crawling and indexing.

Sitemap URL

Enter your sitemap URL (optional)

User Agents

Allow all user agents

Allow Googlebot

Allow Bingbot

Disallow Paths

Example: /admin/, /private/, /temp/

Allow Paths

Example: /public/, /images/, /css/

Crawl Delay (seconds)

Time between crawler requests (0 for no delay)

Valid

Configure your settings and click "Generate Robots.txt" to see results

The robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site.