What is robots.txt?

A robots.txt file gives instructions to search engine crawlers about which areas of a website they can or cannot crawl. It is usually placed in the root folder of a website.

How to use this tool

Select your crawl settings, add your sitemap URL, and choose any paths you want to block. The tool will generate a simple robots.txt file that you can copy.

Important warning

Do not block important pages that you want indexed in search engines. Robots.txt controls crawling, but it does not guarantee that a URL will never appear in search results.

Best practice

For most WordPress websites, it is usually enough to allow crawlers, block sensitive admin paths, and include your XML sitemap URL.