Create and download a properly formatted robots.txt file for your website. Control which pages search engines can crawl with an easy visual interface. Essential for SEO professionals and web developers.
Crawl Rules
# Your robots.txt will appear here...
Place this file in your website root directory (e.g. yourdomain.com/robots.txt).
See your robots.txt update in real time as you configure rules. No need to click generate -- the output refreshes automatically.
One-click toggles for common rules like blocking /admin/ or /wp-login.php. Saves time and reduces errors for WordPress and CMS sites.
Add any custom paths you want to block from crawlers. Supports multiple paths, one per line, with automatic formatting.
Download the generated file directly as robots.txt or copy the content to clipboard for manual deployment.
A robots.txt file is a plain text file that sits in the root directory of your website and tells search engine crawlers which pages or directories they should or should not visit. It follows the Robots Exclusion Protocol, a standard used by all major search engines including Google, Bing, and Yahoo.
When a search engine bot arrives at your site, the first thing it does is look for a robots.txt file at yourdomain.com/robots.txt. The instructions in this file help crawlers understand the structure of your site and avoid wasting crawl budget on pages that are not important for indexing.
Proper robots.txt configuration is a fundamental part of technical SEO. It allows you to manage your crawl budget by preventing search engines from wasting resources on duplicate content, staging environments, admin panels, and other non-public pages.
Without a robots.txt file, search engines will attempt to crawl every page on your site. This can lead to indexing of sensitive areas, thin content appearing in search results, and inefficient use of Googlebot's limited crawl budget for your domain.
User-agent: * Disallow: /admin/ Disallow: /wp-login.php Disallow: /private/ Sitemap: https://example.com/sitemap.xml
This example allows all crawlers except for the /admin/, /wp-login.php, and /private/ directories, and includes a sitemap reference.
A robots.txt file is a plain text file placed in the root directory of a website that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It follows the Robots Exclusion Protocol.
The robots.txt file must be placed in the root directory of your website, accessible at yourdomain.com/robots.txt. Search engines check this location automatically before crawling.
Robots.txt can prevent crawlers from accessing pages, but it does not guarantee removal from search results. If other pages link to a blocked URL, Google may still index it. Use a noindex meta tag for guaranteed removal.
If you use 'Disallow: /' for all user agents, search engines will stop crawling your website entirely. Your pages will gradually be removed from search results as Google can no longer access them.