What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website (https://example.com/robots.txt) that tells web crawlers which pages they are allowed or disallowed from crawling. It follows the Robots Exclusion Protocol (REP). Search engine bots like Googlebot check this file before crawling your site. It's important to note that robots.txt is a request, not a technical barrier. Well-behaved bots respect it, but malicious bots may ignore it entirely.
Does blocking a page with robots.txt remove it from Google?
No. Disallowing a page in robots.txt prevents Google from crawling it, but if other pages link to the disallowed URL, Google can still discover and index it. It just won't crawl the content. The page may appear in search results with no description (Google knows it exists but can't read it). To remove a page from the index entirely, use the noindex meta tag instead. For pages that must never be indexed, combine both: noindex in the HTML and allow crawling so Google can see the noindex tag.
What is Crawl-delay?
Crawl-delay tells bots how many seconds to wait between requests to your server. For example, Crawl-delay: 10 means the bot waits 10 seconds between each page crawl. This is useful for sites with limited server resources that get overwhelmed by aggressive crawling. Important: Google officially does not respect Crawl-delay in robots.txt. Use Google Search Console's crawl rate settings instead. Most other bots (Bing, Yandex, etc.) do respect it.
Should I block AI training bots?
This is a personal/business decision. AI companies like OpenAI (GPTBot), Anthropic (anthropic-ai), and Common Crawl (CCBot) use web crawlers to collect training data. If you want to prevent your content from being used to train AI models, you can disallow these specific user agents. However, this only affects bots that identify themselves honestly. Older training data may already include your content. The Robots Exclusion Protocol now formally supports opting out of AI training via specific directives.
