Term
Robots.txt
Robots.txt is a text file that tells web crawlers which parts of a website they may crawl.
What it means
The file sits at the root of a site, such as example.com/robots.txt. Its rules can apply to all crawlers that follow the standard or to a particular bot. Site owners use it to manage crawler access to pages and files.
Example
An online store might block crawlers from internal search results while allowing access to product and category pages.
Keep in mind
Blocking crawling does not reliably remove a URL from search results. Use a noindex instruction on a crawlable page to prevent indexing, and password protection for private content.