Term

Crawling

Crawling is how search engines visit web pages and download their content.

What it means

Automated programs called crawlers or bots visit URLs to read text, images, and other page resources. Search engines discover URLs through links, sitemaps, and pages they already know.

Example

You publish an article and link to it from your blog page. A search engine can discover that link and visit the article to read its content.

Keep in mind

Crawling and indexing are separate steps. A search engine can visit a page without adding it to its searchable index.