Block Search Engines

When you request a new site, all search engines (Google, Bing, etc.) can start indexing your published site content as soon as your site is located.

Block search engines

While building your site, you may want to temporarily block it from search engines to avoid inaccurate or incomplete content from appearing in search results.

To temporarily or permanently block your site from crawling by search engines, go to Configuration > System > Site Settings. Click on the Search tab. Check the box that says Block Search Engines.

When we launch your site, we will assume you want the site unblocked unless otherwise specified.

Unpublished pages or paragraphs will never be indexed by search engines. If you are not ready for users to find your pages, you can always unpublish the content. Learn more about unpublishing content.

Stanford Sites Intranet content is never indexed by search engines.

Common questions about search engines

What will get indexed by search engines?

Search engines will be able to index all published pages on your site. Unpublished pages cannot be seen by the search engines. If you wish to see a full list of what is visible to a search engine, you can go to https://[yoursite]/sitemap.xml.

What if my site has already been indexed?

If you wish to remove your site from the index, you can wait for the site crawler to attempt to visit your site again, or you can typically use tools from the search engine provider to remove pages from the index, such as Google Search Console.