3pp-mirror/docusaurus
1
0
Fork 0
mirror of https://github.com/facebook/docusaurus.git synced 2025-08-03 00:39:45 +02:00
Code Issues Projects Releases Packages Wiki Activity

fix(sitemap): filter all routes with robots meta containing noindex (#7964)

Browse source
This commit is contained in:
Sébastien Lorber 2022-08-17 16:41:41 +02:00 • committed by GitHub
parent 39883e70de
commit a335a69982
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
4 changed files with 90 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

20
website/docs/seo.md
View file

@ -124,7 +124,11 @@ Read more about the robots file in [the Google documentation](https://developers
:::caution
**Important**: the `robots.txt` file does **not** prevent HTML pages from being indexed. Use `<meta name="robots" content="noindex">` as [page metadata](#single-page-metadata) to prevent it from appearing in search results entirely.
**Important**: the `robots.txt` file does **not** prevent HTML pages from being indexed.
To prevent your whole Docusaurus site from being indexed, use the [`noIndex`](./api/docusaurus.config.js.md#noIndex) site config. Some [hosting providers](./deployment.mdx) may also let you configure a `X-Robots-Tag: noindex` HTTP header (GitHub Pages does not support this).
To prevent a single page from being indexed, use `<meta name="robots" content="noindex">` as [page metadata](#single-page-metadata). Read more about the [robots meta tag](https://developers.google.com/search/docs/advanced/robots/robots_meta_tag).
:::
@ -132,6 +136,20 @@ Read more about the robots file in [the Google documentation](https://developers
Docusaurus provides the [`@docusaurus/plugin-sitemap`](./api/plugins/plugin-sitemap.md) plugin, which is shipped with `preset-classic` by default. It autogenerates a `sitemap.xml` file which will be available at `https://example.com/[baseUrl]/sitemap.xml` after the production build. This sitemap metadata helps search engine crawlers crawl your site more accurately.
:::tip
The sitemap plugin automatically filters pages containing a `noindex` [robots meta directive](https://developers.google.com/search/docs/advanced/robots/robots_meta_tag).
For example, [`/examples/noIndex`](/examples/noIndex) is not included in the [Docusaurus sitemap.xml file](pathname:///sitemap.xml) because it contains the following [page metadata](#single-page-metadata):
```html
<head>
<meta name="robots" content="noindex, nofollow" />
</head>
```
:::
## Human readable links {#human-readable-links}
Docusaurus uses your file names as links, but you can always change that using slugs, see this [tutorial](./guides/docs/docs-introduction.md#document-id) for more details.