Bootstrap/site/layouts/robots.txt

# www.robotstxt.org

{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}
{{- $allowCrawling := and (not $isNetlify) hugo.IsProduction -}}

{{ if $allowCrawling }}
# Allow crawling of all content
{{- end }}
User-agent: *
Disallow:{{ if not $allowCrawling }} /{{ end }}
Sitemap: {{ "/sitemap.xml" | absURL }}
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-08 17:33:28 +01:00			`# www.robotstxt.org`

Revert v5.getbootstrap.com changes 2020-11-24 20:05:11 +01:00			`{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}`
Switch to using `hugo.IsProduction` (#39763) 2024-03-08 11:41:17 +01:00			`{{- $allowCrawling := and (not $isNetlify) hugo.IsProduction -}}`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 16:26:25 +02:00
			`{{ if $allowCrawling }}`
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-08 17:33:28 +01:00			`# Allow crawling of all content`
			`{{- end }}`
			`User-agent: *`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 16:26:25 +02:00			`Disallow:{{ if not $allowCrawling }} /{{ end }}`
Use `absURL` so that we get consistent URLs. (#29226) 2019-08-08 17:30:43 +02:00			`Sitemap: {{ "/sitemap.xml" \| absURL }}`