wires/hypnagaga_old
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fixed robots.txt rules to be on separate lines

Browse source 
Each rule in robots.txt should be a separate line based on https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
This commit is contained in:
itsjeffleong 2024-06-08 23:51:33 +10:00
parent ed9b306b1c
commit 00419718aa
1 changed files with 16 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
src/common/robots.njk
View file

@ -3,5 +3,19 @@ permalink: /robots.txt
eleventyExcludeFromCollections: true
excludeFromSitemap: true
---
User-agent: * Disallow: /404.html User-agent: GPTbot Disallow: / User-agent: ChatGPT-User Disallow: /
User-agent: Google-Extended Disallow: / User-agent: Omgilibot Disallow: / Sitemap: {{ meta.url }}/sitemap.xml
User-agent: *
Disallow: /404.html
User-agent: GPTbot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Omgilibot
Disallow: /
Sitemap: {{ meta.url }}/sitemap.xml