liquidex/treehouse
liquidex/treehouse
1
Fork 0
Code Issues 30 Activity

update robots.txt to only exclude specific subdirectories of /static

Browse source
This commit is contained in:
liquidex 2024-11-27 19:02:30 +01:00
parent 2e14197fd1
commit 1e3a1f3527
1 changed files with 7 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
static/robots.txt
View file

@ -4,8 +4,13 @@ User-Agent: *
Allow: *
# /static does not contain any pages.
Disallow: /static
# Some /static subdirectories are not crucial to rendering content and should not be crawled by bots.
Disallow: /static/chat
Disallow: /static/font
Disallow: /static/js
Disallow: /static/syntax
Disallow: /static/text
Disallow: /static/wasm
# /_treehouse contains system pages such as the 404 page.
Disallow: /_treehouse