explore-services/explore/src/robots.txt

19 lines
360 B
Plaintext

User-Agent: *
Disallow: /cache
Disallow: /upload
Disallow: /participate/direct-claim
Disallow: /myclaims
Disallow: /claims
Disallow: /search/dataset
Disallow: /search/other
Disallow: /search/project
Disallow: /search/dataprovider
Disallow: /search/organization
Disallow: /search/project-report
#Disallow: /
Sitemap: https://explore.openaire.eu/sitemap.xml