User-agent: * Disallow: /*/zip Disallow: /download:* Disallow: /login* Disallow: /*/likes Disallow: /cdn-cgi/ Disallow: /rss/* Disallow: /messages/compose Disallow: /apps/customizer/run Disallow: */following Disallow: */followers Disallow: /*/collections/*/things Disallow: /thing:*/attribution_card Disallow: /thing:*/collections Disallow: /thing:*/apps # Block search crawlers from crawling search results User-agent: Googlebot Disallow: /search Disallow: /tag:* User-agent: Bingbot Disallow: /search Disallow: /tag:* User-agent: DuckDuckBot Disallow: /search Disallow: /tag:* User-agent: Yandex Disallow: /search Disallow: /tag:* User-agent: SemrushBot Disallow: /