User-agent: AhrefsBot User-agent: MJ12bot User-agent: Exabot User-agent: Dotbot User-agent: SemrushBot User-agent: SemrushBot-SA User-agent: OnCrawl User-agent: cognitiveSEO User-agent: Screaming Frog SEO Spider User-agent: rogerbot User-agent: Ezooms User-agent: Gigabot User-agent: Yandex User-agent: YandexBot User-agent: UbiCrawler User-agent: DOC User-agent: Zao User-agent: sitecheck.internetseer.com User-agent: Zealbot User-agent: MSIECrawler User-agent: SiteSnagger User-agent: WebStripper User-agent: WebCopier User-agent: Fetch User-agent: Offline Explorer User-agent: Teleport User-agent: TeleportPro User-agent: WebZIP User-agent: linko User-agent: HTTrack User-agent: Microsoft.URL.Control User-agent: Xenu User-agent: larbin User-agent: libwww User-agent: ZyBORG User-agent: Download Ninja User-agent: spbot User-agent: Sosospider User-agent: BacklinkCrawler User-agent: ia_archiver User-agent: findlinks User-agent: SurveyBot User-agent: SEOENGBot User-agent: BPImageWalker User-agent: bdbrandprotect User-agent: LinkWalker User-agent: Updownerbot User-agent: HTTrack User-agent: AppEngine-Google User-agent: BLEXBot User-agent: Baiduspider User-agent: Baiduspider-image User-agent: Baiduspider-mobile User-agent: Baiduspider-news User-agent: Baiduspider-favo User-agent: Baiduspider-video User-agent: archive.org_bot disallow: / User-agent: * Disallow: /*? Disallow: /*/? Disallow: /? Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /cgi-bin Disallow: */trackback Disallow: /*/comments Disallow: /*.inc$ Disallow: /*.gz Disallow: /*.cgi Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: */feed Disallow: */rss2 Disallow: /go/