General User Agent List for SiteMap Indexing

robots.txt

################################# ROBOTS.TXT ###################################
# Updates can be retrieved from: https://github.com/jonasjacek/robots.txt      #
# This document is licensed with a CC BY-NC-SA 4.0 license.                    #
# Last update: 2019-03-07                                                      #
################################################################################
Sitemap: (https://website/sitemap.xml)

User-agent: 360Spider
User-agent: 360Spider-Image
User-agent: 360Spider-Video
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: adidxbot
User-agent: Applebot
User-agent: AppleNewsBot
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: bingbot
User-agent: BingPreview
User-agent: BublupBot
User-agent: CCBot
User-agent: Cliqzbot
User-agent: coccoc
User-agent: coccocbot-image
User-agent: coccocbot-web
User-agent: Daumoa
User-agent: Dazoobot
User-agent: DeuSu
User-agent: DuckDuckBot
User-agent: DuckDuckGo-Favicons-Bot
User-agent: EuripBot
User-agent: Exploratodo
User-agent: Facebot
User-agent: Feedly
User-agent: Findxbot
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: HaoSouSpider
User-agent: ichiro
User-agent: istellabot
User-agent: JikeSpider
User-agent: Lycos
User-agent: Mail.Ru
User-agent: Mediapartners-Google
User-agent: MojeekBot
User-agent: msnbot
User-agent: msnbot-media
User-agent: OrangeBot
User-agent: Pinterest
User-agent: Plukkie
User-agent: Qwantify
User-agent: Rambler
User-agent: SeznamBot
User-agent: Sosospider
User-agent: Slurp
User-agent: Sogou blog
User-agent: Sogou inst spider
User-agent: Sogou News Spider
User-agent: Sogou Orion spider
User-agent: Sogou spider2
User-agent: Sogou web spider
User-agent: SputnikBot
User-agent: Teoma
User-agent: Twitterbot
User-agent: wotbox
User-agent: yacybot
User-agent: Yandex
User-agent: YandexMobileBot
User-agent: Yeti
User-agent: YioopBot
User-agent: yoozBot
User-agent: YoudaoBot
User-agent: SemrushBot
Disallow: /user/
Disallow: /login
Disallow: /policies
Disallow: /watchlists
User-agent: *
Disallow: /