Skip to content

Instantly share code, notes, and snippets.

@josuamarcelc
Created January 7, 2024 04:56
Show Gist options
  • Save josuamarcelc/c2c9caef79cde38865bb36f4fad11483 to your computer and use it in GitHub Desktop.
Save josuamarcelc/c2c9caef79cde38865bb36f4fad11483 to your computer and use it in GitHub Desktop.
Template of User Agents - Robots.txt
User-Agent: Googlebot
Allow: /
User-Agent: Googlebot-Mobile
Allow: /
User-Agent: Googlebot-Image
Allow: /
User-Agent: Googlebot-News
Allow: /
User-Agent: Googlebot-Video
Allow: /
User-Agent: AdsBot-Google
Allow: /
User-Agent: AdsBot-Google-Mobile
Allow: /
User-Agent: Feedfetcher-Google
Allow: /
User-Agent: Mediapartners-Google
Allow: /
User-Agent: Mediapartners Googlebot
Allow: /
User-Agent: APIs-Google
Allow: /
User-Agent: Google-InspectionTool
Allow: /
User-Agent: Storebot-Google
Allow: /
User-Agent: GoogleOther
Allow: /
User-Agent: bingbot
Allow: /
User-Agent: Slurp
Allow: /
User-Agent: wWget
Allow: /
User-Agent: LinkedInBot
Allow: /
User-Agent: Python-urllib
Allow: /
User-Agent: python-requests
Allow: /
User-Agent: aiohttp
Allow: /
User-Agent: httpx
Allow: /
User-Agent: libwww-perl
Allow: /
User-Agent: httpunit
Allow: /
User-Agent: nutch
Allow: /
User-Agent: Go-http-client
Allow: /
User-Agent: phpcrawl
Allow: /
User-Agent: msnbot
Allow: /
User-Agent: jyxobot
Allow: /
User-Agent: FAST-WebCrawler
Allow: /
User-Agent: FAST Enterprise Crawler
Allow: /
User-Agent: BIGLOTRON
Allow: /
User-Agent: Teoma
Allow: /
User-Agent: convera
Allow: /
User-Agent: seekbot
Allow: /
User-Agent: Gigabot
Allow: /
User-Agent: Gigablast
Allow: /
User-Agent: exabot
Allow: /
User-Agent: ia_archiver
Allow: /
User-Agent: GingerCrawler
Allow: /
User-Agent: webmon
Allow: /
User-Agent: HTTrack
Allow: /
User-Agent: gruborg
Allow: /
User-Agent: UsineNouvelleCrawler
Allow: /
User-Agent: antibot
Allow: /
User-Agent: netresearchserver
Allow: /
User-Agent: speedy
Allow: /
User-Agent: fluffy
Allow: /
User-Agent: findlink
Allow: /
User-Agent: msrbot
Allow: /
User-Agent: panscient
Allow: /
User-Agent: yacybot
Allow: /
User-Agent: AISearchBot
Allow: /
User-Agent: ips-agent
Allow: /
User-Agent: tagoobot
Allow: /
User-Agent: MJ12bot
Allow: /
User-Agent: woriobot
Allow: /
User-Agent: yanga
Allow: /
User-Agent: buzzbot
Allow: /
User-Agent: mlbot
Allow: /
User-Agent: yandexcombots
Allow: /
User-Agent: purebot
Allow: /
User-Agent: Linguee Bot
Allow: /
User-Agent: CyberPatrol
Allow: /
User-Agent: voilabot
Allow: /
User-Agent: Baiduspider
Allow: /
User-Agent: citeseerxbot
Allow: /
User-Agent: spbot
Allow: /
User-Agent: twengabot
Allow: /
User-Agent: postrank
Allow: /
User-Agent: Turnitin
Allow: /
User-Agent: scribdbot
Allow: /
User-Agent: page2rss
Allow: /
User-Agent: sitebot
Allow: /
User-Agent: linkdex
Allow: /
User-Agent: Adidxbot
Allow: /
User-Agent: ezooms
Allow: /
User-Agent: dotbot
Allow: /
User-Agent: MailRU_Bot
Allow: /
User-Agent: discobot
Allow: /
User-Agent: heritrix
Allow: /
User-Agent: findthatfile
Allow: /
User-Agent: europarchiveorg
Allow: /
User-Agent: NerdByNatureBot
Allow: /
User-Agent: sistrix crawler
Allow: /
User-Agent: AhrefsBotSiteAudit
Allow: /
User-Agent: fuelbot
Allow: /
User-Agent: CrunchBot
Allow: /
User-Agent: IndeedBot
Allow: /
User-Agent: mappydata
Allow: /
User-Agent: woobot
Allow: /
User-Agent: ZoominfoBot
Allow: /
User-Agent: PrivacyAwareBot
Allow: /
User-Agent: Multiviewbot
Allow: /
User-Agent: SWIMGBot
Allow: /
User-Agent: Grobbot
Allow: /
User-Agent: eright
Allow: /
User-Agent: Apercite
Allow: /
User-Agent: semanticbot
Allow: /
User-Agent: Aboundex
Allow: /
User-Agent: domaincrawler
Allow: /
User-Agent: wbsearchbot
Allow: /
User-Agent: summify
Allow: /
User-Agent: CCBot
Allow: /
User-Agent: edisterbot
Allow: /
User-Agent: SeznamBot
Allow: /
User-Agent: ec2linkfinder
Allow: /
User-Agent: gslfbot
Allow: /
User-Agent: aiHitBot
Allow: /
User-Agent: intelium_bot
Allow: /
User-Agent: facebookexternalhit
Allow: /
User-Agent: Yeti
Allow: /
User-Agent: RetrevoPageAnalyzer
Allow: /
User-Agent: lb-spider
Allow: /
User-Agent: Sogou
Allow: /
User-Agent: lssbot
Allow: /
User-Agent: careerbot
Allow: /
User-Agent: wotbox
Allow: /
User-Agent: wocbot
Allow: /
User-Agent: ichiro
Allow: /
User-Agent: DuckDuckBot
Allow: /
User-Agent: lssrocketcrawler
Allow: /
User-Agent: drupact
Allow: /
User-Agent: webcompanycrawler
Allow: /
User-Agent: acoonbot
Allow: /
User-Agent: openindexspider
Allow: /
User-Agent: gnam gnam spider
Allow: /
User-Agent: web-archive-netcombot
Allow: /
User-Agent: backlinkcrawler
Allow: /
User-Agent: coccoc
Allow: /
User-Agent: integromedb
Allow: /
User-Agent: content crawler spider
Allow: /
User-Agent: toplistbot
Allow: /
User-Agent: it2media-domain-crawler
Allow: /
User-Agent: ip-web-crawlercom
Allow: /
User-Agent: siteexplorerinfo
Allow: /
User-Agent: elisabot
Allow: /
User-Agent: proximic
Allow: /
User-Agent: changedetection
Allow: /
User-Agent: arabot
Allow: /
User-Agent: WeSEESearch
Allow: /
User-Agent: niki-bot
Allow: /
User-Agent: CrystalSemanticsBot
Allow: /
User-Agent: rogerbot
Allow: /
User-Agent: 360Spider
Allow: /
User-Agent: psbot
Allow: /
User-Agent: InterfaxScanBot
Allow: /
User-Agent: CC Metadata Scaper
Allow: /
User-Agent: g00g1enet
Allow: /
User-Agent: GrapeshotCrawler
Allow: /
User-Agent: urlappendbot
Allow: /
User-Agent: brainobot
Allow: /
User-Agent: fr-crawler
Allow: /
User-Agent: binlar
Allow: /
User-Agent: SimpleCrawler
Allow: /
User-Agent: Twitterbot
Allow: /
User-Agent: cXensebot
Allow: /
User-Agent: smtbot
Allow: /
User-Agent: bnffr_bot
Allow: /
User-Agent: A6-Indexer
Allow: /
User-Agent: ADmantX
Allow: /
User-Agent: Facebot
Allow: /
User-Agent: OrangeBot
Allow: /
User-Agent: memorybot
Allow: /
User-Agent: AdvBot
Allow: /
User-Agent: MegaIndex
Allow: /
User-Agent: SemanticScholarBot
Allow: /
User-Agent: ltx71
Allow: /
User-Agent: nerdybot
Allow: /
User-Agent: xovibot
Allow: /
User-Agent: BUbiNG
Allow: /
User-Agent: Qwantify
Allow: /
User-Agent: archiveorg_bot
Allow: /
User-Agent: Applebot
Allow: /
User-Agent: TweetmemeBot
Allow: /
User-Agent: crawler4j
Allow: /
User-Agent: findxbot
Allow: /
User-Agent: SeEmMrushBot
Allow: /
User-Agent: yoozBot
Allow: /
User-Agent: lipperhey
Allow: /
User-Agent: YJ
Allow: /
User-Agent: Domain Re-Animator Bot
Allow: /
User-Agent: AddThis
Allow: /
User-Agent: Screaming Frog SEO Spider
Allow: /
User-Agent: MetaURI
Allow: /
User-Agent: Scrapy
Allow: /
User-Agent: LivelapbBot
Allow: /
User-Agent: OpenHoseBot
Allow: /
User-Agent: CapsuleChecker
Allow: /
User-Agent: collectioninfegycom
Allow: /
User-Agent: IstellaBot
Allow: /
User-Agent: DeuSu
Allow: /
User-Agent: betaBot
Allow: /
User-Agent: Cliqzbot
Allow: /
User-Agent: MojeekBot
Allow: /
User-Agent: netEstate NE Crawler
Allow: /
User-Agent: SafeSearch microdata crawler
Allow: /
User-Agent: Gluten Free Crawler
Allow: /
User-Agent: Sonic
Allow: /
User-Agent: Sysomos
Allow: /
User-Agent: Trove
Allow: /
User-Agent: deadlinkchecker
Allow: /
User-Agent: Slack-ImgProxy
Allow: /
User-Agent: Embedly
Allow: /
User-Agent: RankActiveLinkBot
Allow: /
User-Agent: iskanie
Allow: /
User-Agent: SafeDNSBot
Allow: /
User-Agent: SkypeUriPreview
Allow: /
User-Agent: Veoozbot
Allow: /
User-Agent: Slackbot
Allow: /
User-Agent: redditbot
Allow: /
User-Agent: datagnionbot
Allow: /
User-Agent: Google-Adwords-Instant
Allow: /
User-Agent: adbeat_bot
Allow: /
User-Agent: WhatsApp
Allow: /
User-Agent: contxbot
Allow: /
User-Agent: pinterestcombot
Allow: /
User-Agent: electricmonk
Allow: /
User-Agent: GarlikCrawler
Allow: /
User-Agent: BingPreview
Allow: /
User-Agent: vebidoobot
Allow: /
User-Agent: FemtosearchBot
Allow: /
User-Agent: Yahoo Link Preview
Allow: /
User-Agent: MetaJobBot
Allow: /
User-Agent: DomainStatsBot
Allow: /
User-Agent: mindUpBot
Allow: /
User-Agent: Daum
Allow: /
User-Agent: Jugendschutzprogramm-Crawler
Allow: /
User-Agent: Xenu Link Sleuth
Allow: /
User-Agent: Pcore-HTTP
Allow: /
User-Agent: moatbot
Allow: /
User-Agent: KosmioBot
Allow: /
User-Agent: pPingdom
Allow: /
User-Agent: AppInsights
Allow: /
User-Agent: PhantomJS
Allow: /
User-Agent: Gowikibot
Allow: /
User-Agent: PiplBot
Allow: /
User-Agent: Discordbot
Allow: /
User-Agent: TelegramBot
Allow: /
User-Agent: Jetslide
Allow: /
User-Agent: newsharecounts
Allow: /
User-Agent: James BOT
Allow: /
User-Agent: BarkrRowler
Allow: /
User-Agent: TinEye
Allow: /
User-Agent: SocialRankIOBot
Allow: /
User-Agent: trendictionbot
Allow: /
User-Agent: Ocarinabot
Allow: /
User-Agent: epicbot
Allow: /
User-Agent: Primalbot
Allow: /
User-Agent: DuckDuckGo-Favicons-Bot
Allow: /
User-Agent: GnowitNewsbot
Allow: /
User-Agent: Leikibot
Allow: /
User-Agent: LinkArchiver
Allow: /
User-Agent: YaK
Allow: /
User-Agent: PaperLiBot
Allow: /
User-Agent: Digg Deeper
Allow: /
User-Agent: dcrawl
Allow: /
User-Agent: Snacktory
Allow: /
User-Agent: AndersPinkBot
Allow: /
User-Agent: Fyrebot
Allow: /
User-Agent: EveryoneSocialBot
Allow: /
User-Agent: Mediatoolkitbot
Allow: /
User-Agent: Luminator-robots
Allow: /
User-Agent: ExtLinksBot
Allow: /
User-Agent: SurveyBot
Allow: /
User-Agent: NING
Allow: /
User-Agent: okhttp
Allow: /
User-Agent: Nuzzel
Allow: /
User-Agent: omgili
Allow: /
User-Agent: PocketParser
Allow: /
User-Agent: YisouSpider
Allow: /
User-Agent: um-LN
Allow: /
User-Agent: ToutiaoSpider
Allow: /
User-Agent: MuckRack
Allow: /
User-Agent: Jamies Spider
Allow: /
User-Agent: AHC
Allow: /
User-Agent: NetcraftSurveyAgent
Allow: /
User-Agent: Laserlikebot
Allow: /
User-Agent: Apache-HttpClient
Allow: /
User-Agent: AppEngine-Google
Allow: /
User-Agent: Jetty
Allow: /
User-Agent: Upflow
Allow: /
User-Agent: Thinklab
Allow: /
User-Agent: Traackrcom
Allow: /
User-Agent: Twurly
Allow: /
User-Agent: Mastodon
Allow: /
User-Agent: http_get
Allow: /
User-Agent: DnyzBot
Allow: /
User-Agent: botify
Allow: /
User-Agent: 007ac9 Crawler
Allow: /
User-Agent: BehloolBot
Allow: /
User-Agent: BrandVerity
Allow: /
User-Agent: check_http
Allow: /
User-Agent: BDCbot
Allow: /
User-Agent: ZumBot
Allow: /
User-Agent: EZID
Allow: /
User-Agent: ICC-Crawler
Allow: /
User-Agent: ArchiveBot
Allow: /
User-Agent: LCC
Allow: /
User-Agent: filterdbissnetcrawler
Allow: /
User-Agent: BLP_bbot
Allow: /
User-Agent: BomboraBot
Allow: /
User-Agent: Buck
Allow: /
User-Agent: Companybook-Crawler
Allow: /
User-Agent: Genieo
Allow: /
User-Agent: magpie-crawler
Allow: /
User-Agent: MeltwaterNews
Allow: /
User-Agent: Moreover
Allow: /
User-Agent: newspaper
Allow: /
User-Agent: ScoutJet
Allow: /
User-Agent: sentry
Allow: /
User-Agent: StorygizeBot
Allow: /
User-Agent: UptimeRobot
Allow: /
User-Agent: OutclicksBot
Allow: /
User-Agent: seoscanners
Allow: /
User-Agent: Hatena
Allow: /
User-Agent: Google Web Preview
Allow: /
User-Agent: MauiBot
Allow: /
User-Agent: AlphaBot
Allow: /
User-Agent: SBL-BOT
Allow: /
User-Agent: IAS crawler
Allow: /
User-Agent: adscanner
Allow: /
User-Agent: Netvibes
Allow: /
User-Agent: acapbot
Allow: /
User-Agent: Baidu-YunGuanCe
Allow: /
User-Agent: bitlybot
Allow: /
User-Agent: blogmuraBot
Allow: /
User-Agent: BotAraTurkacom
Allow: /
User-Agent: bot-pgechlooecom
Allow: /
User-Agent: BoxcarBot
Allow: /
User-Agent: BTWebClient
Allow: /
User-Agent: ContextAd Bot
Allow: /
User-Agent: Digincore bot
Allow: /
User-Agent: Disqus
Allow: /
User-Agent: Feedly
Allow: /
User-Agent: Fetch
Allow: /
User-Agent: Fever
Allow: /
User-Agent: Flamingo_SearchEngine
Allow: /
User-Agent: FlipboardProxy
Allow: /
User-Agent: g2reader-bot
Allow: /
User-Agent: G2 Web Services
Allow: /
User-Agent: imrbot
Allow: /
User-Agent: K7MLWCBot
Allow: /
User-Agent: Kemvibot
Allow: /
User-Agent: Landau-Media-Spider
Allow: /
User-Agent: linkapediabot
Allow: /
User-Agent: vkShare
Allow: /
User-Agent: Siteimprovecom
Allow: /
User-Agent: BLEXBot
Allow: /
User-Agent: DareBoost
Allow: /
User-Agent: ZuperlistBot
Allow: /
User-Agent: Miniflux
Allow: /
User-Agent: Feedspot
Allow: /
User-Agent: Diffbot
Allow: /
User-Agent: SEOkicks
Allow: /
User-Agent: tracemyfile
Allow: /
User-Agent: Nimbostratus-Bot
Allow: /
User-Agent: zgrab
Allow: /
User-Agent: PR-CYRU
Allow: /
User-Agent: AdsTxtCrawler
Allow: /
User-Agent: Datafeedwatch
Allow: /
User-Agent: Zabbix
Allow: /
User-Agent: TangibleeBot
Allow: /
User-Agent: google-xrawler
Allow: /
User-Agent: axios
Allow: /
User-Agent: Amazon CloudFront
Allow: /
User-Agent: Pulsepoint
Allow: /
User-Agent: CloudFlare-AlwaysOnline
Allow: /
User-Agent: Google-Structured-Data-Testing-Tool
Allow: /
User-Agent: WordupInfoSearch
Allow: /
User-Agent: WebDataStats
Allow: /
User-Agent: HttpUrlConnection
Allow: /
User-Agent: Seekport Crawler
Allow: /
User-Agent: ZoomBot
Allow: /
User-Agent: VelenPublicWebCrawler
Allow: /
User-Agent: MoodleBot
Allow: /
User-Agent: jpg-newsbot
Allow: /
User-Agent: outbrain
Allow: /
User-Agent: W3C_Validator
Allow: /
User-Agent: Validatornu
Allow: /
User-Agent: W3C-checklink
Allow: /
User-Agent: W3C-mobileOK
Allow: /
User-Agent: W3C_I18n-Checker
Allow: /
User-Agent: FeedValidator
Allow: /
User-Agent: W3C_CSS_Validator
Allow: /
User-Agent: W3C_Unicorn
Allow: /
User-Agent: Google-PhysicalWeb
Allow: /
User-Agent: Blackboard
Allow: /
User-Agent: ICBot
Allow: /
User-Agent: BazQux
Allow: /
User-Agent: Twingly
Allow: /
User-Agent: Rivva
Allow: /
User-Agent: Experibot
Allow: /
User-Agent: awesomecrawler
Allow: /
User-Agent: Dataprovidercom
Allow: /
User-Agent: GroupHigh
Allow: /
User-Agent: theoldreadercom
Allow: /
User-Agent: AnyEvent
Allow: /
User-Agent: Uptimebotorg
Allow: /
User-Agent: Nmap Scripting Engine
Allow: /
User-Agent: 2ipru
Allow: /
User-Agent: Clickagy
Allow: /
User-Agent: Caliperbot
Allow: /
User-Agent: MBCrawler
Allow: /
User-Agent: online-webceo-bot
Allow: /
User-Agent: B2B Bot
Allow: /
User-Agent: AddSearchBot
Allow: /
User-Agent: Google Favicon
Allow: /
User-Agent: HubSpot
Allow: /
User-Agent: Chrome-Lighthouse
Allow: /
User-Agent: HeadlessChrome
Allow: /
User-Agent: CheckMarkNetwork
Allow: /
User-Agent: wwwuptimecom
Allow: /
User-Agent: Streamline3Bot
Allow: /
User-Agent: serpstatbot
Allow: /
User-Agent: MixnodeCache
Allow: /
User-Agent: curl
Allow: /
User-Agent: SimpleScraper
Allow: /
User-Agent: RSSingBot
Allow: /
User-Agent: Jooblebot
Allow: /
User-Agent: fedoraplanet
Allow: /
User-Agent: Friendica
Allow: /
User-Agent: NextCloud
Allow: /
User-Agent: Tiny Tiny RSS
Allow: /
User-Agent: RegionStuttgartBot
Allow: /
User-Agent: Bytespider
Allow: /
User-Agent: Datanyze
Allow: /
User-Agent: Google-Site-Verification
Allow: /
User-Agent: TrendsmapResolver
Allow: /
User-Agent: tweetedtimes
Allow: /
User-Agent: NTENTbot
Allow: /
User-Agent: Gwene
Allow: /
User-Agent: SimplePie
Allow: /
User-Agent: SearchAtlas
Allow: /
User-Agent: Superfeedr
Allow: /
User-Agent: feedbot
Allow: /
User-Agent: UT-Dorkbot
Allow: /
User-Agent: Amazonbot
Allow: /
User-Agent: SerendeputyBot
Allow: /
User-Agent: Eyeotabot
Allow: /
User-Agent: officestorebot
Allow: /
User-Agent: Neticle Crawler
Allow: /
User-Agent: SurdotlyBot
Allow: /
User-Agent: LinkisBot
Allow: /
User-Agent: AwarioSmartBot
Allow: /
User-Agent: AwarioRssBot
Allow: /
User-Agent: RyteBot
Allow: /
User-Agent: FreeWebMonitoring SiteChecker
Allow: /
User-Agent: AspiegelBot
Allow: /
User-Agent: NAVER Blog Rssbot
Allow: /
User-Agent: zenback bot
Allow: /
User-Agent: SentiBot
Allow: /
User-Agent: Domains Project
Allow: /
User-Agent: Pandalytics
Allow: /
User-Agent: VKRobot
Allow: /
User-Agent: bidswitchbot
Allow: /
User-Agent: tigerbot
Allow: /
User-Agent: NIXStatsbot
Allow: /
User-Agent: Atom Feed Robot
Allow: /
User-Agent: Ccurebot
Allow: /
User-Agent: PagePeeker
Allow: /
User-Agent: Vigil
Allow: /
User-Agent: rssbot
Allow: /
User-Agent: startmebot
Allow: /
User-Agent: JobboerseBot
Allow: /
User-Agent: seewithkids
Allow: /
User-Agent: NINJA bot
Allow: /
User-Agent: Cutbot
Allow: /
User-Agent: BublupBot
Allow: /
User-Agent: BrandONbot
Allow: /
User-Agent: RidderBot
Allow: /
User-Agent: Taboolabot
Allow: /
User-Agent: Dubbotbot
Allow: /
User-Agent: FindITAnswersbot
Allow: /
User-Agent: infoobot
Allow: /
User-Agent: Refindbot
Allow: /
User-Agent: BlogTrafficdd Feed-Fetcher
Allow: /
User-Agent: SeobilityBot
Allow: /
User-Agent: Cincraw
Allow: /
User-Agent: Dragonbot
Allow: /
User-Agent: VoluumDSP-content-bot
Allow: /
User-Agent: FreshRSS
Allow: /
User-Agent: BitBot
Allow: /
User-Agent: PHP-Curl-Class
Allow: /
User-Agent: Google-Certificates-Bridge
Allow: /
User-Agent: centurybot
Allow: /
User-Agent: Viber
Allow: /
User-Agent: eventures Investment Crawler
Allow: /
User-Agent: evc-batch
Allow: /
User-Agent: PetalBot
Allow: /
User-Agent: virustotal
Allow: /
User-Agent: PTST
Allow: /
User-Agent: minicrawler
Allow: /
User-Agent: Cookiebot
Allow: /
User-Agent: trovitBot
Allow: /
User-Agent: seostarco
Allow: /
User-Agent: IonCrawl
Allow: /
User-Agent: Uptime-Kuma
Allow: /
User-Agent: SeekportBot
Allow: /
User-Agent: FreshpingBot
Allow: /
User-Agent: Feedbin
Allow: /
User-Agent: CriteoBot
Allow: /
User-Agent: Snap URL Preview Service
Allow: /
User-Agent: Better Uptime Bot
Allow: /
User-Agent: RuxitSynthetic
Allow: /
User-Agent: Google-Read-Aloud
Allow: /
User-Agent: ValveSteam
Allow: /
User-Agent: OdklBot
Allow: /
User-Agent: GPTBot
Allow: /
User-Agent: YandexRenderResourcesBot
Allow: /
User-Agent: LightspeedSystemsCrawler
Allow: /
User-Agent: ev-crawler
Allow: /
User-Agent: BitSightBot
Allow: /
User-Agent: woorankreview
Allow: /
User-Agent: Google-Safety
Allow: /
User-Agent: AwarioBot
Allow: /
User-Agent: DataForSeoBot
Allow: /
User-Agent: Linespider
Allow: /
User-Agent: WellKnownBot
Allow: /
User-Agent: A Patent Crawler
Allow: /
User-Agent: StractBot
Allow: /
User-Agent: searchmarginalianu
Allow: /
User-Agent: YouBot
Allow: /
User-Agent: Nicecrawler
Allow: /
User-Agent: Neevabot
Allow: /
User-Agent: BrightEdge Crawler
Allow: /
User-Agent: SiteCheckerBotCrawler
Allow: /
User-Agent: TombaPublicWebCrawler
Allow: /
User-Agent: CrawlyProjectCrawler
Allow: /
User-Agent: KomodiaBot
Allow: /
User-Agent: KStandBot
Allow: /
User-Agent: CISPA Webcrawler
Allow: /
User-Agent: MTRobot
Allow: /
User-Agent: hyscoreio
Allow: /
User-Agent: AlexandriaOrgBot
Allow: /
User-Agent: 2ip bot
Allow: /
User-Agent: Yellowbrandprotectionbot
Allow: /
User-Agent: SEOlizer
Allow: /
User-Agent: vuhuvBot
Allow: /
User-Agent: INETDEX-BOT
Allow: /
User-Agent: Synapse
Allow: /
User-Agent: t3versionsBot
Allow: /
User-Agent: deepnoc
Allow: /
User-Agent: Cocolyzebot
Allow: /
User-Agent: hypestat
Allow: /
User-Agent: ReverseEngineeringBot
Allow: /
User-Agent: sempitech
Allow: /
User-Agent: Iframely
Allow: /
User-Agent: MetaInspector
Allow: /
User-Agent: node-fetch
Allow: /
User-Agent: lkxscan
Allow: /
User-Agent: python-opengraph
Allow: /
User-Agent: OpenGraphCheck
Allow: /
User-Agent: developersgooglecomwebsnippet
Allow: /
User-Agent: SenutoBot
Allow: /
User-Agent: MaCoCu
Allow: /
User-Agent: NewsBlur
Allow: /
User-Agent: inoreader
Allow: /
User-Agent: NetSystemsResearch
Allow: /
User-Agent: PageThing
Allow: /
User-Agent: WordPress
Allow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment