Skip to content

Instantly share code, notes, and snippets.

@anjackson

anjackson/crawl.log.sample.log

Created July 13, 2022 10:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save anjackson/52410e80858b3f67b11292fd1fa2c4d2 to your computer and use it in GitHub Desktop.
Save anjackson/52410e80858b3f67b11292fd1fa2c4d2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
crawl.log.sample.log
2022-07-13T09:58:56.201Z 200 1039 https://www.vectisradio.com/wp-sitemap.xml I https://www.vectisradio.com/robots.txt application/xml #127 20220713095855289+777 sha1:VFT6NNA5DW5POM6PGF7ERSJYBV6LDNAJ tid:97213:https://www.vectisradio.com/schedule/ isSitemap,launchTimestamp:20220713095854,duplicate:digest,ip:185.151.30.133 {"contentSize":1483,"warcFilename":"BL-NPLD-20220713084656856-19144-80~npld-heritrix3-worker-1~8443.warc.gz","warcFileOffset":380231951,"scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule","warcFileRecordLength":1623}
2022-07-13T09:59:06.496Z -5002 - https://healthwatchwarrington.co.uk/get-involved/ - https://healthwatchwarrington.co.uk/get-involved/ unknown #061 20220713095152869+433395 - tid:1673:http://www.healthwatchwarrington.co.uk/ launchTimestamp:20220713090000,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:08.275Z -5002 - https://www.create.ac.uk/blog/2020/04/02/coronavirus-covid-19-supporting-all-copyright-users/ - https://www.create.ac.uk/blog/2020/04/02/coronavirus-covid-19-supporting-all-copyright-users/ unknown #197 20220713095126026+462076 - tid:111249:https://www.create.ac.uk/blog/2020/04/02/coronavirus-covid-19-supporting-all-copyright-users/ 3t,launchTimestamp:20220713090000,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:14.900Z -5002 - https://charitysowhite.org/ - https://charitysowhite.org/ unknown #110 20220713095224100+410799 - tid:110568:https://charitysowhite.org/ 3t,launchTimestamp:20220713090000,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:15.863Z -5002 - https://www.computing.co.uk/content-hub/4033927/computing-tech-impact-hub/ - https://www.computing.co.uk/content-hub/4033927/computing-tech-impact-hub/ unknown #015 20220713095158708+436398 - tid:33009:http://www.computing.co.uk/ launchTimestamp:20220713090000,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:18.802Z -5002 - http://maxdundee.org.uk/news/ - http://maxdundee.org.uk/news/ unknown #058 20220713095209666+429136 - tid:134235:http://maxdundee.org.uk/ 3t,launchTimestamp:20220713090000,WebRenderStatus:500,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:19.263Z -5002 - https://www.cofeguildford.org.uk/about/welcome - https://www.cofeguildford.org.uk/about/welcome unknown #192 20220713095201563+437480 - tid:141836:https://www.cofeguildford.org.uk/ launchTimestamp:20220713090000,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:21.468Z -5002 - https://www.florence-nightingale.co.uk/resources/ - https://www.florence-nightingale.co.uk/resources/ unknown #184 20220713095120024+481415 - tid:5311:http://florence-nightingale.co.uk/ launchTimestamp:20220713090000,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:23.306Z -5002 - https://aessolar.co.uk/sitemap.xml - https://aessolar.co.uk/sitemap.xml unknown #004 20220713095216069+427236 - tid:128229:https://aessolar.co.uk/ launchTimestamp:20220713090000,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
2022-07-13T09:59:24.060Z -5002 - https://www.merthyr.gov.uk/coronavirus - https://www.merthyr.gov.uk/coronavirus unknown #050 20220713095208039+435822 - tid:108538:https://www.merthyr.gov.uk/coronavirus launchTimestamp:20220713090000,2t,WebRenderStatus:200,resetQuotas,WebRenderCount:1 {"warcPrefix":"BL-NPLD-WEBRENDER-frequent-npld-20220606093552","scopeDecision":"ACCEPT by rule #1 WatchedFileSurtPrefixedDecideRule"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment