Skip to content

Instantly share code, notes, and snippets.

@niels

niels/console.log Secret

Created December 9, 2015 17:25
Show Gist options
  • Save niels/d18cce1f26e026c3ca50 to your computer and use it in GitHub Desktop.
Save niels/d18cce1f26e026c3ca50 to your computer and use it in GitHub Desktop.
Download ZIP
Norconex ES committer never terminates
Raw
console.log
DEBUG [CollectorConfigLoader] Loading configuration file: test.xml
DEBUG [CrawlerConfigLoader] Crawler configuration loaded: test-crawler
INFO [AbstractCollectorConfig] Configuration loaded: id=test-collector; logsDir=./logs; progressDir=./progress
INFO [JobSuite] JEF work directory is: ./progress
INFO [JobSuite] JEF log manager is : FileLogManager
INFO [JobSuite] JEF job status store is : FileJobStatusStore
INFO [AbstractCollector] Suite of 1 crawler jobs created.
INFO [JobSuite] Initialization...
INFO [JobSuite] No previous execution detected.
DEBUG [FileLogManager] Log directory: /home/niels/Projects/UMF-crawler/vendor/test4/./logs
INFO [JobSuite] Starting execution.
INFO [AbstractCollector] Version: Norconex HTTP Collector 2.4.0-SNAPSHOT (Norconex Inc.)
INFO [AbstractCollector] Version: Norconex Collector Core 1.3.0 (Norconex Inc.)
INFO [AbstractCollector] Version: Norconex Importer 2.4.0 (Norconex Inc.)
INFO [AbstractCollector] Version: Norconex JEF 4.0.7 (Norconex Inc.)
INFO [AbstractCollector] Version: Norconex Committer Core 2.0.3 (Norconex Inc.)
INFO [AbstractCollector] Version: "ElasticsearchCommitter" version is undefined.
INFO [JobSuite] Running test-crawler: BEGIN (Wed Dec 09 18:24:12 CET 2015)
INFO [MapDBCrawlDataStore] Initializing reference store ./work/crawlstore/mapdb/test-crawler/
DEBUG [MapDBCrawlDataStore] ./work/crawlstore/mapdb/test-crawler/: New databases created.
INFO [MapDBCrawlDataStore] ./work/crawlstore/mapdb/test-crawler/: Done initializing databases.
INFO [HttpCrawler] test-crawler: RobotsTxt support: true
INFO [HttpCrawler] test-crawler: RobotsMeta support: true
INFO [HttpCrawler] test-crawler: Sitemap support: true
INFO [HttpCrawler] test-crawler: Canonical links support: true
INFO [HttpCrawler] test-crawler: User-Agent: <None specified>
DEBUG [HttpCrawler] It is recommended you identify yourself to web sites by specifying a user agent (https://en.wikipedia.org/wiki/User_agent)
INFO [SitemapStore] test-crawler: Initializing sitemap store...
DEBUG [SitemapStore] test-crawler: Sitemap store created.
INFO [SitemapStore] test-crawler: Done initializing sitemap store.
DEBUG [StandardRobotsTxtProvider$RobotData] Add filter from robots.txt: Robots.txt (Disallow:/warenkorb)
DEBUG [StandardRobotsTxtProvider$RobotData] Add filter from robots.txt: Robots.txt (Disallow:/info/agb)
DEBUG [StandardRobotsTxtProvider$RobotData] Add filter from robots.txt: Robots.txt (Disallow:/info/copyright)
DEBUG [StandardRobotsTxtProvider$RobotData] Add filter from robots.txt: Robots.txt (Disallow:/info/datenschutz)
DEBUG [StandardRobotsTxtProvider] Fetched and parsed robots.txt: https://biertastisch.de/robots.txt
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [StandardSitemapResolver] Sitemap locations: [https://biertastisch.de/sitemap.xml, https://biertastisch.de/sitemap_index.xml]
DEBUG [HttpCrawlerRedirectStrategy] URL redirect: https://biertastisch.de/sitemap.xml -> http://biertastisch.de/404
ERROR [StandardSitemapResolver] Could not obtain sitemap: https://biertastisch.de/sitemap.xml. Expected status code 200, but got 301
DEBUG [HttpCrawlerRedirectStrategy] URL redirect: https://biertastisch.de/sitemap_index.xml -> http://biertastisch.de/404
ERROR [StandardSitemapResolver] Could not obtain sitemap: https://biertastisch.de/sitemap_index.xml. Expected status code 200, but got 301
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de
INFO [HttpCrawler] 1 start URLs identified.
INFO [CrawlerEventManager] CRAWLER_STARTED (Subject: com.norconex.collector.http.crawler.HttpCrawler@140fc357)
INFO [AbstractCrawler] test-crawler: Crawling references...
DEBUG [AbstractCrawler] test-crawler: Crawler thread #1 started.
DEBUG [AbstractCrawler] test-crawler: Crawler thread #2 started.
DEBUG [AbstractCrawler] test-crawler: Processing reference: https://biertastisch.de
DEBUG [GenericDocumentFetcher] Fetching document: https://biertastisch.de
DEBUG [GenericDocumentFetcher] Encoded URI: https://biertastisch.de
INFO [CrawlerEventManager] DOCUMENT_FETCHED: https://biertastisch.de (Subject: HttpFetchResponse [crawlState=NEW, statusCode=200, reasonPhrase=OK])
DEBUG [CachedInputStream] Creating memory cache from cached stream.
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [StandardRobotsMetaProvider] No meta robots found for: https://biertastisch.de
INFO [CrawlerEventManager] CREATED_ROBOTS_META: https://biertastisch.de (Subject: none)
DEBUG [GenericLinkExtractor$Referer] DOCUMENT URL ----> https://biertastisch.de
DEBUG [GenericLinkExtractor$Referer] BASE RELATIVE -> https://biertastisch.de
DEBUG [GenericLinkExtractor$Referer] BASE ABSOLUTE -> https://biertastisch.de
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-kolner-bucht-paket Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-kolner-bucht-paket Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-kolner-bucht-paket Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-kolner-bucht-paket Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [AbstractCrawler] test-crawler: Processing reference: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
DEBUG [AbstractDelay] Thread pool-1-thread-2 sleeping for 2.52241537E12 seconds.
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/info/hilfe/wie-bestelle-ich
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Already queued: https://biertastisch.de/info/hilfe/wie-bestelle-ich
DEBUG [LinkExtractorStage] URL modified from "https://biertastisch.de/info/hilfe/wie-bestelle-ich#bestellung_versand" to "https://biertastisch.de/info/hilfe/wie-bestelle-ich
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png
DEBUG [URLCrawlScopeStrategy] Rejected domain for: https://twitter.com/biertastisch
DEBUG [LinkExtractorStage] URL not in crawl scope: https://twitter.com/biertastisch
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/impressum Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/impressum Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/impressum Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/impressum Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/info/impressum
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-frankenland-paket Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-frankenland-paket Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-frankenland-paket Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-frankenland-paket Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/das-frankenland-paket
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-abos Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-abos Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-abos Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-abos Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-abos
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/kontakt Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/kontakt Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/kontakt Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/kontakt Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/info/kontakt
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/datenschutz Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/datenschutz Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/datenschutz Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] REJECTED document reference (robots.txt). Reference=https://biertastisch.de/info/datenschutz Filter=Robots.txt (Disallow:/info/datenschutz)
INFO [CrawlerEventManager] REJECTED_FILTER: https://biertastisch.de/info/datenschutz (Subject: Robots.txt (Disallow:/info/datenschutz))
INFO [CrawlerEventManager] REJECTED_ROBOTS_TXT: https://biertastisch.de/info/datenschutz (Subject: RobotsTxt[filters={Robots.txt (Disallow:/warenkorb),Robots.txt (Disallow:/info/agb),Robots.txt (Disallow:/info/copyright),Robots.txt (Disallow:/info/datenschutz)},crawlDelay=-1.0,sitemapLocations={}])
DEBUG [Pipeline] Pipeline execution stopped at stage: com.norconex.collector.http.pipeline.queue.HttpQueuePipeline$RobotsTxtFiltersStage@24d24bbd
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/biere Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/biere Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/biere Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/biere Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/biere
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiMTg5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiMTg5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiMTg5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiMTg5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiMTg5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/ueber-uns Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/ueber-uns Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/ueber-uns Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/ueber-uns Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/info/ueber-uns
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png
DEBUG [LinkExtractorStage] URL modified from "https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png" to "https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/
DEBUG [URLCrawlScopeStrategy] Rejected domain for: https://www.facebook.com/biertastisch.de
DEBUG [LinkExtractorStage] URL not in crawl scope: https://www.facebook.com/biertastisch.de
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiNjUweDM2Nlx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiNjUweDM2Nlx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiNjUweDM2Nlx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiNjUweDM2Nlx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiNjUweDM2Nlx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/copyright Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/copyright Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] REJECTED document reference (robots.txt). Reference=https://biertastisch.de/info/copyright Filter=Robots.txt (Disallow:/info/copyright)
INFO [CrawlerEventManager] REJECTED_FILTER: https://biertastisch.de/info/copyright (Subject: Robots.txt (Disallow:/info/copyright))
INFO [CrawlerEventManager] REJECTED_ROBOTS_TXT: https://biertastisch.de/info/copyright (Subject: RobotsTxt[filters={Robots.txt (Disallow:/warenkorb),Robots.txt (Disallow:/info/agb),Robots.txt (Disallow:/info/copyright),Robots.txt (Disallow:/info/datenschutz)},crawlDelay=-1.0,sitemapLocations={}])
DEBUG [Pipeline] Pipeline execution stopped at stage: com.norconex.collector.http.pipeline.queue.HttpQueuePipeline$RobotsTxtFiltersStage@79e5d223
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-oberbayern-paket Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-oberbayern-paket Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-oberbayern-paket Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/das-oberbayern-paket Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/das-oberbayern-paket
DEBUG [URLCrawlScopeStrategy] Rejected domain for: https://herimedia.com
DEBUG [LinkExtractorStage] URL not in crawl scope: https://herimedia.com
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCIxODl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCIxODl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCIxODl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCIxODl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCIxODl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png
DEBUG [LinkExtractorStage] URL modified from "https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png" to "https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png
DEBUG [ReferenceFiltersStageUtil] REJECTED document reference (robots.txt). Reference=https://biertastisch.de/warenkorb Filter=Robots.txt (Disallow:/warenkorb)
INFO [CrawlerEventManager] REJECTED_FILTER: https://biertastisch.de/warenkorb (Subject: Robots.txt (Disallow:/warenkorb))
INFO [CrawlerEventManager] REJECTED_ROBOTS_TXT: https://biertastisch.de/warenkorb (Subject: RobotsTxt[filters={Robots.txt (Disallow:/warenkorb),Robots.txt (Disallow:/info/agb),Robots.txt (Disallow:/info/copyright),Robots.txt (Disallow:/info/datenschutz)},crawlDelay=-1.0,sitemapLocations={}])
DEBUG [Pipeline] Pipeline execution stopped at stage: com.norconex.collector.http.pipeline.queue.HttpQueuePipeline$RobotsTxtFiltersStage@7e35ab86
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/info/agb Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] REJECTED document reference (robots.txt). Reference=https://biertastisch.de/info/agb Filter=Robots.txt (Disallow:/info/agb)
INFO [CrawlerEventManager] REJECTED_FILTER: https://biertastisch.de/info/agb (Subject: Robots.txt (Disallow:/info/agb))
INFO [CrawlerEventManager] REJECTED_ROBOTS_TXT: https://biertastisch.de/info/agb (Subject: RobotsTxt[filters={Robots.txt (Disallow:/warenkorb),Robots.txt (Disallow:/info/agb),Robots.txt (Disallow:/info/copyright),Robots.txt (Disallow:/info/datenschutz)},crawlDelay=-1.0,sitemapLocations={}])
DEBUG [Pipeline] Pipeline execution stopped at stage: com.norconex.collector.http.pipeline.queue.HttpQueuePipeline$RobotsTxtFiltersStage@75902b25
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png
INFO [CrawlerEventManager] URLS_EXTRACTED: https://biertastisch.de (Subject: [https://biertastisch.de/info/hilfe/wie-bestelle-ich, https://biertastisch.de/assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png, https://biertastisch.de/assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png, https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiNjUweDM2Nlx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png, https://biertastisch.de/warenkorb, https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE4OXgxODkiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png, https://biertastisch.de/info/impressum, https://biertastisch.de/assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png, https://biertastisch.de/info/copyright, https://biertastisch.de/bier-pakete/das-kolner-bucht-paket, https://biertastisch.de/info/kontakt, https://biertastisch.de/info/datenschutz, https://biertastisch.de/biere, https://biertastisch.de/bier-pakete/das-frankenland-paket, https://biertastisch.de/assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png, https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png, https://biertastisch.de/bier-pakete, https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMDBfMjk3X0ZyYW5rZW5sYW5kLnBuZyJdLFsicCIsInRodW1iIiwiMTg5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/Frankenland.png, https://biertastisch.de/, https://biertastisch.de/assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png, https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png, https://biertastisch.de/bier-pakete/das-oberbayern-paket, https://biertastisch.de/bier-abos, https://biertastisch.de/media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDFfMDZfNzk1X29iZXJiYXllcm4ucG5nIl0sWyJwIiwidGh1bWIiLCIxODl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/oberbayern.png, https://biertastisch.de/info/ueber-uns, https://biertastisch.de/assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png, https://biertastisch.de/info/agb])
DEBUG [LastModifiedMetadataChecksummer] HTTP Header "Last-Modified" value: Mon, 23 Nov 2015 14:54:45 GMT
DEBUG [ChecksumStageUtil] ACCEPTED metadata checkum (new): Reference=https://biertastisch.de
DEBUG [GenericDocumentParserFactory] OCR parsing is disabled.
DEBUG [Importer] Parser "com.norconex.importer.parser.impl.FallbackParser" about to parse "https://biertastisch.de".
DEBUG [CachedInputStream] Creating new input stream from memory cache.
INFO [CrawlerEventManager] DOCUMENT_IMPORTED: https://biertastisch.de (Subject: com.norconex.importer.response.ImporterResponse@31969e5d)
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [MD5DocumentChecksummer] Document checksum from content: 0120c01700e4642cb4cc381f669cf1bd
DEBUG [ChecksumStageUtil] ACCEPTED document checkum (new): Reference=https://biertastisch.de
DEBUG [CachedInputStream] Creating new input stream from memory cache.
INFO [AbstractCommitter] Max queue size reached (1). Committing
INFO [AbstractFileQueueCommitter] Committing 1 files
INFO [ElasticsearchCommitter] Sending 1 operations to Elasticsearch.
INFO [InternalNode] [Alpha the Ultimate Mutant] version[1.5.0], pid[4287], build[5448160/2015-03-23T14:30:58Z]
INFO [InternalNode] [Alpha the Ultimate Mutant] initializing ...
DEBUG [InternalNode] [Alpha the Ultimate Mutant] using home [/home/niels/Projects/UMF-crawler/vendor/test4], config [/home/niels/Projects/UMF-crawler/vendor/test4/config], data [[/home/niels/Projects/UMF-crawler/vendor/test4/data]], logs [/home/niels/Projects/UMF-crawler/vendor/test4/logs], work [/home/niels/Projects/UMF-crawler/vendor/test4/work], plugins [/home/niels/Projects/UMF-crawler/vendor/test4/plugins]
DEBUG [FileSystemUtils] [Alpha the Ultimate Mutant] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Alpha the Ultimate Mutant] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
INFO [PluginsService] [Alpha the Ultimate Mutant] loaded [], sites []
DEBUG [LZFCompressor] using encoder [VanillaChunkDecoder] and decoder[{}]
DEBUG [ScriptModule] [Alpha the Ultimate Mutant] failed to load groovy
java.lang.ClassNotFoundException: groovy.lang.GroovyClassLoader
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.elasticsearch.script.ScriptModule.configure(ScriptModule.java:84)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204)
at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85)
at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at com.norconex.committer.elasticsearch.DefaultClientFactory.createClient(DefaultClientFactory.java:44)
at com.norconex.committer.elasticsearch.ElasticsearchCommitter.commitBatch(ElasticsearchCommitter.java:182)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.core.AbstractCommitter.commitIfReady(AbstractCommitter.java:146)
at com.norconex.committer.core.AbstractCommitter.add(AbstractCommitter.java:97)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:34)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:27)
at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)
at com.norconex.collector.http.crawler.HttpCrawler.executeCommitterPipeline(HttpCrawler.java:316)
at com.norconex.collector.core.crawler.AbstractCrawler.processImportResponse(AbstractCrawler.java:534)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:491)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:377)
at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:723)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
DEBUG [ScriptModule] [Alpha the Ultimate Mutant] failed to load lucene expressions
java.lang.ClassNotFoundException: org.apache.lucene.expressions.Expression
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.elasticsearch.script.ScriptModule.configure(ScriptModule.java:98)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204)
at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85)
at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at com.norconex.committer.elasticsearch.DefaultClientFactory.createClient(DefaultClientFactory.java:44)
at com.norconex.committer.elasticsearch.ElasticsearchCommitter.commitBatch(ElasticsearchCommitter.java:182)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.core.AbstractCommitter.commitIfReady(AbstractCommitter.java:146)
at com.norconex.committer.core.AbstractCommitter.add(AbstractCommitter.java:97)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:34)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:27)
at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)
at com.norconex.collector.http.crawler.HttpCrawler.executeCommitterPipeline(HttpCrawler.java:316)
at com.norconex.collector.core.crawler.AbstractCrawler.processImportResponse(AbstractCrawler.java:534)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:491)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:377)
at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:723)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
DEBUG [FileJobStatusStore] Status serialization directory: /home/niels/Projects/UMF-crawler/vendor/test4/./progress
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [generic], type [cached], keep_alive [30s]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [index], type [fixed], size [4], queue_size [200]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [bulk], type [fixed], size [4], queue_size [50]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [get], type [fixed], size [4], queue_size [1k]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [search], type [fixed], size [12], queue_size [1k]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [suggest], type [fixed], size [4], queue_size [1k]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [percolate], type [fixed], size [4], queue_size [1k]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [listener], type [fixed], size [2], queue_size [null]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [flush], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [merge], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [refresh], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [warmer], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [snapshot], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Alpha the Ultimate Mutant] creating thread_pool [optimize], type [fixed], size [1], queue_size [null]
DEBUG [JvmMonitorService] [Alpha the Ultimate Mutant] enabled [true], last_gc_enabled [false], interval [1s], gc_threshold [{old=GcThreshold{name='old', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, default=GcThreshold{name='default', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, young=GcThreshold{name='young', warnThreshold=1000, infoThreshold=700, debugThreshold=400}}]
DEBUG [OsService] [Alpha the Ultimate Mutant] Using probe [org.elasticsearch.monitor.os.JmxOsProbe@3687ef65] with refresh_interval [1s]
DEBUG [ProcessService] [Alpha the Ultimate Mutant] Using probe [org.elasticsearch.monitor.process.JmxProcessProbe@1d897725] with refresh_interval [1s]
DEBUG [JvmService] [Alpha the Ultimate Mutant] Using refresh_interval [1s]
DEBUG [NetworkService] [Alpha the Ultimate Mutant] Using probe [org.elasticsearch.monitor.network.JmxNetworkProbe@58ed737c] with refresh_interval [5s]
DEBUG [NetworkService] [Alpha the Ultimate Mutant] net_info
host [ordielite]
eth0 display_name [eth0]
address [/fe80:0:0:0:e611:5bff:fe2a:41c0%2] [/192.168.178.27]
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual [false]
lo display_name [lo]
address [/0:0:0:0:0:0:0:1%1] [/127.0.0.1]
mtu [65536] multicast [false] ptp [false] loopback [true] up [true] virtual [false]
DEBUG [FsService] [Alpha the Ultimate Mutant] Using probe [org.elasticsearch.monitor.fs.JmxFsProbe@71fa2083] with refresh_interval [1s]
DEBUG [NettyUtils] using gathering [true]
DEBUG [ElectMasterService] [Alpha the Ultimate Mutant] using minimum_master_nodes [-1]
DEBUG [MulticastZenPing] [Alpha the Ultimate Mutant] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
DEBUG [UnicastZenPing] [Alpha the Ultimate Mutant] using initial hosts [], with concurrent_connects [10]
DEBUG [ZenDiscovery] [Alpha the Ultimate Mutant] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
DEBUG [MasterFaultDetection] [Alpha the Ultimate Mutant] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
DEBUG [NodesFaultDetection] [Alpha the Ultimate Mutant] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
DEBUG [ScriptService] [Alpha the Ultimate Mutant] using script cache with max_size [100], expire [null]
DEBUG [GenericDocumentFetcher] Fetching document: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
DEBUG [GenericDocumentFetcher] Encoded URI: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
DEBUG [ThrottlingAllocationDecider] [Alpha the Ultimate Mutant] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
DEBUG [ClusterRebalanceAllocationDecider] [Alpha the Ultimate Mutant] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active]
DEBUG [ConcurrentRebalanceAllocationDecider] [Alpha the Ultimate Mutant] using [cluster_concurrent_rebalance] with [2]
DEBUG [RecoverySettings] [Alpha the Ultimate Mutant] using max_bytes_per_sec[40mb], concurrent_streams [3], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
DEBUG [LocalGatewayAllocator] [Alpha the Ultimate Mutant] using initial_shards [quorum], list_timeout [30s]
DEBUG [NettyHttpServerTransport] [Alpha the Ultimate Mutant] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[512kb->512kb], pipelining[true], pipelining_max_events[10000]
DEBUG [IndicesStore] [Alpha the Ultimate Mutant] using indices.store.throttle.type [MERGE], with index.store.throttle.max_bytes_per_sec [20mb]
DEBUG [IndexingMemoryController] [Alpha the Ultimate Mutant] using index_buffer_size [355.5mb], with min_shard_index_buffer_size [4mb], max_shard_index_buffer_size [512mb], shard_inactive_time [30m]
DEBUG [IndicesFilterCache] [Alpha the Ultimate Mutant] using [node] weighted filter cache with size [10%], actual_size [355.5mb], expire [null], clean_interval [1m]
DEBUG [IndicesFieldDataCache] [Alpha the Ultimate Mutant] using size [-1] [-1b], expire [null]
DEBUG [LocalGatewayMetaState] [Alpha the Ultimate Mutant] using gateway.local.auto_import_dangled [YES], gateway.local.delete_timeout [30s], with gateway.local.dangling_timeout [2h]
DEBUG [BulkUdpService] [Alpha the Ultimate Mutant] using enabled [false], host [null], port [9700-9800], bulk_actions [1000], bulk_size [5mb], flush_interval [5s], concurrent_requests [4]
INFO [InternalNode] [Alpha the Ultimate Mutant] initialized
INFO [InternalNode] [Alpha the Ultimate Mutant] starting ...
DEBUG [NettyInternalESLogger] Using select timeout of 500
DEBUG [NettyInternalESLogger] Epoll-bug workaround enabled = false
DEBUG [NettyTransport] [Alpha the Ultimate Mutant] using profile[default], worker_count[8], port[9300-9400], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb]
DEBUG [NettyTransport] [Alpha the Ultimate Mutant] Bound profile [default] to address [/0:0:0:0:0:0:0:0:9303]
INFO [TransportService] [Alpha the Ultimate Mutant] bound_address {inet[/0:0:0:0:0:0:0:0:9303]}, publish_address {inet[/192.168.178.27:9303]}
INFO [DiscoveryService] [Alpha the Ultimate Mutant] elasticsearch/1fKq9DuRTY6MBr8SMj4wgQ
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [initial_join]: execute
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [initial_join]: no change in cluster_state
INFO [CrawlerEventManager] DOCUMENT_FETCHED: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket (Subject: HttpFetchResponse [crawlState=NEW, statusCode=200, reasonPhrase=OK])
DEBUG [CachedInputStream] Creating memory cache from cached stream.
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [StandardRobotsMetaProvider] No meta robots found for: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
INFO [CrawlerEventManager] CREATED_ROBOTS_META: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket (Subject: none)
DEBUG [GenericLinkExtractor$Referer] DOCUMENT URL ----> https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
DEBUG [GenericLinkExtractor$Referer] BASE RELATIVE -> https://biertastisch.de/bier-pakete/
DEBUG [GenericLinkExtractor$Referer] BASE ABSOLUTE -> https://biertastisch.de
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/hellers-hefe-hell_hellers-koelsch.jpg
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-hefeweizen Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-hefeweizen Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-hefeweizen Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-hefeweizen Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../biere/sunner/sunner-hefeweizen
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjZfNDlfNjczX211ZWhsZW5fbWFsei5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-malz.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjZfNDlfNjczX211ZWhsZW5fbWFsei5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-malz.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjZfNDlfNjczX211ZWhsZW5fbWFsei5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-malz.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjZfNDlfNjczX211ZWhsZW5fbWFsei5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-malz.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjZfNDlfNjczX211ZWhsZW5fbWFsei5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-malz.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzRfNDNfNTkyX3N1ZW5uZXJfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-koelsch.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzRfNDNfNTkyX3N1ZW5uZXJfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-koelsch.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzRfNDNfNTkyX3N1ZW5uZXJfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-koelsch.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzRfNDNfNTkyX3N1ZW5uZXJfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-koelsch.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzRfNDNfNTkyX3N1ZW5uZXJfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-koelsch.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/suenner-hefe-hell_suenner-koelsch.jpg
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/agb Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/agb Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/agb Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/agb Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../info/agb
DEBUG [URLCrawlScopeStrategy] Rejected domain for: https://twitter.com/biertastisch
DEBUG [LinkExtractorStage] URL not in crawl scope: https://twitter.com/biertastisch
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/mueheln-koelsch_muehlen-malz.jpg
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/datenschutz Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/datenschutz Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/datenschutz Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/datenschutz Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../info/datenschutz
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-kolsch Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-kolsch Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-kolsch Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-kolsch Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../biere/heller/hellers-kolsch
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzFfMzJfNjIxX211ZWhsZW5fa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-koelsch.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzFfMzJfNjIxX211ZWhsZW5fa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-koelsch.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzFfMzJfNjIxX211ZWhsZW5fa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-koelsch.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzFfMzJfNjIxX211ZWhsZW5fa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-koelsch.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzFfMzJfNjIxX211ZWhsZW5fa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-koelsch.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../bier-pakete Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../bier-pakete Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../bier-pakete Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../bier-pakete Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../bier-pakete
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/kölner bucht.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/kölner bucht.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/kölner bucht.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/kölner bucht.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/k%C3%B6lner%20bucht.png
DEBUG [LinkExtractorStage] URL modified from "https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/kölner bucht.png" to "https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/k%C3%B6lner%20bucht.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzNfMDRfMzEyX3N1ZW5uZXJfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/suenner-hefe-hell.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzNfMDRfMzEyX3N1ZW5uZXJfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/suenner-hefe-hell.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzNfMDRfMzEyX3N1ZW5uZXJfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/suenner-hefe-hell.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzNfMDRfMzEyX3N1ZW5uZXJfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/suenner-hefe-hell.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzNfMDRfMzEyX3N1ZW5uZXJfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/suenner-hefe-hell.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/muhlen-kolsch Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/muhlen-kolsch Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/muhlen-kolsch Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/muhlen-kolsch Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../biere/malzmuhle/muhlen-kolsch
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../biere
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-hefe-hell_suenner-koelsch.jpg Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-hefe-hell_suenner-koelsch.jpg
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../warenkorb Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../warenkorb Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../warenkorb Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../warenkorb Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../warenkorb
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/impressum Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/impressum Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/impressum Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/impressum Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../info/impressum
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-weizen Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-weizen Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-weizen Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/heller/hellers-weizen Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../biere/heller/hellers-weizen
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/copyright Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/copyright Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/copyright Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/copyright Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../info/copyright
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png
DEBUG [LinkExtractorStage] URL modified from "https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/kölner bucht.png" to "https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich#bestellung_versand Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Already queued: https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich
DEBUG [LinkExtractorStage] URL modified from "https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich#bestellung_versand" to "https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/ueber-uns Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/ueber-uns Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/ueber-uns Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/ueber-uns Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../info/ueber-uns
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-kolsch Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-kolsch Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-kolsch Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/sunner/sunner-kolsch Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../biere/sunner/sunner-kolsch
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/kochsches-malzbier Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/kochsches-malzbier Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/kochsches-malzbier Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../biere/malzmuhle/kochsches-malzbier Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../biere/malzmuhle/kochsches-malzbier
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/ Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Already queued: https://biertastisch.de/
DEBUG [URLCrawlScopeStrategy] Rejected domain for: https://www.facebook.com/biertastisch.de
DEBUG [LinkExtractorStage] URL not in crawl scope: https://www.facebook.com/biertastisch.de
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-hefe-hell_hellers-koelsch.jpg Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-hefe-hell_hellers-koelsch.jpg
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMTRfMzZfMTE5X2hlbGxlcnNfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-koelsch.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMTRfMzZfMTE5X2hlbGxlcnNfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-koelsch.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMTRfMzZfMTE5X2hlbGxlcnNfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-koelsch.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMTRfMzZfMTE5X2hlbGxlcnNfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-koelsch.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMTRfMzZfMTE5X2hlbGxlcnNfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-koelsch.png
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjBfNTNfMTUzX2hlbGxlcnNfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/hellers-hefe-hell.png Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjBfNTNfMTUzX2hlbGxlcnNfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/hellers-hefe-hell.png Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjBfNTNfMTUzX2hlbGxlcnNfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/hellers-hefe-hell.png Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjBfNTNfMTUzX2hlbGxlcnNfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/hellers-hefe-hell.png Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjBfNTNfMTUzX2hlbGxlcnNfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/hellers-hefe-hell.png
DEBUG [URLCrawlScopeStrategy] Rejected domain for: https://herimedia.com
DEBUG [LinkExtractorStage] URL not in crawl scope: https://herimedia.com
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/kontakt Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/kontakt Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/kontakt Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../info/kontakt Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../info/kontakt
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCIxOTIweDEyMDBcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/warenkorb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCIxOTIweDEyMDBcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/info/agb)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCIxOTIweDEyMDBcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/info/copyright)
DEBUG [ReferenceFiltersStageUtil] ACCEPTED document reference (robots.txt). Reference=https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCIxOTIweDEyMDBcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/mueheln-koelsch_muehlen-malz.jpg Filter=Robots.txt (Disallow:/info/datenschutz)
DEBUG [QueueReferenceStage] Queued for processing: https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCIxOTIweDEyMDBcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/mueheln-koelsch_muehlen-malz.jpg
INFO [CrawlerEventManager] URLS_EXTRACTED: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket (Subject: [https://biertastisch.de/bier-pakete/../assets/frontend/logo-wide-neon-300x50-a068a3ddfea157d97186decef6379840.png, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzFfMzJfNjIxX211ZWhsZW5fa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-koelsch.png, https://biertastisch.de/bier-pakete/../info/copyright, https://biertastisch.de/bier-pakete/../biere/heller/hellers-weizen, https://biertastisch.de/bier-pakete/../assets/frontend/logo-177x177-197ace0eee8f22b3271a6448901df6e3.png, https://biertastisch.de/bier-pakete/../biere/sunner/sunner-hefeweizen, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/k%C3%B6lner%20bucht.png, https://biertastisch.de/bier-pakete/../info/kontakt, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMTRfMzZfMTE5X2hlbGxlcnNfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-koelsch.png, https://biertastisch.de/bier-pakete/../assets/frontend/startssl-128x32-40d3920f727f7ca5a990fb19a1970ed3.png, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjZfNDlfNjczX211ZWhsZW5fbWFsei5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/muehlen-malz.png, https://biertastisch.de/bier-pakete/../info/agb, https://biertastisch.de/bier-pakete/../biere/heller/hellers-kolsch, https://biertastisch.de/bier-pakete/../assets/frontend/logo_siegel-100x104-ef9db4b6a9a4682ef1eabc61b039bc25.png, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/hellers-hefe-hell_hellers-koelsch.jpg, https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/favicon-16x16-16a0b3a1c5ff2d6c9a22e9a2b93255d9.png, https://biertastisch.de/bier-pakete/../biere/malzmuhle/kochsches-malzbier, https://biertastisch.de/bier-pakete/../warenkorb, https://biertastisch.de/bier-pakete/../bier-pakete, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTI2X2hlbGxlcnNfaGVmZV9oZWxsX2hlbGxlcnNfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/hellers-hefe-hell_hellers-koelsch.jpg, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/suenner-hefe-hell_suenner-koelsch.jpg, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCIxOTIweDEyMDBcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/mueheln-koelsch_muehlen-malz.jpg, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMDdfMzRfNTM1X211ZWhlbG5fa29lbHNjaF9tdWVobGVuX21hbHouanBnIl0sWyJwIiwidGh1bWIiLCI2NTB4MzY2XHUwMDNFIl0sWyJwIiwid2F0ZXJtYXJrIl1d/mueheln-koelsch_muehlen-malz.jpg, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzRfNDNfNTkyX3N1ZW5uZXJfa29lbHNjaC5wbmciXSxbInAiLCJ0aHVtYiIsIjY5eDE4OSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-koelsch.png, https://biertastisch.de/bier-pakete/../biere/malzmuhle/muhlen-kolsch, https://biertastisch.de/, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMzNfMDRfMzEyX3N1ZW5uZXJfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/suenner-hefe-hell.png, https://biertastisch.de/bier-pakete/../info/impressum, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMjgvMThfMjBfNTNfMTUzX2hlbGxlcnNfaGVmZV9oZWxsLnBuZyJdLFsicCIsInRodW1iIiwiNjl4MTg5Il0sWyJwIiwid2F0ZXJtYXJrIl1d/hellers-hefe-hell.png, https://biertastisch.de/bier-pakete/../info/ueber-uns, https://biertastisch.de/bier-pakete/../info/hilfe/wie-bestelle-ich, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDQvMzAvMjBfMDBfMjFfODU5X2tfbG5lcl9idWNodC5wbmciXSxbInAiLCJ0aHVtYiIsIjY1MHgzNjZcdTAwM0UiXSxbInAiLCJ3YXRlcm1hcmsiXV0/k%C3%B6lner%20bucht.png, https://biertastisch.de/bier-pakete/../biere/sunner/sunner-kolsch, https://biertastisch.de/bier-pakete/../info/datenschutz, https://biertastisch.de/bier-pakete/../assets/frontend/herimedia/cologne_cathedral-16x16-52dc3f8462f25f080201dd126c2b1619.png, https://biertastisch.de/bier-pakete/../biere, https://biertastisch.de/bier-pakete/../media/W1siZiIsIjIwMTMvMDYvMDcvMTRfMTBfMjBfOTEwX3N1ZW5uZXJfaGVmZV9oZWxsX3N1ZW5uZXJfa29lbHNjaC5qcGciXSxbInAiLCJ0aHVtYiIsIjE5MjB4MTIwMFx1MDAzRSJdLFsicCIsIndhdGVybWFyayJdXQ/suenner-hefe-hell_suenner-koelsch.jpg])
DEBUG [LastModifiedMetadataChecksummer] HTTP Header "Last-Modified" value: Mon, 23 Nov 2015 14:54:45 GMT
DEBUG [ChecksumStageUtil] ACCEPTED metadata checkum (new): Reference=https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
DEBUG [Importer] Parser "com.norconex.importer.parser.impl.FallbackParser" about to parse "https://biertastisch.de/bier-pakete/das-kolner-bucht-paket".
DEBUG [CachedInputStream] Creating new input stream from memory cache.
INFO [CrawlerEventManager] DOCUMENT_IMPORTED: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket (Subject: com.norconex.importer.response.ImporterResponse@7bbc07a4)
DEBUG [CachedInputStream] Creating new input stream from memory cache.
DEBUG [MD5DocumentChecksummer] Document checksum from content: 1aa50484b00aee978ffb7e20a9604017
DEBUG [ChecksumStageUtil] ACCEPTED document checkum (new): Reference=https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
DEBUG [CachedInputStream] Creating new input stream from memory cache.
INFO [AbstractCommitter] Max queue size reached (1). Committing
INFO [AbstractFileQueueCommitter] Committing 1 files
INFO [ElasticsearchCommitter] Sending 1 operations to Elasticsearch.
INFO [InternalNode] [Diablo] version[1.5.0], pid[4287], build[5448160/2015-03-23T14:30:58Z]
INFO [InternalNode] [Diablo] initializing ...
DEBUG [InternalNode] [Diablo] using home [/home/niels/Projects/UMF-crawler/vendor/test4], config [/home/niels/Projects/UMF-crawler/vendor/test4/config], data [[/home/niels/Projects/UMF-crawler/vendor/test4/data]], logs [/home/niels/Projects/UMF-crawler/vendor/test4/logs], work [/home/niels/Projects/UMF-crawler/vendor/test4/work], plugins [/home/niels/Projects/UMF-crawler/vendor/test4/plugins]
DEBUG [FileSystemUtils] [Diablo] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Diablo] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
INFO [PluginsService] [Diablo] loaded [], sites []
DEBUG [ScriptModule] [Diablo] failed to load groovy
java.lang.ClassNotFoundException: groovy.lang.GroovyClassLoader
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.elasticsearch.script.ScriptModule.configure(ScriptModule.java:84)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204)
at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85)
at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at com.norconex.committer.elasticsearch.DefaultClientFactory.createClient(DefaultClientFactory.java:44)
at com.norconex.committer.elasticsearch.ElasticsearchCommitter.commitBatch(ElasticsearchCommitter.java:182)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.core.AbstractCommitter.commitIfReady(AbstractCommitter.java:146)
at com.norconex.committer.core.AbstractCommitter.add(AbstractCommitter.java:97)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:34)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:27)
at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)
at com.norconex.collector.http.crawler.HttpCrawler.executeCommitterPipeline(HttpCrawler.java:316)
at com.norconex.collector.core.crawler.AbstractCrawler.processImportResponse(AbstractCrawler.java:534)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:491)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:377)
at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:723)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
DEBUG [ScriptModule] [Diablo] failed to load lucene expressions
java.lang.ClassNotFoundException: org.apache.lucene.expressions.Expression
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.elasticsearch.script.ScriptModule.configure(ScriptModule.java:98)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204)
at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85)
at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at com.norconex.committer.elasticsearch.DefaultClientFactory.createClient(DefaultClientFactory.java:44)
at com.norconex.committer.elasticsearch.ElasticsearchCommitter.commitBatch(ElasticsearchCommitter.java:182)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.core.AbstractCommitter.commitIfReady(AbstractCommitter.java:146)
at com.norconex.committer.core.AbstractCommitter.add(AbstractCommitter.java:97)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:34)
at com.norconex.collector.core.pipeline.committer.CommitModuleStage.execute(CommitModuleStage.java:27)
at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)
at com.norconex.collector.http.crawler.HttpCrawler.executeCommitterPipeline(HttpCrawler.java:316)
at com.norconex.collector.core.crawler.AbstractCrawler.processImportResponse(AbstractCrawler.java:534)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:491)
at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:377)
at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:723)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
DEBUG [ThreadPool] [Diablo] creating thread_pool [generic], type [cached], keep_alive [30s]
DEBUG [ThreadPool] [Diablo] creating thread_pool [index], type [fixed], size [4], queue_size [200]
DEBUG [ThreadPool] [Diablo] creating thread_pool [bulk], type [fixed], size [4], queue_size [50]
DEBUG [ThreadPool] [Diablo] creating thread_pool [get], type [fixed], size [4], queue_size [1k]
DEBUG [ThreadPool] [Diablo] creating thread_pool [search], type [fixed], size [12], queue_size [1k]
DEBUG [ThreadPool] [Diablo] creating thread_pool [suggest], type [fixed], size [4], queue_size [1k]
DEBUG [ThreadPool] [Diablo] creating thread_pool [percolate], type [fixed], size [4], queue_size [1k]
DEBUG [ThreadPool] [Diablo] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m]
DEBUG [ThreadPool] [Diablo] creating thread_pool [listener], type [fixed], size [2], queue_size [null]
DEBUG [ThreadPool] [Diablo] creating thread_pool [flush], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Diablo] creating thread_pool [merge], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Diablo] creating thread_pool [refresh], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Diablo] creating thread_pool [warmer], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Diablo] creating thread_pool [snapshot], type [scaling], min [1], size [2], keep_alive [5m]
DEBUG [ThreadPool] [Diablo] creating thread_pool [optimize], type [fixed], size [1], queue_size [null]
DEBUG [JvmMonitorService] [Diablo] enabled [true], last_gc_enabled [false], interval [1s], gc_threshold [{old=GcThreshold{name='old', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, default=GcThreshold{name='default', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, young=GcThreshold{name='young', warnThreshold=1000, infoThreshold=700, debugThreshold=400}}]
DEBUG [OsService] [Diablo] Using probe [org.elasticsearch.monitor.os.JmxOsProbe@5360fc0b] with refresh_interval [1s]
DEBUG [ProcessService] [Diablo] Using probe [org.elasticsearch.monitor.process.JmxProcessProbe@11bebf14] with refresh_interval [1s]
DEBUG [JvmService] [Diablo] Using refresh_interval [1s]
DEBUG [NetworkService] [Diablo] Using probe [org.elasticsearch.monitor.network.JmxNetworkProbe@49bc8e20] with refresh_interval [5s]
DEBUG [NetworkService] [Diablo] net_info
host [ordielite]
eth0 display_name [eth0]
address [/fe80:0:0:0:e611:5bff:fe2a:41c0%2] [/192.168.178.27]
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual [false]
lo display_name [lo]
address [/0:0:0:0:0:0:0:1%1] [/127.0.0.1]
mtu [65536] multicast [false] ptp [false] loopback [true] up [true] virtual [false]
DEBUG [FsService] [Diablo] Using probe [org.elasticsearch.monitor.fs.JmxFsProbe@4e0a8d49] with refresh_interval [1s]
DEBUG [ElectMasterService] [Diablo] using minimum_master_nodes [-1]
DEBUG [MulticastZenPing] [Diablo] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
DEBUG [UnicastZenPing] [Diablo] using initial hosts [], with concurrent_connects [10]
DEBUG [ZenDiscovery] [Diablo] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
DEBUG [MasterFaultDetection] [Diablo] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
DEBUG [NodesFaultDetection] [Diablo] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
DEBUG [ScriptService] [Diablo] using script cache with max_size [100], expire [null]
DEBUG [ThrottlingAllocationDecider] [Diablo] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
DEBUG [ClusterRebalanceAllocationDecider] [Diablo] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active]
DEBUG [ConcurrentRebalanceAllocationDecider] [Diablo] using [cluster_concurrent_rebalance] with [2]
DEBUG [RecoverySettings] [Diablo] using max_bytes_per_sec[40mb], concurrent_streams [3], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
DEBUG [LocalGatewayAllocator] [Diablo] using initial_shards [quorum], list_timeout [30s]
DEBUG [NettyHttpServerTransport] [Diablo] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[512kb->512kb], pipelining[true], pipelining_max_events[10000]
DEBUG [IndicesStore] [Diablo] using indices.store.throttle.type [MERGE], with index.store.throttle.max_bytes_per_sec [20mb]
DEBUG [IndexingMemoryController] [Diablo] using index_buffer_size [355.5mb], with min_shard_index_buffer_size [4mb], max_shard_index_buffer_size [512mb], shard_inactive_time [30m]
DEBUG [IndicesFilterCache] [Diablo] using [node] weighted filter cache with size [10%], actual_size [355.5mb], expire [null], clean_interval [1m]
DEBUG [IndicesFieldDataCache] [Diablo] using size [-1] [-1b], expire [null]
DEBUG [LocalGatewayMetaState] [Diablo] using gateway.local.auto_import_dangled [YES], gateway.local.delete_timeout [30s], with gateway.local.dangling_timeout [2h]
DEBUG [BulkUdpService] [Diablo] using enabled [false], host [null], port [9700-9800], bulk_actions [1000], bulk_size [5mb], flush_interval [5s], concurrent_requests [4]
INFO [InternalNode] [Diablo] initialized
INFO [InternalNode] [Diablo] starting ...
DEBUG [NettyTransport] [Diablo] using profile[default], worker_count[8], port[9300-9400], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb]
DEBUG [NettyTransport] [Diablo] Bound profile [default] to address [/0:0:0:0:0:0:0:0:9304]
INFO [TransportService] [Diablo] bound_address {inet[/0:0:0:0:0:0:0:0:9304]}, publish_address {inet[/192.168.178.27:9304]}
INFO [DiscoveryService] [Diablo] elasticsearch/XLavJvn2TeC_jJgtbqexkw
DEBUG [InternalClusterService$UpdateTask] [Diablo] processing [initial_join]: execute
DEBUG [InternalClusterService$UpdateTask] [Diablo] processing [initial_join]: no change in cluster_state
DEBUG [ZenDiscovery] [Alpha the Ultimate Mutant] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]], id[51], master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]], hasJoinedOnce [true], cluster_name[elasticsearch]}
DEBUG [NettyTransport] [Alpha the Ultimate Mutant] connected to node [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]]
DEBUG [PublishClusterStateAction$PublishClusterStateRequestHandler] [Alpha the Ultimate Mutant] received cluster state version 43
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]: execute
DEBUG [MasterFaultDetection] [Alpha the Ultimate Mutant] [master] restarting fault detection against master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]], reason [new cluster state received and we are monitoring the wrong master [null]]
DEBUG [ZenDiscovery$10] [Alpha the Ultimate Mutant] got first state from fresh master [o73B3EdZQSaYRomBwSeP7A]
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] cluster state updated, version [43], source [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]
INFO [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] detected_master [Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]], added {[Belathauzer][HGPuMZyJTK2oLzLgIXnAKw][ordielite][inet[/192.168.178.27:9302]]{client=true, data=false},[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]],[Sweetface][X14YYANjSf25y7d5W8wAlg][ordielite][inet[/192.168.178.27:9301]]{client=true, data=false},}, reason: zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] set local cluster state to version 43
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]: done applying updated cluster_state (version: 43)
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [finalize_join ([Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]])]: execute
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [finalize_join ([Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]])]: no change in cluster_state
INFO [HttpServer] [Alpha the Ultimate Mutant] bound_address {inet[/0:0:0:0:0:0:0:0:9203]}, publish_address {inet[/192.168.178.27:9203]}
INFO [InternalNode] [Alpha the Ultimate Mutant] started
INFO [ElasticsearchCommitter] Done sending operations to Elasticsearch.
DEBUG [AbstractFileQueueCommitter] Deleted 4 empty directories under committer-queue
INFO [CrawlerEventManager] DOCUMENT_COMMITTED_ADD: https://biertastisch.de (Subject: com.norconex.committer.elasticsearch.ElasticsearchCommitter@1667f48[queueSize=1,docCount=5,queue=FileSystemCommitter[directory=committer-queue],commitBatchSize=1,maxRetries=0,maxRetryWait=0,operations=[],docCount=0,targetReferenceField=<null>,sourceReferenceField=<null>,keepSourceReferenceField=false,targetContentField=content,sourceContentField=<null>,keepSourceContentField=false,clientFactory=com.norconex.committer.elasticsearch.DefaultClientFactory@702e50af,client=org.elasticsearch.client.node.NodeClient@70b34ab2,clusterName=<null>,indexName=crawler_test,typeName=test_doc])
DEBUG [FileJobStatusStore] Writing status file: /home/niels/Projects/UMF-crawler/vendor/test4/./progress/latest/status/test-crawler__test-crawler.job
DEBUG [FileJobStatusStore] Writing status file: /home/niels/Projects/UMF-crawler/vendor/test4/./progress/latest/status/test-crawler__test-crawler.job
INFO [AbstractCrawler] test-crawler: 1% completed (1 processed/59 total)
DEBUG [AbstractCrawler] test-crawler: 0:00:06.600 to process: https://biertastisch.de
INFO [AbstractCrawler] test-crawler: Maximum documents reached: 1
DEBUG [ZenDiscovery] [Diablo] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]], id[52], master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]], hasJoinedOnce [true], cluster_name[elasticsearch]}
DEBUG [NettyTransport] [Diablo] connected to node [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]]
DEBUG [PublishClusterStateAction$PublishClusterStateRequestHandler] [Alpha the Ultimate Mutant] received cluster state version 44
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]: execute
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] cluster state updated, version [44], source [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]
INFO [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] added {[Diablo][XLavJvn2TeC_jJgtbqexkw][ordielite][inet[/192.168.178.27:9304]]{client=true, data=false},}, reason: zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] set local cluster state to version 44
DEBUG [InternalClusterService$UpdateTask] [Alpha the Ultimate Mutant] processing [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]: done applying updated cluster_state (version: 44)
DEBUG [PublishClusterStateAction$PublishClusterStateRequestHandler] [Diablo] received cluster state version 44
DEBUG [InternalClusterService$UpdateTask] [Diablo] processing [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]: execute
DEBUG [MasterFaultDetection] [Diablo] [master] restarting fault detection against master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]], reason [new cluster state received and we are monitoring the wrong master [null]]
DEBUG [ZenDiscovery$10] [Diablo] got first state from fresh master [o73B3EdZQSaYRomBwSeP7A]
DEBUG [InternalClusterService$UpdateTask] [Diablo] cluster state updated, version [44], source [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]
INFO [InternalClusterService$UpdateTask] [Diablo] detected_master [Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]], added {[Belathauzer][HGPuMZyJTK2oLzLgIXnAKw][ordielite][inet[/192.168.178.27:9302]]{client=true, data=false},[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]],[Sweetface][X14YYANjSf25y7d5W8wAlg][ordielite][inet[/192.168.178.27:9301]]{client=true, data=false},[Alpha the Ultimate Mutant][1fKq9DuRTY6MBr8SMj4wgQ][ordielite][inet[/192.168.178.27:9303]]{client=true, data=false},}, reason: zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])
DEBUG [InternalClusterService$UpdateTask] [Diablo] set local cluster state to version 44
DEBUG [InternalClusterService$UpdateTask] [Diablo] processing [zen-disco-receive(from master [[Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]]])]: done applying updated cluster_state (version: 44)
DEBUG [InternalClusterService$UpdateTask] [Diablo] processing [finalize_join ([Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]])]: execute
DEBUG [InternalClusterService$UpdateTask] [Diablo] processing [finalize_join ([Book][o73B3EdZQSaYRomBwSeP7A][ordielite][inet[/192.168.178.27:9300]])]: no change in cluster_state
INFO [HttpServer] [Diablo] bound_address {inet[/0:0:0:0:0:0:0:0:9204]}, publish_address {inet[/192.168.178.27:9204]}
INFO [InternalNode] [Diablo] started
INFO [ElasticsearchCommitter] Done sending operations to Elasticsearch.
DEBUG [AbstractFileQueueCommitter] Deleted 0 empty directories under committer-queue
INFO [CrawlerEventManager] DOCUMENT_COMMITTED_ADD: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket (Subject: com.norconex.committer.elasticsearch.ElasticsearchCommitter@1667f48[queueSize=1,docCount=5,queue=FileSystemCommitter[directory=committer-queue],commitBatchSize=1,maxRetries=0,maxRetryWait=0,operations=[],docCount=0,targetReferenceField=<null>,sourceReferenceField=<null>,keepSourceReferenceField=false,targetContentField=content,sourceContentField=<null>,keepSourceContentField=false,clientFactory=com.norconex.committer.elasticsearch.DefaultClientFactory@702e50af,client=org.elasticsearch.client.node.NodeClient@23558b01,clusterName=<null>,indexName=crawler_test,typeName=test_doc])
DEBUG [FileJobStatusStore] Writing status file: /home/niels/Projects/UMF-crawler/vendor/test4/./progress/latest/status/test-crawler__test-crawler.job
DEBUG [FileJobStatusStore] Writing status file: /home/niels/Projects/UMF-crawler/vendor/test4/./progress/latest/status/test-crawler__test-crawler.job
DEBUG [AbstractCrawler] test-crawler: 0:00:06.479 to process: https://biertastisch.de/bier-pakete/das-kolner-bucht-paket
INFO [AbstractCrawler] test-crawler: Maximum documents reached: 1
INFO [AbstractCrawler] test-crawler: Max documents reached. Not reprocessing orphans (if any).
INFO [AbstractCrawler] test-crawler: Crawler finishing: committing documents.
INFO [AbstractFileQueueCommitter] Committing 1 files
INFO [ElasticsearchCommitter] Sending 1 operations to Elasticsearch.
INFO [ElasticsearchCommitter] Done sending operations to Elasticsearch.
DEBUG [AbstractFileQueueCommitter] Deleted 0 empty directories under committer-queue
INFO [AbstractCrawler] test-crawler: 2 reference(s) processed.
DEBUG [AbstractCrawler] test-crawler: Removing empty directories
INFO [CrawlerEventManager] CRAWLER_FINISHED (Subject: com.norconex.collector.http.crawler.HttpCrawler@140fc357)
INFO [AbstractCrawler] test-crawler: Crawler completed.
INFO [AbstractCrawler] test-crawler: Crawler executed in 9 seconds.
INFO [MapDBCrawlDataStore] Closing reference store: ./work/crawlstore/mapdb/test-crawler/
DEBUG [FileJobStatusStore] Writing status file: /home/niels/Projects/UMF-crawler/vendor/test4/./progress/latest/status/test-crawler__test-crawler.job
INFO [JobSuite] Running test-crawler: END (Wed Dec 09 18:24:12 CET 2015)
DEBUG [FileSystemUtils] [Diablo] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Alpha the Ultimate Mutant] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Diablo] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Alpha the Ultimate Mutant] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Diablo] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Alpha the Ultimate Mutant] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Diablo] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Alpha the Ultimate Mutant] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Diablo] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
DEBUG [FileSystemUtils] [Alpha the Ultimate Mutant] [/home/niels/Projects/UMF-crawler/vendor/test4/plugins] directory does not exist.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment