Skip to content

Instantly share code, notes, and snippets.

@raulk

raulk/gist:7d6713aa7b3d21ecaacd

Created August 24, 2015 10:58
Show Gist options
  • Save raulk/7d6713aa7b3d21ecaacd to your computer and use it in GitHub Desktop.
Save raulk/7d6713aa7b3d21ecaacd to your computer and use it in GitHub Desktop.
Download ZIP
wget spider output for ignite.apache.org
Raw
gistfile1.txt
[raul@~/kk$] wget -S --reject-regex ".*(java|scala)doc.*" -r -l4 --spider -D apache.org http://ignite.apache.org 2>&1 | grep -E '(\-\-|HTTP\/1\.1)'
--2015-08-24 11:56:46-- http://ignite.apache.org/
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:47-- http://ignite.apache.org/
HTTP/1.1 200 OK
--2015-08-24 11:56:47-- http://ignite.apache.org/robots.txt
HTTP/1.1 404 Not Found
--2015-08-24 11:56:48-- http://ignite.apache.org/css/all.css
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:56:48-- http://ignite.apache.org/index.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:48-- http://ignite.apache.org/index.html
HTTP/1.1 200 OK
--2015-08-24 11:56:49-- http://ignite.apache.org/images/logo.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:56:49-- http://ignite.apache.org/features.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:49-- http://ignite.apache.org/features.html
HTTP/1.1 200 OK
--2015-08-24 11:56:50-- http://ignite.apache.org/features/datagrid.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:50-- http://ignite.apache.org/features/datagrid.html
HTTP/1.1 200 OK
--2015-08-24 11:56:50-- http://ignite.apache.org/features/computegrid.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:51-- http://ignite.apache.org/features/computegrid.html
HTTP/1.1 200 OK
--2015-08-24 11:56:51-- http://ignite.apache.org/features/streaming.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:51-- http://ignite.apache.org/features/streaming.html
HTTP/1.1 200 OK
--2015-08-24 11:56:52-- http://ignite.apache.org/features/datastructures.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:52-- http://ignite.apache.org/features/datastructures.html
HTTP/1.1 200 OK
--2015-08-24 11:56:52-- http://ignite.apache.org/features/messaging.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:52-- http://ignite.apache.org/features/messaging.html
HTTP/1.1 200 OK
--2015-08-24 11:56:53-- http://ignite.apache.org/features/servicegrid.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:53-- http://ignite.apache.org/features/servicegrid.html
HTTP/1.1 200 OK
--2015-08-24 11:56:53-- http://ignite.apache.org/features/igniterdd.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:54-- http://ignite.apache.org/features/igniterdd.html
HTTP/1.1 200 OK
--2015-08-24 11:56:54-- http://ignite.apache.org/features/igfs.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:54-- http://ignite.apache.org/features/igfs.html
HTTP/1.1 200 OK
--2015-08-24 11:56:54-- http://ignite.apache.org/features/mapreduce.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:55-- http://ignite.apache.org/features/mapreduce.html
HTTP/1.1 200 OK
--2015-08-24 11:56:55-- http://ignite.apache.org/features/clientprotos.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:55-- http://ignite.apache.org/features/clientprotos.html
HTTP/1.1 200 OK
--2015-08-24 11:56:55-- http://ignite.apache.org/features/deploy.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:56-- http://ignite.apache.org/features/deploy.html
HTTP/1.1 200 OK
--2015-08-24 11:56:56-- http://ignite.apache.org/usecases.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:56-- http://ignite.apache.org/usecases.html
HTTP/1.1 200 OK
--2015-08-24 11:56:57-- http://ignite.apache.org/use-cases/caching/database-caching.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:57-- http://ignite.apache.org/use-cases/caching/database-caching.html
HTTP/1.1 200 OK
--2015-08-24 11:56:57-- http://ignite.apache.org/use-cases/caching/jcache-provider.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:57-- http://ignite.apache.org/use-cases/caching/jcache-provider.html
HTTP/1.1 200 OK
--2015-08-24 11:56:58-- http://ignite.apache.org/use-cases/caching/key-value-store
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:58-- http://ignite.apache.org/use-cases/caching/key-value-store
HTTP/1.1 200 OK
--2015-08-24 11:56:58-- http://ignite.apache.org/use-cases/caching/hibernate-l2-cache.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:59-- http://ignite.apache.org/use-cases/caching/hibernate-l2-cache.html
HTTP/1.1 200 OK
--2015-08-24 11:56:59-- http://ignite.apache.org/use-cases/caching/web-session-clustering.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:56:59-- http://ignite.apache.org/use-cases/caching/web-session-clustering.html
HTTP/1.1 200 OK
--2015-08-24 11:56:59-- http://ignite.apache.org/use-cases/spark/shared-memory-layer.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:57:00-- http://ignite.apache.org/use-cases/spark/shared-memory-layer.html
HTTP/1.1 200 OK
--2015-08-24 11:57:00-- http://ignite.apache.org/use-cases/spark/sql-queries
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:57:00-- http://ignite.apache.org/use-cases/spark/sql-queries
HTTP/1.1 200 OK
--2015-08-24 11:57:00-- http://ignite.apache.org/use-cases/hadoop/hdfs-cache
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:57:01-- http://ignite.apache.org/use-cases/hadoop/hdfs-cache
HTTP/1.1 200 OK
--2015-08-24 11:57:01-- http://ignite.apache.org/use-cases/hadoop/mapreduce
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:57:01-- http://ignite.apache.org/use-cases/hadoop/mapreduce
HTTP/1.1 200 OK
--2015-08-24 11:57:02-- http://ignite.apache.org/community/resources.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:57:02-- http://ignite.apache.org/community/resources.html
HTTP/1.1 200 OK
--2015-08-24 11:57:02-- http://ignite.apache.org/community/contribute.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:57:02-- http://ignite.apache.org/community/contribute.html
HTTP/1.1 200 OK
--2015-08-24 11:57:03-- http://ignite.apache.org/download.html
HTTP/1.1 200 OK
Remote file exists and could contain links to other resources -- retrieving.
--2015-08-24 11:57:03-- http://ignite.apache.org/download.html
HTTP/1.1 200 OK
--2015-08-24 11:57:03-- http://ignite.apache.org/images/img1.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:03-- http://ignite.apache.org/images/compute_grid.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:04-- http://ignite.apache.org/images/streaming.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:04-- http://ignite.apache.org/images/apache-ignite.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:04-- http://ignite.apache.org/images/logo3.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:04-- http://ignite.apache.org/js/jquery-1.11.1.min.js
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:05-- http://ignite.apache.org/js/shCore.js
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:05-- http://ignite.apache.org/js/shBrushBash.js
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:05-- http://ignite.apache.org/js/shBrushJava.js
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:05-- http://ignite.apache.org/js/shBrushXml.js
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:06-- http://ignite.apache.org/js/shBrushScala.js
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:06-- http://ignite.apache.org/css/shCore.css
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:06-- http://ignite.apache.org/css/shThemeEclipse.css
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:06-- http://ignite.apache.org/images/ignite-db-cache.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:07-- http://ignite.apache.org/features/js/jquery-1.11.1.min.js
HTTP/1.1 404 Not Found
Remote file does not exist -- broken link!!!
--2015-08-24 11:57:07-- http://ignite.apache.org/images/in_memory_compute.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:07-- http://ignite.apache.org/images/ignite-stream-query.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:07-- http://ignite.apache.org/images/in_memory_streaming.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:08-- http://ignite.apache.org/images/ignite_service.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:08-- http://ignite.apache.org/images/spark-ignite-rdd.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:08-- http://ignite.apache.org/images/spark-igfs.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:08-- http://ignite.apache.org/images/hadoop_sequence.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:08-- http://ignite.apache.org/images/ignite-deploy.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:09-- http://ignite.apache.org/css/syntaxhighlighter.css
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:09-- http://ignite.apache.org/images/in_memory_data.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:09-- http://ignite.apache.org/use-cases/caching/js/jquery-1.11.1.min.js
HTTP/1.1 404 Not Found
Remote file does not exist -- broken link!!!
--2015-08-24 11:57:10-- http://ignite.apache.org/images/partitioned_cache.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:10-- http://ignite.apache.org/images/replicated_cache.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:10-- http://ignite.apache.org/images/hibernate-L2-cache.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:11-- http://ignite.apache.org/images/web_sessions_3.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:11-- http://ignite.apache.org/use-cases/spark/js/jquery-1.11.1.min.js
HTTP/1.1 404 Not Found
Remote file does not exist -- broken link!!!
--2015-08-24 11:57:11-- http://ignite.apache.org/images/ignite_filesystem.png
HTTP/1.1 200 OK
Remote file exists but does not contain any link -- not retrieving.
--2015-08-24 11:57:11-- http://ignite.apache.org/use-cases/hadoop/js/jquery-1.11.1.min.js
HTTP/1.1 404 Not Found
Remote file does not exist -- broken link!!!
--2015-08-24 11:57:12-- http://ignite.apache.org/community/js/jquery-1.11.1.min.js
HTTP/1.1 404 Not Found
Remote file does not exist -- broken link!!!
FINISHED --2015-08-24 11:57:12--
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment