Skip to content

Instantly share code, notes, and snippets.

@steveloughran

steveloughran/gist:c8b39a7b87a9bd63d7a383bda8687e7e

Created August 23, 2016 15:20
Show Gist options
  • Save steveloughran/c8b39a7b87a9bd63d7a383bda8687e7e to your computer and use it in GitHub Desktop.
Save steveloughran/c8b39a7b87a9bd63d7a383bda8687e7e to your computer and use it in GitHub Desktop.
Download ZIP
log of a streaming against s3 test
Raw
gistfile1.txt
2016-08-23 16:17:51,877 INFO server.Server (Server.java:doStart(379)) - Started @4407ms
2016-08-23 16:17:51,878 INFO util.Utils (Logging.scala:logInfo(54)) - Successfully started service 'SparkUI' on port 4040.
2016-08-23 16:17:51,881 INFO ui.SparkUI (Logging.scala:logInfo(54)) - Bound SparkUI to 0.0.0.0, and started at http://192.168.1.7:4040
2016-08-23 16:17:51,953 INFO executor.Executor (Logging.scala:logInfo(54)) - Starting executor ID driver on host localhost
2016-08-23 16:17:51,984 INFO util.Utils (Logging.scala:logInfo(54)) - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 61294.
2016-08-23 16:17:51,985 INFO netty.NettyBlockTransferService (Logging.scala:logInfo(54)) - Server created on 192.168.1.7:61294
2016-08-23 16:17:51,986 INFO storage.BlockManagerMaster (Logging.scala:logInfo(54)) - Registering BlockManager BlockManagerId(driver, 192.168.1.7, 61294)
2016-08-23 16:17:51,991 INFO storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(54)) - Registering block manager 192.168.1.7:61294 with 1458.6 MB RAM, BlockManagerId(driver, 192.168.1.7, 61294)
2016-08-23 16:17:51,995 INFO storage.BlockManagerMaster (Logging.scala:logInfo(54)) - Registered BlockManager BlockManagerId(driver, 192.168.1.7, 61294)
2016-08-23 16:17:52,010 INFO handler.ContextHandler (ContextHandler.java:doStart(744)) - Started o.s.j.s.ServletContextHandler@d5bb1c4{/metrics/json,null,AVAILABLE}
2016-08-23 16:17:52,026 WARN streaming.StreamingContext (Logging.scala:logWarning(66)) - spark.master should be set as local[n], n > 1 in local mode if you have receivers to get data, otherwise Spark jobs will not get resources to process the received data.
2016-08-23 16:17:52,985 INFO examples.S3Streaming (Logging.scala:logInfo(54)) - Looking for text files under /test/streaming/streaming/sub*
2016-08-23 16:17:53,193 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Duration for remembering RDDs set to 60000 ms for org.apache.spark.streaming.dstream.FileInputDStream@6418e39e
2016-08-23 16:17:53,962 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Slide time = 1000 ms
2016-08-23 16:17:53,963 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Storage level = Serialized 1x Replicated
2016-08-23 16:17:53,964 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Checkpoint interval = null
2016-08-23 16:17:53,964 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Remember interval = 60000 ms
2016-08-23 16:17:53,964 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Initialized and validated org.apache.spark.streaming.dstream.FileInputDStream@6418e39e
2016-08-23 16:17:53,965 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Slide time = 1000 ms
2016-08-23 16:17:53,965 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Storage level = Serialized 1x Replicated
2016-08-23 16:17:53,965 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Checkpoint interval = null
2016-08-23 16:17:53,965 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Remember interval = 1000 ms
2016-08-23 16:17:53,965 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@7b5fa130
2016-08-23 16:17:53,965 INFO dstream.FilteredDStream (Logging.scala:logInfo(54)) - Slide time = 1000 ms
2016-08-23 16:17:53,966 INFO dstream.FilteredDStream (Logging.scala:logInfo(54)) - Storage level = Serialized 1x Replicated
2016-08-23 16:17:53,966 INFO dstream.FilteredDStream (Logging.scala:logInfo(54)) - Checkpoint interval = null
2016-08-23 16:17:53,966 INFO dstream.FilteredDStream (Logging.scala:logInfo(54)) - Remember interval = 1000 ms
2016-08-23 16:17:53,966 INFO dstream.FilteredDStream (Logging.scala:logInfo(54)) - Initialized and validated org.apache.spark.streaming.dstream.FilteredDStream@23c40b5f
2016-08-23 16:17:53,966 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Slide time = 1000 ms
2016-08-23 16:17:53,966 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Storage level = Serialized 1x Replicated
2016-08-23 16:17:53,966 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Checkpoint interval = null
2016-08-23 16:17:53,966 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Remember interval = 1000 ms
2016-08-23 16:17:53,966 INFO dstream.MappedDStream (Logging.scala:logInfo(54)) - Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@42bde6c2
2016-08-23 16:17:53,966 INFO dstream.ForEachDStream (Logging.scala:logInfo(54)) - Slide time = 1000 ms
2016-08-23 16:17:53,967 INFO dstream.ForEachDStream (Logging.scala:logInfo(54)) - Storage level = Serialized 1x Replicated
2016-08-23 16:17:53,967 INFO dstream.ForEachDStream (Logging.scala:logInfo(54)) - Checkpoint interval = null
2016-08-23 16:17:53,967 INFO dstream.ForEachDStream (Logging.scala:logInfo(54)) - Remember interval = 1000 ms
2016-08-23 16:17:53,967 INFO dstream.ForEachDStream (Logging.scala:logInfo(54)) - Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@3b888ef3
2016-08-23 16:17:54,012 INFO util.RecurringTimer (Logging.scala:logInfo(54)) - Started timer for JobGenerator at time 1471965474000
2016-08-23 16:17:54,013 INFO scheduler.JobGenerator (Logging.scala:logInfo(54)) - Started JobGenerator at 1471965474000 ms
2016-08-23 16:17:54,014 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Started JobScheduler
2016-08-23 16:17:54,016 INFO handler.ContextHandler (ContextHandler.java:doStart(744)) - Started o.s.j.s.ServletContextHandler@691124ee{/streaming,null,AVAILABLE}
2016-08-23 16:17:54,017 INFO handler.ContextHandler (ContextHandler.java:doStart(744)) - Started o.s.j.s.ServletContextHandler@5403799b{/streaming/json,null,AVAILABLE}
2016-08-23 16:17:54,017 INFO handler.ContextHandler (ContextHandler.java:doStart(744)) - Started o.s.j.s.ServletContextHandler@27f3f512{/streaming/batch,null,AVAILABLE}
2016-08-23 16:17:54,018 INFO handler.ContextHandler (ContextHandler.java:doStart(744)) - Started o.s.j.s.ServletContextHandler@3946075{/streaming/batch/json,null,AVAILABLE}
2016-08-23 16:17:54,019 INFO handler.ContextHandler (ContextHandler.java:doStart(744)) - Started o.s.j.s.ServletContextHandler@1de4bee0{/static/streaming,null,AVAILABLE}
2016-08-23 16:17:54,020 INFO streaming.StreamingContext (Logging.scala:logInfo(54)) - StreamingContext started
2016-08-23 16:17:54,136 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 115 ms
2016-08-23 16:17:54,137 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965474000 ms:
2016-08-23 16:17:54,159 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965474000 ms
2016-08-23 16:17:54,162 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965474000 ms.0 from job set of time 1471965474000 ms
-------------------------------------------
Time: 1471965474000 ms
-------------------------------------------
2016-08-23 16:17:54,167 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965474000 ms.0 from job set of time 1471965474000 ms
2016-08-23 16:17:54,168 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.167 s for time 1471965474000 ms (execution: 0.007 s)
2016-08-23 16:17:54,175 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965414000 ms:
2016-08-23 16:17:54,178 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:17:54,182 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:17:55,106 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 101 ms
2016-08-23 16:17:55,106 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965475000 ms:
2016-08-23 16:17:55,112 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965475000 ms
2016-08-23 16:17:55,113 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965475000 ms.0 from job set of time 1471965475000 ms
-------------------------------------------
Time: 1471965475000 ms
-------------------------------------------
2016-08-23 16:17:55,114 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965475000 ms.0 from job set of time 1471965475000 ms
2016-08-23 16:17:55,114 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.114 s for time 1471965475000 ms (execution: 0.001 s)
2016-08-23 16:17:55,115 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 3 from persistence list
2016-08-23 16:17:55,128 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 3
2016-08-23 16:17:55,131 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 2 from persistence list
2016-08-23 16:17:55,133 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 2
2016-08-23 16:17:55,135 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 1 from persistence list
2016-08-23 16:17:55,135 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 1
2016-08-23 16:17:55,136 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965415000 ms:
2016-08-23 16:17:55,136 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:17:55,136 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:17:56,109 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 103 ms
2016-08-23 16:17:56,109 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965476000 ms:
2016-08-23 16:17:56,117 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965476000 ms
2016-08-23 16:17:56,117 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965476000 ms.0 from job set of time 1471965476000 ms
-------------------------------------------
Time: 1471965476000 ms
-------------------------------------------
2016-08-23 16:17:56,118 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965476000 ms.0 from job set of time 1471965476000 ms
2016-08-23 16:17:56,118 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 7 from persistence list
2016-08-23 16:17:56,118 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.118 s for time 1471965476000 ms (execution: 0.001 s)
2016-08-23 16:17:56,119 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 7
2016-08-23 16:17:56,119 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 6 from persistence list
2016-08-23 16:17:56,119 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 6
2016-08-23 16:17:56,119 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 5 from persistence list
2016-08-23 16:17:56,120 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 5
2016-08-23 16:17:56,120 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965416000 ms:
2016-08-23 16:17:56,120 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:17:56,120 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:17:56,523 INFO examples.S3Streaming (Logging.scala:logInfo(54)) - Renaming /test/streaming/generated/subdir to /test/streaming/streaming/subdir
2016-08-23 16:17:57,641 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 637 ms
2016-08-23 16:17:57,641 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965477000 ms:
2016-08-23 16:17:57,646 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965477000 ms
2016-08-23 16:17:57,647 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965477000 ms.0 from job set of time 1471965477000 ms
-------------------------------------------
Time: 1471965477000 ms
-------------------------------------------
2016-08-23 16:17:57,647 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965477000 ms.0 from job set of time 1471965477000 ms
2016-08-23 16:17:57,648 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 11 from persistence list
2016-08-23 16:17:57,648 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.647 s for time 1471965477000 ms (execution: 0.000 s)
2016-08-23 16:17:57,648 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 11
2016-08-23 16:17:57,648 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 10 from persistence list
2016-08-23 16:17:57,649 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 10
2016-08-23 16:17:57,649 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 9 from persistence list
2016-08-23 16:17:57,649 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 9
2016-08-23 16:17:57,649 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965417000 ms:
2016-08-23 16:17:57,650 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:17:57,650 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:17:58,450 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 446 ms
2016-08-23 16:17:58,450 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965478000 ms:
s3a://stevel-ireland-new/test/streaming/streaming/subdir/body1.txt
2016-08-23 16:17:58,518 INFO memory.MemoryStore (Logging.scala:logInfo(54)) - Block broadcast_0 stored as values in memory (estimated size 282.1 KB, free 1458.3 MB)
2016-08-23 16:17:58,576 INFO memory.MemoryStore (Logging.scala:logInfo(54)) - Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.1 KB, free 1458.3 MB)
2016-08-23 16:17:58,578 INFO storage.BlockManagerInfo (Logging.scala:logInfo(54)) - Added broadcast_0_piece0 in memory on 192.168.1.7:61294 (size: 23.1 KB, free: 1458.6 MB)
2016-08-23 16:17:58,579 INFO spark.SparkContext (Logging.scala:logInfo(54)) - Created broadcast 0 from apply at Transformer.scala:22
2016-08-23 16:17:58,615 INFO input.FileInputFormat (FileInputFormat.java:listStatus(289)) - Total input files to process : 1
2016-08-23 16:17:58,630 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965478000 ms
2016-08-23 16:17:58,631 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965478000 ms.0 from job set of time 1471965478000 ms
2016-08-23 16:17:58,668 INFO spark.SparkContext (Logging.scala:logInfo(54)) - Starting job: apply at Transformer.scala:22
2016-08-23 16:17:58,678 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Got job 0 (apply at Transformer.scala:22) with 1 output partitions
2016-08-23 16:17:58,679 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Final stage: ResultStage 0 (apply at Transformer.scala:22)
2016-08-23 16:17:58,679 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Parents of final stage: List()
2016-08-23 16:17:58,680 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Missing parents: List()
2016-08-23 16:17:58,684 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Submitting ResultStage 0 (MapPartitionsRDD[20] at apply at Transformer.scala:22), which has no missing parents
2016-08-23 16:17:58,701 INFO memory.MemoryStore (Logging.scala:logInfo(54)) - Block broadcast_1 stored as values in memory (estimated size 3.8 KB, free 1458.3 MB)
2016-08-23 16:17:58,706 INFO memory.MemoryStore (Logging.scala:logInfo(54)) - Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1458.3 MB)
2016-08-23 16:17:58,707 INFO storage.BlockManagerInfo (Logging.scala:logInfo(54)) - Added broadcast_1_piece0 in memory on 192.168.1.7:61294 (size: 2.2 KB, free: 1458.6 MB)
2016-08-23 16:17:58,709 INFO spark.SparkContext (Logging.scala:logInfo(54)) - Created broadcast 1 from broadcast at DAGScheduler.scala:996
2016-08-23 16:17:58,712 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[20] at apply at Transformer.scala:22)
2016-08-23 16:17:58,713 INFO scheduler.TaskSchedulerImpl (Logging.scala:logInfo(54)) - Adding task set 0.0 with 1 tasks
2016-08-23 16:17:58,749 INFO scheduler.TaskSetManager (Logging.scala:logInfo(54)) - Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0, PROCESS_LOCAL, 7453 bytes)
2016-08-23 16:17:58,759 INFO executor.Executor (Logging.scala:logInfo(54)) - Running task 0.0 in stage 0.0 (TID 0)
2016-08-23 16:17:58,790 INFO rdd.NewHadoopRDD (Logging.scala:logInfo(54)) - Input split: s3a://stevel-ireland-new/test/streaming/streaming/subdir/body1.txt:0+292
2016-08-23 16:17:58,881 INFO executor.Executor (Logging.scala:logInfo(54)) - Finished task 0.0 in stage 0.0 (TID 0). 1021 bytes result sent to driver
2016-08-23 16:17:58,887 INFO scheduler.TaskSetManager (Logging.scala:logInfo(54)) - Finished task 0.0 in stage 0.0 (TID 0) in 154 ms on localhost (1/1)
2016-08-23 16:17:58,888 INFO scheduler.TaskSchedulerImpl (Logging.scala:logInfo(54)) - Removed TaskSet 0.0, whose tasks have all completed, from pool
2016-08-23 16:17:58,890 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - ResultStage 0 (apply at Transformer.scala:22) finished in 0.168 s
2016-08-23 16:17:58,894 INFO scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Job 0 finished: apply at Transformer.scala:22, took 0.225491 s
-------------------------------------------
Time: 1471965478000 ms
-------------------------------------------
3
13
23
33
43
53
63
73
83
93
2016-08-23 16:17:58,896 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965478000 ms.0 from job set of time 1471965478000 ms
2016-08-23 16:17:58,896 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.896 s for time 1471965478000 ms (execution: 0.265 s)
2016-08-23 16:17:58,896 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 15 from persistence list
2016-08-23 16:17:58,897 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 15
2016-08-23 16:17:58,897 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 14 from persistence list
2016-08-23 16:17:58,897 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 14
2016-08-23 16:17:58,897 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 13 from persistence list
2016-08-23 16:17:58,897 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 13
2016-08-23 16:17:58,898 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965418000 ms:
2016-08-23 16:17:58,898 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:17:58,898 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:17:59,580 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 576 ms
2016-08-23 16:17:59,580 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965479000 ms:
2016-08-23 16:17:59,584 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965479000 ms
2016-08-23 16:17:59,584 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965479000 ms.0 from job set of time 1471965479000 ms
-------------------------------------------
Time: 1471965479000 ms
-------------------------------------------
2016-08-23 16:17:59,585 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965479000 ms.0 from job set of time 1471965479000 ms
2016-08-23 16:17:59,585 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 20 from persistence list
2016-08-23 16:17:59,585 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.585 s for time 1471965479000 ms (execution: 0.001 s)
2016-08-23 16:17:59,586 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 20
2016-08-23 16:17:59,586 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 19 from persistence list
2016-08-23 16:17:59,586 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 19
2016-08-23 16:17:59,586 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 18 from persistence list
2016-08-23 16:17:59,586 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 18
2016-08-23 16:17:59,587 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965419000 ms:
2016-08-23 16:17:59,587 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:17:59,587 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:00,447 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 443 ms
2016-08-23 16:18:00,448 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965480000 ms:
2016-08-23 16:18:00,451 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965480000 ms
2016-08-23 16:18:00,452 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965480000 ms.0 from job set of time 1471965480000 ms
-------------------------------------------
Time: 1471965480000 ms
-------------------------------------------
2016-08-23 16:18:00,452 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965480000 ms.0 from job set of time 1471965480000 ms
2016-08-23 16:18:00,453 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.452 s for time 1471965480000 ms (execution: 0.000 s)
2016-08-23 16:18:00,453 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 24 from persistence list
2016-08-23 16:18:00,453 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 24
2016-08-23 16:18:00,453 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 23 from persistence list
2016-08-23 16:18:00,453 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 23
2016-08-23 16:18:00,454 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 22 from persistence list
2016-08-23 16:18:00,454 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 22
2016-08-23 16:18:00,454 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965420000 ms:
2016-08-23 16:18:00,454 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:00,454 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:01,486 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 482 ms
2016-08-23 16:18:01,486 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965481000 ms:
2016-08-23 16:18:01,490 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965481000 ms
2016-08-23 16:18:01,491 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965481000 ms.0 from job set of time 1471965481000 ms
-------------------------------------------
Time: 1471965481000 ms
-------------------------------------------
2016-08-23 16:18:01,492 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965481000 ms.0 from job set of time 1471965481000 ms
2016-08-23 16:18:01,492 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.491 s for time 1471965481000 ms (execution: 0.000 s)
2016-08-23 16:18:01,492 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 28 from persistence list
2016-08-23 16:18:01,492 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 28
2016-08-23 16:18:01,492 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 27 from persistence list
2016-08-23 16:18:01,493 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 27
2016-08-23 16:18:01,493 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 26 from persistence list
2016-08-23 16:18:01,493 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 26
2016-08-23 16:18:01,493 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965421000 ms:
2016-08-23 16:18:01,494 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:01,494 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:02,454 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 450 ms
2016-08-23 16:18:02,454 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965482000 ms:
2016-08-23 16:18:02,458 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965482000 ms
2016-08-23 16:18:02,458 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965482000 ms.0 from job set of time 1471965482000 ms
-------------------------------------------
Time: 1471965482000 ms
-------------------------------------------
2016-08-23 16:18:02,459 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965482000 ms.0 from job set of time 1471965482000 ms
2016-08-23 16:18:02,459 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 32 from persistence list
2016-08-23 16:18:02,459 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.459 s for time 1471965482000 ms (execution: 0.001 s)
2016-08-23 16:18:02,460 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 32
2016-08-23 16:18:02,460 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 31 from persistence list
2016-08-23 16:18:02,460 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 31
2016-08-23 16:18:02,460 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 30 from persistence list
2016-08-23 16:18:02,461 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 30
2016-08-23 16:18:02,461 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965422000 ms:
2016-08-23 16:18:02,461 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:02,461 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:03,519 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 513 ms
2016-08-23 16:18:03,520 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965483000 ms:
2016-08-23 16:18:03,524 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965483000 ms
2016-08-23 16:18:03,524 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965483000 ms.0 from job set of time 1471965483000 ms
-------------------------------------------
Time: 1471965483000 ms
-------------------------------------------
2016-08-23 16:18:03,525 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965483000 ms.0 from job set of time 1471965483000 ms
2016-08-23 16:18:03,525 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 36 from persistence list
2016-08-23 16:18:03,525 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.525 s for time 1471965483000 ms (execution: 0.001 s)
2016-08-23 16:18:03,525 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 36
2016-08-23 16:18:03,526 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 35 from persistence list
2016-08-23 16:18:03,526 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 35
2016-08-23 16:18:03,526 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 34 from persistence list
2016-08-23 16:18:03,527 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 34
2016-08-23 16:18:03,527 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965423000 ms:
2016-08-23 16:18:03,527 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:03,527 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:04,709 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 706 ms
2016-08-23 16:18:04,709 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965484000 ms:
2016-08-23 16:18:04,713 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965484000 ms
2016-08-23 16:18:04,714 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965484000 ms.0 from job set of time 1471965484000 ms
-------------------------------------------
Time: 1471965484000 ms
-------------------------------------------
2016-08-23 16:18:04,714 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965484000 ms.0 from job set of time 1471965484000 ms
2016-08-23 16:18:04,714 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 40 from persistence list
2016-08-23 16:18:04,714 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.714 s for time 1471965484000 ms (execution: 0.000 s)
2016-08-23 16:18:04,715 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 40
2016-08-23 16:18:04,715 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 39 from persistence list
2016-08-23 16:18:04,715 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 39
2016-08-23 16:18:04,716 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 38 from persistence list
2016-08-23 16:18:04,716 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 38
2016-08-23 16:18:04,716 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965424000 ms:
2016-08-23 16:18:04,716 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:04,716 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:05,715 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 711 ms
2016-08-23 16:18:05,715 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965485000 ms:
2016-08-23 16:18:05,718 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965485000 ms
2016-08-23 16:18:05,719 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965485000 ms.0 from job set of time 1471965485000 ms
-------------------------------------------
Time: 1471965485000 ms
-------------------------------------------
2016-08-23 16:18:05,720 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965485000 ms.0 from job set of time 1471965485000 ms
2016-08-23 16:18:05,720 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.719 s for time 1471965485000 ms (execution: 0.000 s)
2016-08-23 16:18:05,720 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 44 from persistence list
2016-08-23 16:18:05,720 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 44
2016-08-23 16:18:05,720 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 43 from persistence list
2016-08-23 16:18:05,721 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 43
2016-08-23 16:18:05,721 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 42 from persistence list
2016-08-23 16:18:05,721 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 42
2016-08-23 16:18:05,721 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965425000 ms:
2016-08-23 16:18:05,721 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:05,721 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:06,550 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 546 ms
2016-08-23 16:18:06,550 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965486000 ms:
2016-08-23 16:18:06,554 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965486000 ms
2016-08-23 16:18:06,554 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965486000 ms.0 from job set of time 1471965486000 ms
-------------------------------------------
Time: 1471965486000 ms
-------------------------------------------
2016-08-23 16:18:06,555 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965486000 ms.0 from job set of time 1471965486000 ms
2016-08-23 16:18:06,555 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.555 s for time 1471965486000 ms (execution: 0.001 s)
2016-08-23 16:18:06,555 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 48 from persistence list
2016-08-23 16:18:06,556 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 48
2016-08-23 16:18:06,556 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 47 from persistence list
2016-08-23 16:18:06,556 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 47
2016-08-23 16:18:06,556 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 46 from persistence list
2016-08-23 16:18:06,557 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 46
2016-08-23 16:18:06,557 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965426000 ms:
2016-08-23 16:18:06,557 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:06,557 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:07,507 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 503 ms
2016-08-23 16:18:07,507 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965487000 ms:
2016-08-23 16:18:07,511 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965487000 ms
2016-08-23 16:18:07,512 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965487000 ms.0 from job set of time 1471965487000 ms
-------------------------------------------
Time: 1471965487000 ms
-------------------------------------------
2016-08-23 16:18:07,513 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965487000 ms.0 from job set of time 1471965487000 ms
2016-08-23 16:18:07,513 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 52 from persistence list
2016-08-23 16:18:07,513 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.512 s for time 1471965487000 ms (execution: 0.000 s)
2016-08-23 16:18:07,513 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 52
2016-08-23 16:18:07,513 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 51 from persistence list
2016-08-23 16:18:07,514 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 51
2016-08-23 16:18:07,514 INFO rdd.MapPartitionsRDD (Logging.scala:logInfo(54)) - Removing RDD 50 from persistence list
2016-08-23 16:18:07,514 INFO storage.BlockManager (Logging.scala:logInfo(54)) - Removing RDD 50
2016-08-23 16:18:07,515 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Cleared 0 old files that were older than 1471965427000 ms:
2016-08-23 16:18:07,515 INFO scheduler.ReceivedBlockTracker (Logging.scala:logInfo(54)) - Deleting batches:
2016-08-23 16:18:07,515 INFO scheduler.InputInfoTracker (Logging.scala:logInfo(54)) - remove old batch metadata:
2016-08-23 16:18:08,544 INFO examples.S3Streaming (Logging.scala:logInfo(54)) - Total number of lines ending in 3 sighted: 10
2016-08-23 16:18:08,545 INFO examples.S3Streaming (Logging.scala:logInfo(54)) - Total number of lines ending in 3 sighted: 10
2016-08-23 16:18:08,546 INFO examples.S3Streaming (Logging.scala:logInfo(54)) - FileSystem local stats: S3AFileSystem{uri=s3a://hwdev-steve-ireland-new, workingDir=s3a://stevel-ireland-new/user/stevel, inputPolicy=sequential, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=1048576, multiPartThreshold=2147483647, statistics {292 bytes read, 292 bytes written, 101 read ops, 0 large read ops, 11 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=343b706a-c238-4d71-9ed8-8083601ac28a-hwdev-steve-ireland-new} {fsURI=s3a://hwdev-steve-ireland-new} {files_created=1} {files_copied=1} {files_copied_bytes=292} {files_deleted=1} {directories_created=3} {directories_deleted=0} {ignored_errors=2} {op_copy_from_local_file=0} {op_exists=1} {op_get_file_status=72} {op_glob_status=16} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=27} {op_mkdirs=2} {op_rename=1} {object_copy_requests=0} {object_delete_requests=3} {object_list_requests=88} {object_continue_list_requests=0} {object_metadata_requests=140} {object_multipart_aborted=0} {object_put_bytes=292} {object_put_requests=4} {stream_read_fully_operations=0} {stream_bytes_skipped_on_seek=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=292} {streamOpened=1} {stream_backward_seek_pperations=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=1} {stream_read_operations=1} {stream_aborted=0} {stream_forward_seek_operations=0} {streamClosed=1} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
2016-08-23 16:18:08,547 INFO scheduler.JobGenerator (Logging.scala:logInfo(54)) - Stopping JobGenerator immediately
2016-08-23 16:18:08,548 INFO util.RecurringTimer (Logging.scala:logInfo(54)) - Stopped timer for JobGenerator after time 1471965488000
2016-08-23 16:18:08,566 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - Finding new files took 562 ms
2016-08-23 16:18:08,567 INFO dstream.FileInputDStream (Logging.scala:logInfo(54)) - New files at time 1471965488000 ms:
2016-08-23 16:18:08,570 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Added jobs for time 1471965488000 ms
2016-08-23 16:18:08,571 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Starting job streaming job 1471965488000 ms.0 from job set of time 1471965488000 ms
-------------------------------------------
Time: 1471965488000 ms
-------------------------------------------
2016-08-23 16:18:08,571 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Finished job streaming job 1471965488000 ms.0 from job set of time 1471965488000 ms
2016-08-23 16:18:08,572 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Total delay: 0.571 s for time 1471965488000 ms (execution: 0.001 s)
2016-08-23 16:18:08,572 INFO scheduler.JobGenerator (Logging.scala:logInfo(54)) - Stopped JobGenerator
2016-08-23 16:18:08,573 INFO scheduler.JobScheduler (Logging.scala:logInfo(54)) - Stopped JobScheduler
2016-08-23 16:18:08,578 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@691124ee{/streaming,null,UNAVAILABLE}
2016-08-23 16:18:08,578 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@27f3f512{/streaming/batch,null,UNAVAILABLE}
2016-08-23 16:18:08,581 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@1de4bee0{/static/streaming,null,UNAVAILABLE}
2016-08-23 16:18:08,582 INFO streaming.StreamingContext (Logging.scala:logInfo(54)) - StreamingContext stopped successfully
2016-08-23 16:18:08,592 INFO server.ServerConnector (AbstractConnector.java:doStop(306)) - Stopped ServerConnector@6fc1020a{HTTP/1.1}{0.0.0.0:4040}
2016-08-23 16:18:08,593 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@7af3874e{/stages/stage/kill,null,UNAVAILABLE}
2016-08-23 16:18:08,593 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@4832f03b{/api,null,UNAVAILABLE}
2016-08-23 16:18:08,593 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@4f3faa70{/,null,UNAVAILABLE}
2016-08-23 16:18:08,594 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@18371d89{/static,null,UNAVAILABLE}
2016-08-23 16:18:08,595 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@559cedee{/executors/threadDump/json,null,UNAVAILABLE}
2016-08-23 16:18:08,595 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@c017175{/executors/threadDump,null,UNAVAILABLE}
2016-08-23 16:18:08,595 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@6d074b14{/executors/json,null,UNAVAILABLE}
2016-08-23 16:18:08,595 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@6f044c58{/executors,null,UNAVAILABLE}
2016-08-23 16:18:08,596 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@2f0ed952{/environment/json,null,UNAVAILABLE}
2016-08-23 16:18:08,596 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@5b275811{/environment,null,UNAVAILABLE}
2016-08-23 16:18:08,596 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@43982337{/storage/rdd/json,null,UNAVAILABLE}
2016-08-23 16:18:08,597 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@3e5fd2b1{/storage/rdd,null,UNAVAILABLE}
2016-08-23 16:18:08,597 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@46e64760{/storage/json,null,UNAVAILABLE}
2016-08-23 16:18:08,597 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@22c75c01{/storage,null,UNAVAILABLE}
2016-08-23 16:18:08,598 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@2f9a4401{/stages/pool/json,null,UNAVAILABLE}
2016-08-23 16:18:08,598 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@4667c4c1{/stages/pool,null,UNAVAILABLE}
2016-08-23 16:18:08,598 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@52d6d273{/stages/stage/json,null,UNAVAILABLE}
2016-08-23 16:18:08,599 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@643d2dae{/stages/stage,null,UNAVAILABLE}
2016-08-23 16:18:08,599 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@49f40c00{/stages/json,null,UNAVAILABLE}
2016-08-23 16:18:08,599 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@69afa141{/stages,null,UNAVAILABLE}
2016-08-23 16:18:08,599 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@462e1e64{/jobs/job/json,null,UNAVAILABLE}
2016-08-23 16:18:08,600 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@1806bc4c{/jobs/job,null,UNAVAILABLE}
2016-08-23 16:18:08,600 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@31e2232f{/jobs/json,null,UNAVAILABLE}
2016-08-23 16:18:08,600 INFO handler.ContextHandler (ContextHandler.java:doStop(865)) - Stopped o.s.j.s.ServletContextHandler@2121d1f9{/jobs,null,UNAVAILABLE}
2016-08-23 16:18:08,602 INFO ui.SparkUI (Logging.scala:logInfo(54)) - Stopped Spark web UI at http://192.168.1.7:4040
2016-08-23 16:18:08,609 INFO spark.MapOutputTrackerMasterEndpoint (Logging.scala:logInfo(54)) - MapOutputTrackerMasterEndpoint stopped!
2016-08-23 16:18:08,614 INFO memory.MemoryStore (Logging.scala:logInfo(54)) - MemoryStore cleared
2016-08-23 16:18:08,615 INFO storage.BlockManager (Logging.scala:logInfo(54)) - BlockManager stopped
2016-08-23 16:18:08,615 INFO storage.BlockManagerMaster (Logging.scala:logInfo(54)) - BlockManagerMaster stopped
2016-08-23 16:18:08,617 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint (Logging.scala:logInfo(54)) - OutputCommitCoordinator stopped!
2016-08-23 16:18:08,619 INFO spark.SparkContext (Logging.scala:logInfo(54)) - Successfully stopped SparkContext
2016-08-23 16:18:08,623 INFO s3.S3aStreamingSuite (Logging.scala:logInfo(54)) -
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment