Skip to content

Instantly share code, notes, and snippets.

@skonto

skonto/gist:4a02eeab4fe1ca70e252703a4c26af69

Last active March 15, 2019 20:15
Show Gist options
  • Save skonto/4a02eeab4fe1ca70e252703a4c26af69 to your computer and use it in GitHub Desktop.
Save skonto/4a02eeab4fe1ca70e252703a4c26af69 to your computer and use it in GitHub Desktop.
Download ZIP
log
Raw
gistfile1.txt
I0315 19:45:37.845666 11 webhook.go:136] Serving admission request
I0315 19:45:37.847765 11 webhook.go:290] Pod spark-pi-driver in namespace spark is subject to mutation
I0315 19:45:37.865038 11 spark_pod_eventhandler.go:41] Pod spark-pi-driver added in namespace spark.
I0315 19:45:37.865066 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:37.865370 11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:37.865400 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:37.876064 11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:37.876092 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:38.225406 11 controller.go:595] SparkApplication spark/spark-pi has been submitted
I0315 19:45:38.225482 11 sparkui.go:135] Creating a service spark-pi-ui-svc for the Spark UI for application spark-pi
I0315 19:45:38.225672 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444061", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSubmitted' SparkApplication spark-pi was submitted successfully
I0315 19:45:38.241005 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{ 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 } { } map[] 0 0}] to [{ 2019-03-15 19:45:38.225430886 +0000 UTC m=+16.853557744 0001-01-01 00:00:00 +0000 UTC {spark-pi-ui-svc 30767 } {SUBMITTED } map[] 1 1}]
I0315 19:45:38.263137 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: { 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 } { } map[] 0 0} new status: { 2019-03-15 19:45:38.225430886 +0000 UTC m=+16.853557744 0001-01-01 00:00:00 +0000 UTC {spark-pi-ui-svc 30767 } {SUBMITTED } map[] 1 1}
I0315 19:45:38.263225 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:38.263267 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:38.263556 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444061", FieldPath:""}): type: 'Normal' reason: 'SparkDriverPending' Driver spark-pi-driver is pending
I0315 19:45:38.263638 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:38.269172 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{ 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 } { } map[] 0 0}] to [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0}]
I0315 19:45:38.296645 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: { 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 } { } map[] 0 0} new status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0}
I0315 19:45:38.296803 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:38.296866 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:38.296963 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:38.300253 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0}] to [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0}]
I0315 19:45:38.305790 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0} new status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0}
I0315 19:45:38.305837 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:44.087509 11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:44.087562 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:44.087612 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:44.088166 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444088", FieldPath:""}): type: 'Normal' reason: 'SparkDriverRunning' Driver spark-pi-driver is running
I0315 19:45:44.091211 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0}] to [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}]
I0315 19:45:44.100700 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {SUBMITTED } map[] 0 0} new status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}
I0315 19:45:44.100767 11 metrics.go:111] Incrementing spark_app_running_count with labels map[app_type:Unknown]
I0315 19:45:44.100918 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:44.101652 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:44.101695 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:44.105016 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}] to [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}]
I0315 19:45:44.109519 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0} new status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}
I0315 19:45:44.109624 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:52.436046 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:52.436110 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:52.447947 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}] to [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}]
I0315 19:45:52.452876 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0} new status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}
I0315 19:45:52.452923 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:53.230721 11 webhook.go:136] Serving admission request
I0315 19:45:53.242906 11 spark_pod_eventhandler.go:41] Pod spark-pi-1552679136809-exec-1 added in namespace spark.
I0315 19:45:53.242930 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:53.242953 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:53.243260 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444113", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:53.248995 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-1 updated in namespace spark.
I0315 19:45:53.249054 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:53.249598 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}]
I0315 19:45:53.262538 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-1 updated in namespace spark.
I0315 19:45:53.262587 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:53.263318 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}
I0315 19:45:53.263371 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:53.263402 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:53.263685 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444113", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:53.264536 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:53.266699 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}]
I0315 19:45:53.278436 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-821febdf84054a92a58058e07d1e564c 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}
I0315 19:45:53.278511 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:53.278632 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:53.279333 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444154", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:53.281717 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}]
I0315 19:45:53.281838 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:53.302254 11 webhook.go:136] Serving admission request
I0315 19:45:53.311400 11 spark_pod_eventhandler.go:41] Pod spark-pi-1552679136809-exec-2 added in namespace spark.
I0315 19:45:53.311467 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:53.313393 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:53.314698 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444154", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-2 is pending
I0315 19:45:53.314748 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444154", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:53.316587 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-2 updated in namespace spark.
I0315 19:45:53.316614 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:53.317607 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:53.325263 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}
I0315 19:45:53.325417 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:53.325504 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:53.325834 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444154", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:53.325907 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444154", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-2 is pending
I0315 19:45:53.326828 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:53.329624 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:53.330562 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-2 updated in namespace spark.
I0315 19:45:53.331241 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:53.340397 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}
I0315 19:45:53.340501 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:53.340572 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:53.340753 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444161", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:53.340789 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444161", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-2 is pending
I0315 19:45:53.367106 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:53.367157 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444166", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSpecUpdateProcessed' Successfully processed spec update for SparkApplication spark-pi
I0315 19:45:53.368174 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:53.646313 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {RUNNING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:53.646444 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:53.646478 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:53.646509 11 controller.go:696] Deleting pod with name spark-pi-driver in namespace spark
I0315 19:45:54.050356 11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:54.050385 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:54.054853 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {INVALIDATING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {PENDING_RERUN } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:54.055737 11 spark_pod_eventhandler.go:71] Pod spark-pi-driver deleted in namespace spark.
I0315 19:45:54.055761 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:54.064465 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {INVALIDATING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {PENDING_RERUN } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}
I0315 19:45:54.064517 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:54.064544 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:54.064575 11 controller.go:696] Deleting pod with name spark-pi-driver in namespace spark
I0315 19:45:54.068911 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:54.071748 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-2 updated in namespace spark.
I0315 19:45:54.071770 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:54.073003 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-1 updated in namespace spark.
I0315 19:45:54.073023 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:54.086543 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-2 updated in namespace spark.
I0315 19:45:54.086592 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:54.088215 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-1 updated in namespace spark.
I0315 19:45:54.088238 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:54.446575 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {INVALIDATING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {PENDING_RERUN } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:54.456117 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {INVALIDATING } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {PENDING_RERUN } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}
I0315 19:45:54.456163 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:54.456193 11 controller.go:246] Starting processing key: "spark/spark-pi"
I0315 19:45:54.456301 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444175", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-2 is pending
I0315 19:45:54.456342 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444175", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
W0315 19:45:54.456314 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:54.456381 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {PENDING_RERUN } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54.456362683 +0000 UTC m=+33.084489425 { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:54.456658 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444175", FieldPath:""}): type: 'Warning' reason: 'SparkApplicationFailed' SparkApplication spark-pi terminated with state: FAILED
I0315 19:45:54.463756 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0 spark-pi-driver} {PENDING_RERUN } map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54.456362683 +0000 UTC m=+33.084489425 { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}
I0315 19:45:54.463808 11 metrics.go:124] Decrementing spark_app_running_count with labels map[app_type:Unknown] metricVal to 0
I0315 19:45:54.463826 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:54.464524 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:54.464574 11 controller.go:246] Starting processing key: "spark/spark-pi"
W0315 19:45:54.464665 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:54.464692 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:54.464797 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:54.464833 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444187", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-2 is pending
I0315 19:45:54.464858 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444187", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:55.397203 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-2 updated in namespace spark.
I0315 19:45:55.397232 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:55.397261 11 controller.go:246] Starting processing key: "spark/spark-pi"
W0315 19:45:55.397384 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:55.397414 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444187", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-2 is pending
I0315 19:45:55.397450 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444187", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:55.397404 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}]
I0315 19:45:55.397596 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:55.400708 11 spark_pod_eventhandler.go:71] Pod spark-pi-1552679136809-exec-2 deleted in namespace spark.
I0315 19:45:55.400728 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:55.400752 11 controller.go:246] Starting processing key: "spark/spark-pi"
W0315 19:45:55.400831 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:55.400859 11 controller.go:360] Executor pod spark-pi-1552679136809-exec-2 not found, assuming it was deleted.
I0315 19:45:55.400867 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0}]
I0315 19:45:55.400991 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444187", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:55.408882 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:PENDING] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0}
I0315 19:45:55.408977 11 sparkapp_metrics.go:212] Exporting Metrics for Executor spark-pi-1552679136809-exec-2. OldState: PENDING NewState: FAILED
I0315 19:45:55.409019 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:55.409515 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:55.410846 11 controller.go:246] Starting processing key: "spark/spark-pi"
W0315 19:45:55.411244 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:55.411270 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0}]
I0315 19:45:55.411423 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:55.411436 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444199", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:56.823043 11 spark_pod_eventhandler.go:52] Pod spark-pi-1552679136809-exec-1 updated in namespace spark.
I0315 19:45:56.823070 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:56.823115 11 controller.go:246] Starting processing key: "spark/spark-pi"
W0315 19:45:56.823215 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:56.823234 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0}]
I0315 19:45:56.823328 11 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"e6d242fa-475a-11e9-b674-0288a0a0c3ec", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"1444199", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-1552679136809-exec-1 is pending
I0315 19:45:56.823394 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:56.827553 11 spark_pod_eventhandler.go:71] Pod spark-pi-1552679136809-exec-1 deleted in namespace spark.
I0315 19:45:56.827574 11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:56.827603 11 controller.go:246] Starting processing key: "spark/spark-pi"
W0315 19:45:56.827685 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:56.827714 11 controller.go:360] Executor pod spark-pi-1552679136809-exec-1 not found, assuming it was deleted.
I0315 19:45:56.827724 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:FAILED spark-pi-1552679136809-exec-2:FAILED] 0 0}]
I0315 19:45:56.834021 11 sparkapp_metrics.go:135] Exporting metrics for spark-pi; old status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:PENDING spark-pi-1552679136809-exec-2:FAILED] 0 0} new status: {spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:FAILED spark-pi-1552679136809-exec-2:FAILED] 0 0}
I0315 19:45:56.834068 11 sparkapp_metrics.go:212] Exporting Metrics for Executor spark-pi-1552679136809-exec-1. OldState: PENDING NewState: FAILED
I0315 19:45:56.834084 11 controller.go:253] Ending processing key: "spark/spark-pi"
I0315 19:45:56.834771 11 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0315 19:45:56.834808 11 controller.go:246] Starting processing key: "spark/spark-pi"
W0315 19:45:56.834840 11 controller.go:333] driver not found for SparkApplication: spark/spark-pi
I0315 19:45:56.834866 11 controller.go:522] Trying to update SparkApplication spark/spark-pi, from: [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:FAILED spark-pi-1552679136809-exec-2:FAILED] 0 0}] to [{spark-application-1552679152864 0001-01-01 00:00:00 +0000 UTC 2019-03-15 19:45:54 +0000 UTC { 0 spark-pi-driver} {FAILED Driver Pod not found} map[spark-pi-1552679136809-exec-1:FAILED spark-pi-1552679136809-exec-2:FAILED] 0 0}]
I0315 19:45:56.834997 11 controller.go:253] Ending processing key: "spark/spark-pi"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment