skonto

$ sbt verifyBlueprint
[info] Loading settings for project sensor-data-scala-build from plugins.sbt,cloudflow-plugins.sbt ...
[info] Loading project definition from cloudflow/oss/post_launch/cloudflow/examples/sensor-data-scala/project
[info] Updating ProjectRef(uri("file:cloudflow/oss/post_launch/cloudflow/examples/sensor-data-scala/project/"), "sensor-data-scala-build")...
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Loading settings for project sensorData from build.sbt ...
[info] Set current project to sensor-data-scala (in build file:/cloudflow/oss/post_launch/cloudflow/examples/sensor-data-scala/)
[info] Updating ...
[info] Formatting 14 Scala sources ProjectRef(uri("file:/cloudflow/oss/post_launch/cloudflow/examples/sensor-data-scala/"), "sensorData")(compile) ...

skonto

Replace https://github.com/apache/spark/blob/f2d71f5838f4a611e3ebd60f49ee80521d772524/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala#L138-L153
with
case other =>
        if (seenTypeSet.contains(other)) {
          logInfo("SeenTypeSet:")
          seenTypeSet.foreach{ x => logInfo(s"class: $x")}
          throw new UnsupportedOperationException(
            "Cannot have circular references in bean class, but got the circular reference " +
              s"of class $other")
        }

skonto

https://github.com/confluentinc/ksql
https://github.com/volcano-sh/volcano#overall-architecture
https://github.com/intel-analytics/BigDL#bigdl-distributed-deep-learning-on-apache-spark
https://github.com/netdata/netdata (they show the addiction)
https://github.com/delta-io/delta
https://github.com/keras-team/keras
https://github.com/pytorch/pytorch

skonto

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ echo 0
0

skonto

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ echo 0
0

skonto

I0731 12:45:00.713186      10 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0731 12:45:00.713227      10 controller.go:246] Starting processing key: "spark/spark-pi"
I0731 12:45:00.713404      10 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"spark-pi", UID:"ec2313f6-b390-11e9-a01c-0e17384ad88e", APIVersion:"sparkoperator.k8s.io/v1beta1", ResourceVersion:"25340148", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSpecUpdateProcessed' Successfully processed spec update for SparkApplication spark-pi
I0731 12:45:00.713735      10 controller.go:206] SparkApplication spark/spark-pi was updated, enqueueing it
I0731 12:45:00.718758      10 controller.go:569] Trying to update SparkApplication spark/spark-pi, from: [{spark-7ef7cf6385634dffb50fbf7dc0faf0b1 a0f15b10-b66e-44f4-bd66-e91f2f10562e 2019-07-31 12:44:26 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {spark-pi-ui-svc 30485 10.0.2.27:30485   spark-pi-driver} {RUNNING } map[spark-pi-15645

skonto

driver log:
$ ../spark-k8s/kubectl logs spark-pi2-driver -n spark
++ id -u
+ myuid=185
++ id -g
+ mygid=0
+ set +e
++ getent passwd 185
+ uidentry=
+ set -e

skonto

 ./bin/spark-shell  --conf spark.jars.packages=com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.6
Ivy Default Cache set to: /home/ubuntu/.ivy2/cache
The jars for the packages stored in: /home/ubuntu/.ivy2/jars
:: loading settings :: url = jar:file:/home/ubuntu/workspace/spark-2.4.3-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.amazonaws#aws-java-sdk added as a dependency
org.apache.hadoop#hadoop-aws added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-d3c9d1bd-404e-405d-9bb5-b4419657a1a2;1.0
	confs: [default]
	found com.amazonaws#aws-java-sdk;1.7.4 in central
	found commons-logging#commons-logging;1.1.1 in central

skonto

I0315 19:45:37.845666      11 webhook.go:136] Serving admission request
I0315 19:45:37.847765      11 webhook.go:290] Pod spark-pi-driver in namespace spark is subject to mutation
I0315 19:45:37.865038      11 spark_pod_eventhandler.go:41] Pod spark-pi-driver added in namespace spark.
I0315 19:45:37.865066      11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:37.865370      11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:37.865400      11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:37.876064      11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:37.876092      11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:38.225406      11 controller.go:595] SparkApplication spark/spark-pi has been submitted
I0315 19:45:38.225482      11 spark

skonto

so they earliest msg is:
```
E0721 23:50:08.788032      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:10.785026      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:12.787966      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:14.785313      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:16.787653      52 kubelet_volumes.g