Successful Adam-Submit

output.txt

adam$ bin/adam-submit --jars=gs://mango-initialization-bucket/google-cloud-nio-0.22.0-alpha-shaded.jar -- transformAlignments gs://mango-initialization-bucket/HG01384.mapped.ILLUMINA.bwa.CLM.low_coverage.20120522.bam gs://mango-initialization-bucket/HG01384.mapped.ILLUMINA.bwa.CLM.low_coverage.20120522_transformed_new.adam
Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain
Using spark-submit=/usr/bin/spark-submit
xxxxxx---- USING ADAM CLI JAR /home/george/adam/adam-assembly/target/adam-assembly-spark2_2.11-0.24.0-SNAPSHOT.jar
18/02/28 04:37:37 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.3-hadoop2
Downloading gs://mango-initialization-bucket/google-cloud-nio-0.22.0-alpha-shaded.jar to /tmp/tmp3395184979763179247/google-cloud-nio-0.22.0-alpha-shaded.jar.
18/02/28 04:37:38 INFO org.bdgenomics.adam.cli.ADAMMain: ADAM invoked with args: "transformAlignments" "gs://mango-initialization-bucket/HG01384.mapped.ILLUMINA.bwa.CLM.low_coverage.20120522.bam" "gs://mango-initialization-bucket/HG01384.mapped.ILLUMINA.bwa.CLM.low_coverage.20120522_transformed_new.adam"
18/02/28 04:37:40 INFO org.spark_project.jetty.util.log: Logging initialized @5857ms
18/02/28 04:37:40 INFO org.spark_project.jetty.server.Server: jetty-9.3.z-SNAPSHOT
18/02/28 04:37:40 INFO org.spark_project.jetty.server.Server: Started @6049ms
18/02/28 04:37:40 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector@878fcc3{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/02/28 04:37:42 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at adam-test-3-m/10.142.0.12:8032
18/02/28 04:37:45 WARN org.apache.hadoop.hdfs.DataStreamer: Caught exception
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1252)
	at java.lang.Thread.join(Thread.java:1326)
	at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:973)
	at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:624)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:801)
18/02/28 04:37:45 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1519790048472_0005
18/02/28 04:37:53 INFO org.bdgenomics.adam.rdd.ADAMContext: Loading gs://mango-initialization-bucket/HG01384.mapped.ILLUMINA.bwa.CLM.low_coverage.20120522.bam as BAM/CRAM/SAM and converting to AlignmentRecords.
18/02/28 04:37:53 INFO org.bdgenomics.adam.rdd.ADAMContext: Loaded header from gs://mango-initialization-bucket/HG01384.mapped.ILLUMINA.bwa.CLM.low_coverage.20120522.bam
18/02/28 04:37:54 INFO org.bdgenomics.adam.serialization.ADAMKryoRegistrator: Did not find Spark internal class. This is expected for earlier Spark versions.
18/02/28 04:37:55 INFO org.bdgenomics.adam.rdd.read.RDDBoundAlignmentRecordRDD: Saving data in ADAM format
18/02/28 04:37:56 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input files to process : 1
18/02/28 04:38:10 INFO org.bdgenomics.adam.serialization.ADAMKryoRegistrator: Did not find Spark internal class. This is expected for earlier Spark versions.
[Stage 0:>                                                         (0 + 5) / 78]18/02/28 04:39:57 INFO org.bdgenomics.adam.serialization.ADAMKryoRegistrator: Did not find Spark internal class. This is expected for earlier Spark versions.
[Stage 0:>                                                         (1 + 4) / 78]18/02/28 04:40:02 INFO org.bdgenomics.adam.serialization.ADAMKryoRegistrator: Did not find Spark internal class. This is expected for earlier Spark versions.
[Stage 0:=>                                                        (2 + 4) / 78]18/02/28 04:40:04 INFO org.bdgenomics.adam.serialization.ADAMKryoRegistrator: Did not find Spark internal class. This is expected for earlier Spark versions.
[Stage 0:==>                                                       (3 + 5) / 78]18/02/28 04:40:13 INFO org.bdgenomics.adam.serialization.ADAMKryoRegistrator: Did not find Spark internal class. This is expected for earlier Spark versions.
18/02/28 05:11:55 INFO org.apache.parquet.hadoop.ParquetFileReader: Initiating action with parallelism: 5
18/02/28 05:12:02 INFO org.bdgenomics.adam.cli.TransformAlignments: Overall Duration: 34 mins 23 secs
18/02/28 05:12:02 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@878fcc3{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}