Skip to content

Instantly share code, notes, and snippets.

@buggtb

buggtb/gist:a9e0445f24182bc8eedfe26c0f07a473 Secret

Created June 9, 2021 01:59
Show Gist options
  • Save buggtb/a9e0445f24182bc8eedfe26c0f07a473 to your computer and use it in GitHub Desktop.
Save buggtb/a9e0445f24182bc8eedfe26c0f07a473 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
21/06/09 01:49:02 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:02 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:03 INFO StaticConf$: DB_HOME: /databricks
21/06/09 01:49:03 INFO DatabricksMountsStore: Mount store initialization: Attempting to get the list of mounts from metadata manager of DBFS
21/06/09 01:49:03 INFO log: Logging initialized @2913ms to shaded.v9_4.org.eclipse.jetty.util.log.Slf4jLog
21/06/09 01:49:04 INFO log: Logging initialized @3014ms to org.eclipse.jetty.util.log.Slf4jLog
21/06/09 01:49:04 INFO TypeUtil: JVM Runtime does not support Modules
21/06/09 01:49:04 INFO DatabricksMountsStore: Mount store initialization: Received a list of 11 mounts accessible from metadata manager of DBFS
21/06/09 01:49:04 INFO DatabricksMountsStore: Updated mounts cache. Changes: List((+,DbfsMountPoint(s3a://wayfinder.prod.marts/MMIT, /mnt/wayfinder-prod)), (+,DbfsMountPoint(s3a://crawl-data, /mnt/crawl)), (+,DbfsMountPoint(s3a://databricks-datasets-oregon/, /databricks-datasets)), (+,DbfsMountPoint(s3a://commoncrawl, /mnt/commoncrawl)), (+,DbfsMountPoint(s3n://mg-databricks-mmittest, /mnt/s3data)), (+,DbfsMountPoint(unsupported-access-mechanism-for-path--use-mlflow-client:/, /databricks/mlflow-tracking)), (+,DbfsMountPoint(s3a://kli-mmit-s3-root/ephemeral/oregon-prod/6809408319810459, /databricks-results)), (+,DbfsMountPoint(unsupported-access-mechanism-for-path--use-mlflow-client:/, /databricks/mlflow-registry)), (+,DbfsMountPoint(s3a://wayfinder.config/Wayfinder_Config_Files, /mnt/wayfinder)), (+,DbfsMountPoint(s3a://mmit-wayfinder-exchange, /mnt/mmit)), (+,DbfsMountPoint(s3a://kli-mmit-s3-root/oregon-prod/6809408319810459, /)))
21/06/09 01:49:04 INFO deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
21/06/09 01:49:04 INFO DatabricksFileSystemV2Factory: Creating S3A file system for s3a://kli-mmit-s3-root
21/06/09 01:49:04 INFO S3AFileSystem: Initializing S3AFileSystem as class shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
21/06/09 01:49:04 INFO S3AFileSystem:V3: S3 configuration compatibility mode is enabled
21/06/09 01:49:04 INFO S3AFileSystem:V3: FS_CONF_BACK_COMPAT [UNSUPPORTED] Ignoring value SessionToken for key fs.s3a.credentialsType as unsupported
21/06/09 01:49:04 INFO S3AFileSystem:V3: FS_CONF_BACK_COMPAT [UPDATE] Both configuration keys fs.s3.buffer.dir and fs.s3a.buffer.dir are set, ignoring update
21/06/09 01:49:04 INFO S3AFileSystem:V3: Initializing S3AFileSystem for kli-mmit-s3-root
21/06/09 01:49:04 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
21/06/09 01:49:04 INFO MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
21/06/09 01:49:04 INFO MetricsSystemImpl: s3a-file-system metrics system started
21/06/09 01:49:05 WARN ApacheUtils: NoSuchMethodException was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
21/06/09 01:49:05 INFO S3AFileSystem:V3: Max paging keys: maxKeys=5000
21/06/09 01:49:05 INFO S3AFileSystem:V3: Delete with limit configurations: deleteFileCountLimitEnabled=false, deleteFileCountLimit=-1, deleteTimeLimitEnabled=false, deleteTimeLimitMillis=-1, deleteTimeLimitBatchSize=-1
21/06/09 01:49:05 INFO deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
21/06/09 01:49:05 INFO DBFS: Initialized DBFS with DBFSV2 as the delegate.
21/06/09 01:49:05 INFO SecurityManager: Changing view acls to: root
21/06/09 01:49:05 INFO SecurityManager: Changing modify acls to: root
21/06/09 01:49:05 INFO SecurityManager: Changing view acls groups to:
21/06/09 01:49:05 INFO SecurityManager: Changing modify acls groups to:
21/06/09 01:49:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
21/06/09 01:49:06 INFO Utils: Fetching dbfs:/FileStore/bcf/sparkler7.jar to /tmp/spark-d9ca2405-23c9-49f6-b37f-5cb873b78490/fetchFileTemp5906750484592125577.tmp
21/06/09 01:49:11 INFO Utils: Fetching dbfs:/FileStore/bcf/commons-compress-1.20.jar to /tmp/spark-d9ca2405-23c9-49f6-b37f-5cb873b78490/fetchFileTemp4355019191857090903.tmp
21/06/09 01:49:11 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:11 INFO SparkContext: Running Spark version 3.1.0
21/06/09 01:49:11 INFO ResourceUtils: ==============================================================
21/06/09 01:49:11 INFO ResourceUtils: No custom resources configured for spark.driver.
21/06/09 01:49:11 INFO ResourceUtils: ==============================================================
21/06/09 01:49:11 INFO SparkContext: Submitted application: mytestcrawl10
21/06/09 01:49:11 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 10240, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 16.0)
21/06/09 01:49:11 INFO ResourceProfile: Limiting resource is cpu
21/06/09 01:49:11 INFO ResourceProfileManager: Added ResourceProfile id: 0
21/06/09 01:49:11 INFO SecurityManager: Changing view acls to: root
21/06/09 01:49:11 INFO SecurityManager: Changing modify acls to: root
21/06/09 01:49:11 INFO SecurityManager: Changing view acls groups to:
21/06/09 01:49:11 INFO SecurityManager: Changing modify acls groups to:
21/06/09 01:49:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
21/06/09 01:49:12 INFO Utils: Successfully started service 'sparkDriver' on port 37685.
21/06/09 01:49:12 INFO SparkEnv: Registering MapOutputTracker
21/06/09 01:49:12 INFO SparkEnv: Registering BlockManagerMaster
21/06/09 01:49:12 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/06/09 01:49:12 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/06/09 01:49:12 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
21/06/09 01:49:12 INFO DiskBlockManager: Created local directory at /local_disk0/blockmgr-9d1091ba-97ba-4f49-a334-2d6ee2f84bf5
21/06/09 01:49:12 INFO MemoryStore: MemoryStore started with capacity 5.2 GiB
21/06/09 01:49:12 INFO SparkEnv: Registering OutputCommitCoordinator
21/06/09 01:49:12 INFO SparkContext: Loading Databricks config file
21/06/09 01:49:12 INFO SparkContext: Spark configuration:
spark.akka.frameSize=256
spark.app.name=mytestcrawl10
spark.app.startTime=1623203351666
spark.cleaner.referenceTracking.blocking=false
spark.databricks.acl.client=com.databricks.spark.sql.acl.client.SparkSqlAclClient
spark.databricks.acl.provider=com.databricks.sql.acl.ReflectionBackedAclProvider
spark.databricks.acl.scim.client=com.databricks.spark.sql.acl.client.DriverToWebappScimClient
spark.databricks.cloudProvider=AWS
spark.databricks.cloudfetch.hasRegionSupport=true
spark.databricks.cloudfetch.requesterClassName=*********(redacted)
spark.databricks.clusterSource=JOB
spark.databricks.clusterUsageTags.autoTerminationMinutes=0
spark.databricks.clusterUsageTags.cloudProvider=AWS
spark.databricks.clusterUsageTags.clusterAllTags=[{"key":"Vendor","value":"Databricks"},{"key":"Creator","value":"tom@spicule.co.uk"},{"key":"ClusterName","value":"job-62739-run-1"},{"key":"ClusterId","value":"0609-014729-acorn511"},{"key":"JobId","value":"62739"},{"key":"RunName","value":"testsubmi3t"},{"key":"Name","value":"workerenv-6809408319810459-9af0a058-7c2a-4c62-8bb5-a89aa03105e8-worker"}]
spark.databricks.clusterUsageTags.clusterAvailability=SPOT_WITH_FALLBACK
spark.databricks.clusterUsageTags.clusterCreator=JobLauncher
spark.databricks.clusterUsageTags.clusterEbsVolumeCount=0
spark.databricks.clusterUsageTags.clusterEbsVolumeSize=0
spark.databricks.clusterUsageTags.clusterEbsVolumeType=GENERAL_PURPOSE_SSD
spark.databricks.clusterUsageTags.clusterFirstOnDemand=1
spark.databricks.clusterUsageTags.clusterGeneration=0
spark.databricks.clusterUsageTags.clusterId=0609-014729-acorn511
spark.databricks.clusterUsageTags.clusterLogDeliveryEnabled=false
spark.databricks.clusterUsageTags.clusterLogDestination=
spark.databricks.clusterUsageTags.clusterMetastoreAccessType=RDS_DIRECT
spark.databricks.clusterUsageTags.clusterName=job-62739-run-1
spark.databricks.clusterUsageTags.clusterNoDriverDaemon=true
spark.databricks.clusterUsageTags.clusterNodeType=c5d.4xlarge
spark.databricks.clusterUsageTags.clusterNumSshKeys=0
spark.databricks.clusterUsageTags.clusterOwnerOrgId=6809408319810459
spark.databricks.clusterUsageTags.clusterOwnerUserId=*********(redacted)
spark.databricks.clusterUsageTags.clusterPinned=false
spark.databricks.clusterUsageTags.clusterPythonVersion=2
spark.databricks.clusterUsageTags.clusterResourceClass=default
spark.databricks.clusterUsageTags.clusterScalingType=fixed_size
spark.databricks.clusterUsageTags.clusterSku=STANDARD_SKU
spark.databricks.clusterUsageTags.clusterSpotBidPricePercent=100
spark.databricks.clusterUsageTags.clusterState=Pending
spark.databricks.clusterUsageTags.clusterStateMessage=Starting Spark
spark.databricks.clusterUsageTags.clusterTargetWorkers=3
spark.databricks.clusterUsageTags.clusterWorkers=3
spark.databricks.clusterUsageTags.containerType=LXC
spark.databricks.clusterUsageTags.containerZoneId=us-west-2c
spark.databricks.clusterUsageTags.dataPlaneRegion=us-west-2
spark.databricks.clusterUsageTags.driverContainerId=c6b86df1d3344771ade9c2017178fb5a
spark.databricks.clusterUsageTags.driverContainerPrivateIp=10.57.225.53
spark.databricks.clusterUsageTags.driverInstanceId=i-0179e070a7713d0d4
spark.databricks.clusterUsageTags.driverInstancePrivateIp=10.57.241.32
spark.databricks.clusterUsageTags.driverNodeType=c5d.4xlarge
spark.databricks.clusterUsageTags.driverPublicDns=ec2-52-41-240-181.us-west-2.compute.amazonaws.com
spark.databricks.clusterUsageTags.enableCredentialPassthrough=*********(redacted)
spark.databricks.clusterUsageTags.enableDfAcls=false
spark.databricks.clusterUsageTags.enableElasticDisk=false
spark.databricks.clusterUsageTags.enableJdbcAutoStart=true
spark.databricks.clusterUsageTags.enableJobsAutostart=true
spark.databricks.clusterUsageTags.enableLocalDiskEncryption=false
spark.databricks.clusterUsageTags.enableSqlAclsOnly=false
spark.databricks.clusterUsageTags.hailEnabled=false
spark.databricks.clusterUsageTags.instanceBootstrapType=ssh
spark.databricks.clusterUsageTags.instanceProfileUsed=false
spark.databricks.clusterUsageTags.instanceWorkerEnvId=workerenv-6809408319810459-9af0a058-7c2a-4c62-8bb5-a89aa03105e8
spark.databricks.clusterUsageTags.instanceWorkerEnvNetworkType=default
spark.databricks.clusterUsageTags.isIMv2Enabled=false
spark.databricks.clusterUsageTags.isSingleUserCluster=*********(redacted)
spark.databricks.clusterUsageTags.ngrokNpipEnabled=false
spark.databricks.clusterUsageTags.numPerClusterInitScriptsV2=1
spark.databricks.clusterUsageTags.numPerGlobalInitScriptsV2=0
spark.databricks.clusterUsageTags.privateLinkEnabled=false
spark.databricks.clusterUsageTags.region=us-west-2
spark.databricks.clusterUsageTags.sparkVersion=8.3.x-scala2.12
spark.databricks.clusterUsageTags.userProvidedRemoteVolumeCount=*********(redacted)
spark.databricks.clusterUsageTags.userProvidedRemoteVolumeSizeGb=*********(redacted)
spark.databricks.clusterUsageTags.userProvidedRemoteVolumeType=*********(redacted)
spark.databricks.clusterUsageTags.workerEnvironmentId=workerenv-6809408319810459-9af0a058-7c2a-4c62-8bb5-a89aa03105e8
spark.databricks.credential.redactor=*********(redacted)
spark.databricks.delta.logStore.crossCloud.fatal=true
spark.databricks.delta.multiClusterWrites.enabled=true
spark.databricks.driverNfs.enabled=true
spark.databricks.driverNfs.pathSuffix=.ephemeral_nfs
spark.databricks.driverNodeTypeId=c5d.4xlarge
spark.databricks.eventLog.dir=eventlogs
spark.databricks.io.directoryCommit.enableLogicalDelete=false
spark.databricks.managedCatalog.s3a.tokenProviderClassName=*********(redacted)
spark.databricks.overrideDefaultCommitProtocol=org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol
spark.databricks.passthrough.adls.gen2.tokenProviderClassName=*********(redacted)
spark.databricks.passthrough.adls.tokenProviderClassName=*********(redacted)
spark.databricks.passthrough.oauth.refresher.impl=*********(redacted)
spark.databricks.passthrough.s3a.threadPoolExecutor.factory.class=com.databricks.backend.daemon.driver.aws.S3APassthroughThreadPoolExecutorFactory
spark.databricks.passthrough.s3a.tokenProviderClassName=*********(redacted)
spark.databricks.preemption.enabled=true
spark.databricks.redactor=com.databricks.spark.util.DatabricksSparkLogRedactorProxy
spark.databricks.repl.enableClassFileCleanup=true
spark.databricks.secret.envVar.keys.toRedact=*********(redacted)
spark.databricks.secret.sparkConf.keys.toRedact=*********(redacted)
spark.databricks.service.dbutils.repl.backend=com.databricks.dbconnect.ReplDBUtils
spark.databricks.service.dbutils.server.backend=com.databricks.dbconnect.SparkServerDBUtils
spark.databricks.session.share=false
spark.databricks.sparkContextId=3622046537644675211
spark.databricks.tahoe.logStore.aws.class=com.databricks.tahoe.store.MultiClusterLogStore
spark.databricks.tahoe.logStore.azure.class=com.databricks.tahoe.store.AzureLogStore
spark.databricks.tahoe.logStore.class=com.databricks.tahoe.store.DelegatingLogStore
spark.databricks.tahoe.logStore.gcp.class=com.databricks.tahoe.store.GCPLogStore
spark.databricks.workerNodeTypeId=c5d.4xlarge
spark.databricks.workspace.matplotlibInline.enabled=true
spark.databricks.workspace.multipleResults.enabled=true
spark.driver.allowMultipleContexts=false
spark.driver.extraJavaOptions=-Dpf4j.pluginsDir=/dbfs/FileStore/bcf/plugins/
spark.driver.host=10.57.225.53
spark.driver.maxResultSize=4g
spark.driver.memory=10g
spark.driver.port=37685
spark.driver.tempDirectory=/local_disk0/tmp
spark.eventLog.enabled=false
spark.executor.extraClassPath=/databricks/spark/dbconf/log4j/executor:/databricks/spark/dbconf/jets3t:/databricks/spark/dbconf/hadoop:/databricks/jars/*
spark.executor.extraJavaOptions=-Djava.io.tmpdir=/local_disk0/tmp -XX:ReservedCodeCacheSize=512m -XX:+UseCodeCacheFlushing -Djava.security.properties=/databricks/spark/dbconf/java/extra.security -XX:-UseContainerSupport -XX:+PrintFlagsFinal -XX:+PrintGCDateStamps -XX:+PrintGCDetails -verbose:gc -Xss4m -Djava.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni -Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl -Djavax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema=com.sun.org.apache.xerces.internal.jaxp.validation.XMLSchemaFactory -Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -Dorg.w3c.dom.DOMImplementationSourceList=com.sun.org.apache.xerces.internal.dom.DOMXSImplementationSourceImpl -Djavax.net.ssl.sessionCacheSize=10000 -Dscala.reflect.runtime.disable.typetag.cache=true -Ddatabricks.serviceName=spark-executor-1
spark.executor.id=driver
spark.executor.memory=10g
spark.executor.tempDirectory=/local_disk0/tmp
spark.extraListeners=com.databricks.backend.daemon.driver.DBCEventLoggingListener
spark.files.fetchFailure.unRegisterOutputOnHost=true
spark.files.overwrite=true
spark.files.useFetchCache=false
spark.hadoop.databricks.dbfs.client.version=v2
spark.hadoop.databricks.s3.create.deleteUnnecessaryFakeDirectories=false
spark.hadoop.databricks.s3commit.client.sslTrustAll=false
spark.hadoop.fs.AbstractFileSystem.gs.impl=shaded.databricks.V2_1_4.com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS
spark.hadoop.fs.abfs.impl=shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
spark.hadoop.fs.abfs.impl.disable.cache=true
spark.hadoop.fs.abfss.impl=shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem
spark.hadoop.fs.abfss.impl.disable.cache=true
spark.hadoop.fs.adl.impl=com.databricks.adl.AdlFileSystem
spark.hadoop.fs.adl.impl.disable.cache=true
spark.hadoop.fs.azure.skip.metrics=true
spark.hadoop.fs.cpfs-abfss.impl=*********(redacted)
spark.hadoop.fs.cpfs-abfss.impl.disable.cache=true
spark.hadoop.fs.cpfs-adl.impl=*********(redacted)
spark.hadoop.fs.cpfs-adl.impl.disable.cache=true
spark.hadoop.fs.cpfs-s3.impl=*********(redacted)
spark.hadoop.fs.cpfs-s3a.impl=*********(redacted)
spark.hadoop.fs.cpfs-s3n.impl=*********(redacted)
spark.hadoop.fs.file.impl=com.databricks.backend.daemon.driver.WorkspaceLocalFileSystem
spark.hadoop.fs.gs.impl=shaded.databricks.V2_1_4.com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
spark.hadoop.fs.gs.impl.disable.cache=true
spark.hadoop.fs.gs.outputstream.upload.chunk.size=16777216
spark.hadoop.fs.mcfs-abfss.impl=com.databricks.sql.acl.fs.ManagedCatalogFileSystem
spark.hadoop.fs.mcfs-abfss.impl.disable.cache=true
spark.hadoop.fs.mcfs-s3.impl=com.databricks.sql.acl.fs.ManagedCatalogFileSystem
spark.hadoop.fs.mcfs-s3a.impl=com.databricks.sql.acl.fs.ManagedCatalogFileSystem
spark.hadoop.fs.mcfs-s3n.impl=com.databricks.sql.acl.fs.ManagedCatalogFileSystem
spark.hadoop.fs.s3.impl=shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.connection.maximum=200
spark.hadoop.fs.s3a.fast.upload=true
spark.hadoop.fs.s3a.fast.upload.active.blocks=32
spark.hadoop.fs.s3a.fast.upload.default=true
spark.hadoop.fs.s3a.impl=shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.multipart.size=10485760
spark.hadoop.fs.s3a.multipart.threshold=104857600
spark.hadoop.fs.s3a.threads.max=136
spark.hadoop.fs.s3n.impl=shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.wasb.impl=shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem
spark.hadoop.fs.wasb.impl.disable.cache=true
spark.hadoop.fs.wasbs.impl=shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem
spark.hadoop.fs.wasbs.impl.disable.cache=true
spark.hadoop.hive.server2.enable.doAs=false
spark.hadoop.hive.server2.idle.operation.timeout=7200000
spark.hadoop.hive.server2.idle.session.timeout=900000
spark.hadoop.hive.server2.keystore.password=*********(redacted)
spark.hadoop.hive.server2.keystore.path=/databricks/keys/jetty-ssl-driver-keystore.jks
spark.hadoop.hive.server2.session.check.interval=60000
spark.hadoop.hive.server2.thrift.http.cookie.auth.enabled=false
spark.hadoop.hive.server2.thrift.http.port=10000
spark.hadoop.hive.server2.transport.mode=http
spark.hadoop.hive.server2.use.SSL=true
spark.hadoop.hive.warehouse.subdir.inherit.perms=false
spark.hadoop.mapred.output.committer.class=com.databricks.backend.daemon.data.client.DirectOutputCommitter
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2
spark.hadoop.parquet.block.size.row.check.max=10
spark.hadoop.parquet.block.size.row.check.min=10
spark.hadoop.parquet.memory.pool.ratio=0.5
spark.hadoop.parquet.page.size.check.estimate=false
spark.hadoop.parquet.page.verify-checksum.enabled=true
spark.hadoop.parquet.page.write-checksum.enabled=true
spark.hadoop.spark.databricks.io.parquet.verifyChecksumOnWrite.enabled=true
spark.hadoop.spark.databricks.io.parquet.verifyChecksumOnWrite.throwsException=false
spark.hadoop.spark.driverproxy.customHeadersToProperties=*********(redacted)
spark.hadoop.spark.sql.parquet.output.committer.class=org.apache.spark.sql.parquet.DirectParquetOutputCommitter
spark.hadoop.spark.sql.sources.outputCommitterClass=com.databricks.backend.daemon.data.client.MapReduceDirectOutputCommitter
spark.jars=dbfs:/FileStore/bcf/commons-compress-1.20.jar,dbfs:/FileStore/bcf/sparkler7.jar
spark.logConf=true
spark.master=local[*]
spark.metrics.conf=/databricks/spark/conf/metrics.properties
spark.r.backendConnectionTimeout=604800
spark.r.numRBackendThreads=1
spark.rdd.compress=true
spark.repl.local.jars=file:/tmp/spark-d9ca2405-23c9-49f6-b37f-5cb873b78490/commons-compress-1.20.jar
spark.rpc.message.maxSize=256
spark.scheduler.listenerbus.eventqueue.capacity=20000
spark.scheduler.mode=FAIR
spark.serializer.objectStreamReset=100
spark.shuffle.manager=SORT
spark.shuffle.memoryFraction=0.2
spark.shuffle.reduceLocality.enabled=false
spark.shuffle.service.enabled=true
spark.shuffle.service.port=4048
spark.sparkr.use.daemon=false
spark.speculation=false
spark.speculation.multiplier=3
spark.speculation.quantile=0.9
spark.sql.allowMultipleContexts=false
spark.sql.hive.convertCTAS=true
spark.sql.hive.convertMetastoreParquet=true
spark.sql.hive.metastore.jars=/databricks/hive/*
spark.sql.hive.metastore.sharedPrefixes=org.mariadb.jdbc,com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,microsoft.sql.DateTimeOffset,microsoft.sql.Types,com.databricks,com.codahale,com.fasterxml.jackson,shaded.databricks
spark.sql.hive.metastore.version=0.13.0
spark.sql.legacy.createHiveTableByDefault=false
spark.sql.parquet.cacheMetadata=true
spark.sql.parquet.compression.codec=snappy
spark.sql.sources.commitProtocolClass=com.databricks.sql.transaction.directory.DirectoryAtomicCommitProtocol
spark.sql.sources.default=delta
spark.sql.streaming.checkpointFileManagerClass=com.databricks.spark.sql.streaming.DatabricksCheckpointFileManager
spark.sql.streaming.stopTimeout=15s
spark.sql.warehouse.dir=*********(redacted)
spark.storage.blockManagerTimeoutIntervalMs=300000
spark.storage.memoryFraction=0.5
spark.streaming.driver.writeAheadLog.allowBatching=true
spark.streaming.driver.writeAheadLog.closeFileAfterWrite=true
spark.submit.deployMode=client
spark.submit.pyFiles=
spark.task.cpus=16
spark.task.reaper.enabled=true
spark.task.reaper.killTimeout=60s
spark.ui.port=48160
spark.worker.cleanup.enabled=false
21/06/09 01:49:12 WARN MetricsSystem: Using default name SparkStatusTracker for source because neither spark.metrics.namespace nor spark.app.id is set.
21/06/09 01:49:12 INFO Server: jetty-9.4.36.v20210114; built: 2021-01-14T16:44:28.689Z; git: 238ec6997c7806b055319a6d11f8ae7564adc0de; jvm 1.8.0_282-b08
21/06/09 01:49:12 INFO Server: Started @11558ms
21/06/09 01:49:12 INFO AbstractConnector: Started ServerConnector@2570675e{HTTP/1.1, (http/1.1)}{10.57.225.53:48160}
21/06/09 01:49:12 INFO Utils: Successfully started service 'SparkUI' on port 48160.
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@56ec6960{/jobs,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@77ce88c4{/jobs/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@29b5533{/jobs/job,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@21a6a494{/jobs/job/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4ab66127{/stages,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@37fef327{/stages/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@55951fcd{/stages/stage,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3ba3f40d{/stages/stage/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@46ab4efc{/stages/pool,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@b5312df{/stages/pool/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5f409872{/storage,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@472c9f88{/storage/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5822ecda{/storage/rdd,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@7afbf2a0{/storage/rdd/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@65e4cb84{/environment,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5e0f2c82{/environment/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4afd65fd{/executors,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3356ff58{/executors/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@2aa6bbad{/executors/threadDump,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6f867b0c{/executors/threadDump/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@54be6213{/executors/heapHistogram,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@1426370c{/executors/heapHistogram/json,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@2ef041bb{/static,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@d504137{/,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@e4ca109{/api,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@7eee6c13{/jobs/job/kill,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@677cc4e8{/stages/stage/kill,null,AVAILABLE,@Spark}
21/06/09 01:49:12 INFO SparkUI: Bound SparkUI to 10.57.225.53, and started at http://10.57.225.53:48160
21/06/09 01:49:12 INFO SparkContext: Added JAR dbfs:/FileStore/bcf/commons-compress-1.20.jar at dbfs:/FileStore/bcf/commons-compress-1.20.jar with timestamp 1623203351666
21/06/09 01:49:12 INFO SparkContext: Added JAR dbfs:/FileStore/bcf/sparkler7.jar at dbfs:/FileStore/bcf/sparkler7.jar with timestamp 1623203351666
21/06/09 01:49:12 WARN FairSchedulableBuilder: Fair Scheduler configuration file not found so jobs will be scheduled in FIFO order. To use fair scheduling, configure pools in fairscheduler.xml or set spark.scheduler.allocation.file to a file that contains the configuration.
21/06/09 01:49:12 INFO FairSchedulableBuilder: Created default pool: default, schedulingMode: FIFO, minShare: 0, weight: 1
21/06/09 01:49:12 INFO Executor: Starting executor ID driver on host ip-10-57-225-53.us-west-2.compute.internal
21/06/09 01:49:12 INFO Executor: Fetching dbfs:/FileStore/bcf/commons-compress-1.20.jar with timestamp 1623203351666
21/06/09 01:49:12 INFO Utils: Fetching dbfs:/FileStore/bcf/commons-compress-1.20.jar to /local_disk0/spark-7e0eb9a3-4879-458d-87e7-bcfb5b12a23c/userFiles-f07c0b8d-086b-4969-9f83-d12f10702635/fetchFileTemp3042951168538737787.tmp
21/06/09 01:49:13 INFO Executor: Adding file:/local_disk0/spark-7e0eb9a3-4879-458d-87e7-bcfb5b12a23c/userFiles-f07c0b8d-086b-4969-9f83-d12f10702635/commons-compress-1.20.jar to class loader for default
21/06/09 01:49:13 INFO Executor: Fetching dbfs:/FileStore/bcf/sparkler7.jar with timestamp 1623203351666
21/06/09 01:49:13 INFO Utils: Fetching dbfs:/FileStore/bcf/sparkler7.jar to /local_disk0/spark-7e0eb9a3-4879-458d-87e7-bcfb5b12a23c/userFiles-f07c0b8d-086b-4969-9f83-d12f10702635/fetchFileTemp114423565923948694.tmp
21/06/09 01:49:15 INFO Executor: Adding file:/local_disk0/spark-7e0eb9a3-4879-458d-87e7-bcfb5b12a23c/userFiles-f07c0b8d-086b-4969-9f83-d12f10702635/sparkler7.jar to class loader for default
21/06/09 01:49:15 INFO TaskSchedulerImpl: Task preemption enabled.
21/06/09 01:49:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40357.
21/06/09 01:49:15 INFO NettyBlockTransferService: Server created on 10.57.225.53:40357
21/06/09 01:49:15 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/06/09 01:49:15 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.57.225.53, 40357, None)
21/06/09 01:49:15 INFO BlockManagerMasterEndpoint: Registering block manager 10.57.225.53:40357 with 5.2 GiB RAM, BlockManagerId(driver, 10.57.225.53, 40357, None)
21/06/09 01:49:15 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.57.225.53, 40357, None)
21/06/09 01:49:15 INFO BlockManager: external shuffle service port = 4048
21/06/09 01:49:15 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.57.225.53, 40357, None)
21/06/09 01:49:15 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6785c9fa{/metrics/json,null,AVAILABLE,@Spark}
21/06/09 01:49:15 INFO DBCEventLoggingListener: Initializing DBCEventLoggingListener
21/06/09 01:49:15 INFO DBCEventLoggingListener: Logging events to eventlogs/3622046537644675211/eventlog
21/06/09 01:49:15 INFO SparkContext: Registered listener com.databricks.backend.daemon.driver.DBCEventLoggingListener
21/06/09 01:49:15 INFO SparkContext: Loading Spark Service RPC Server
21/06/09 01:49:15 INFO SparkServiceRPCServer: Starting Spark Service RPC Server
21/06/09 01:49:15 INFO Server: jetty-9.4.36.v20210114; built: 2021-01-14T16:44:28.689Z; git: 238ec6997c7806b055319a6d11f8ae7564adc0de; jvm 1.8.0_282-b08
21/06/09 01:49:15 INFO AbstractConnector: Started ServerConnector@280484c7{HTTP/1.1, (http/1.1)}{0.0.0.0:15001}
21/06/09 01:49:15 INFO Server: Started @14934ms
21/06/09 01:49:15 INFO Crawler$: Setting local job: {User-Agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Sparkler/0.2.2-SNAPSHOT, Accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8, Accept-Language=en-US,en}
21/06/09 01:49:16 INFO Crawler$: Committing crawldb..
21/06/09 01:49:16 INFO Crawler$: Starting the job:mytestcrawl10, task:b59002ff-fe85-4ec9-b196-9491b91191c4
21/06/09 01:49:16 INFO SparkContext: Starting job: runJob at Crawler.scala:265
21/06/09 01:49:16 INFO MemexCrawlDbRDD$: selecting 1 out of 1
21/06/09 01:49:16 INFO DAGScheduler: Registering RDD 1 (repartition at Crawler.scala:235) as input to shuffle 1
21/06/09 01:49:16 INFO DAGScheduler: Registering RDD 5 (map at Crawler.scala:235) as input to shuffle 0
21/06/09 01:49:16 INFO DAGScheduler: Got job 0 (runJob at Crawler.scala:265) with 50 output partitions
21/06/09 01:49:16 INFO DAGScheduler: Final stage: ResultStage 2 (runJob at Crawler.scala:265)
21/06/09 01:49:16 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)
21/06/09 01:49:16 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1)
21/06/09 01:49:16 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at repartition at Crawler.scala:235), which has no missing parents
21/06/09 01:49:16 INFO DAGScheduler: Jars for session None: Map(dbfs:/FileStore/bcf/commons-compress-1.20.jar -> 1623203351666, dbfs:/FileStore/bcf/sparkler7.jar -> 1623203351666)
21/06/09 01:49:16 INFO DAGScheduler: Files for session None: Map()
21/06/09 01:49:16 INFO DAGScheduler: Archives for session None: Map()
21/06/09 01:49:16 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at repartition at Crawler.scala:235) (first 15 tasks are for partitions Vector(0))
21/06/09 01:49:16 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
21/06/09 01:49:16 INFO FairSchedulableBuilder: Added task set TaskSet_0.0 tasks to pool default
21/06/09 01:49:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 0, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:16 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 12.4 KiB, free 5.2 GiB)
21/06/09 01:49:16 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.1 KiB, free 5.2 GiB)
21/06/09 01:49:16 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.57.225.53:40357 (size: 6.1 KiB, free: 5.2 GiB)
21/06/09 01:49:16 INFO SparkContext: Created broadcast 0 from broadcast at TaskSetManager.scala:552
21/06/09 01:49:17 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/06/09 01:49:17 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 878 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3616 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (1/1)
21/06/09 01:49:20 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool default
21/06/09 01:49:20 INFO DAGScheduler: ShuffleMapStage 0 (repartition at Crawler.scala:235) finished in 3.695 s
21/06/09 01:49:20 INFO DAGScheduler: looking for newly runnable stages
21/06/09 01:49:20 INFO DAGScheduler: running: Set()
21/06/09 01:49:20 INFO DAGScheduler: waiting: Set(ShuffleMapStage 1, ResultStage 2)
21/06/09 01:49:20 INFO DAGScheduler: failed: Set()
21/06/09 01:49:20 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[5] at map at Crawler.scala:235), which has no missing parents
21/06/09 01:49:20 INFO DAGScheduler: Jars for session None: Map(dbfs:/FileStore/bcf/commons-compress-1.20.jar -> 1623203351666, dbfs:/FileStore/bcf/sparkler7.jar -> 1623203351666)
21/06/09 01:49:20 INFO DAGScheduler: Files for session None: Map()
21/06/09 01:49:20 INFO DAGScheduler: Archives for session None: Map()
21/06/09 01:49:20 INFO DAGScheduler: Submitting 50 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[5] at map at Crawler.scala:235) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
21/06/09 01:49:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 50 tasks resource profile 0
21/06/09 01:49:20 INFO FairSchedulableBuilder: Added task set TaskSet_1.0 tasks to pool default
21/06/09 01:49:20 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 0, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.2 KiB, free 5.2 GiB)
21/06/09 01:49:20 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 10.57.225.53:40357 in memory (size: 6.1 KiB, free: 5.2 GiB)
21/06/09 01:49:20 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.5 KiB, free 5.2 GiB)
21/06/09 01:49:20 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.57.225.53:40357 (size: 5.5 KiB, free: 5.2 GiB)
21/06/09 01:49:20 INFO SparkContext: Created broadcast 1 from broadcast at TaskSetManager.scala:552
21/06/09 01:49:20 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
21/06/09 01:49:20 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 1, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 181 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (1/50)
21/06/09 01:49:20 INFO Executor: Running task 1.0 in stage 1.0 (TID 2)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 2, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 21 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (2/50)
21/06/09 01:49:20 INFO Executor: Running task 2.0 in stage 1.0 (TID 3)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 3, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 16 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (3/50)
21/06/09 01:49:20 INFO Executor: Running task 3.0 in stage 1.0 (TID 4)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 3.0 in stage 1.0 (TID 4). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 5) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 4, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 14 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (4/50)
21/06/09 01:49:20 INFO Executor: Running task 4.0 in stage 1.0 (TID 5)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 4.0 in stage 1.0 (TID 5). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 6) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 5, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 5) in 14 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (5/50)
21/06/09 01:49:20 INFO Executor: Running task 5.0 in stage 1.0 (TID 6)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 5.0 in stage 1.0 (TID 6). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 7) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 6, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 6) in 13 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (6/50)
21/06/09 01:49:20 INFO Executor: Running task 6.0 in stage 1.0 (TID 7)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 6.0 in stage 1.0 (TID 7). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 8) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 7, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 7) in 11 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (7/50)
21/06/09 01:49:20 INFO Executor: Running task 7.0 in stage 1.0 (TID 8)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 7.0 in stage 1.0 (TID 8). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 9) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 8, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 8) in 14 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (8/50)
21/06/09 01:49:20 INFO Executor: Running task 8.0 in stage 1.0 (TID 9)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 8.0 in stage 1.0 (TID 9). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 10) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 9, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 9) in 17 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (9/50)
21/06/09 01:49:20 INFO Executor: Running task 9.0 in stage 1.0 (TID 10)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 9.0 in stage 1.0 (TID 10). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 11) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 10, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 10) in 18 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (10/50)
21/06/09 01:49:20 INFO Executor: Running task 10.0 in stage 1.0 (TID 11)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 10.0 in stage 1.0 (TID 11). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 12) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 11, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 11) in 18 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (11/50)
21/06/09 01:49:20 INFO Executor: Running task 11.0 in stage 1.0 (TID 12)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 11.0 in stage 1.0 (TID 12). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 13) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 12, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 12) in 13 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (12/50)
21/06/09 01:49:20 INFO Executor: Running task 12.0 in stage 1.0 (TID 13)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 12.0 in stage 1.0 (TID 13). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 14) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 13, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 13) in 13 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (13/50)
21/06/09 01:49:20 INFO Executor: Running task 13.0 in stage 1.0 (TID 14)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 13.0 in stage 1.0 (TID 14). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 15) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 14, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 14) in 14 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (14/50)
21/06/09 01:49:20 INFO Executor: Running task 14.0 in stage 1.0 (TID 15)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 14.0 in stage 1.0 (TID 15). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 16) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 15, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 15) in 12 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (15/50)
21/06/09 01:49:20 INFO Executor: Running task 15.0 in stage 1.0 (TID 16)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 15.0 in stage 1.0 (TID 16). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 17) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 16, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 16) in 13 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (16/50)
21/06/09 01:49:20 INFO Executor: Running task 16.0 in stage 1.0 (TID 17)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 16.0 in stage 1.0 (TID 17). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 18) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 17, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 17) in 14 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (17/50)
21/06/09 01:49:20 INFO Executor: Running task 17.0 in stage 1.0 (TID 18)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 17.0 in stage 1.0 (TID 18). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 19) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 18, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 18) in 13 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (18/50)
21/06/09 01:49:20 INFO Executor: Running task 18.0 in stage 1.0 (TID 19)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 18.0 in stage 1.0 (TID 19). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 19.0 in stage 1.0 (TID 20) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 19, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 19) in 14 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (19/50)
21/06/09 01:49:20 INFO Executor: Running task 19.0 in stage 1.0 (TID 20)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 19.0 in stage 1.0 (TID 20). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 20.0 in stage 1.0 (TID 21) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 20, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 19.0 in stage 1.0 (TID 20) in 10 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (20/50)
21/06/09 01:49:20 INFO Executor: Running task 20.0 in stage 1.0 (TID 21)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:20 INFO Executor: Finished task 20.0 in stage 1.0 (TID 21). 1106 bytes result sent to driver
21/06/09 01:49:20 INFO TaskSetManager: Starting task 21.0 in stage 1.0 (TID 22) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 21, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:20 INFO TaskSetManager: Finished task 20.0 in stage 1.0 (TID 21) in 11 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (21/50)
21/06/09 01:49:20 INFO Executor: Running task 21.0 in stage 1.0 (TID 22)
21/06/09 01:49:20 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 21.0 in stage 1.0 (TID 22). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 22.0 in stage 1.0 (TID 23) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 22, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 21.0 in stage 1.0 (TID 22) in 10 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (22/50)
21/06/09 01:49:21 INFO Executor: Running task 22.0 in stage 1.0 (TID 23)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 22.0 in stage 1.0 (TID 23). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 23.0 in stage 1.0 (TID 24) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 23, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 22.0 in stage 1.0 (TID 23) in 9 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (23/50)
21/06/09 01:49:21 INFO Executor: Running task 23.0 in stage 1.0 (TID 24)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 23.0 in stage 1.0 (TID 24). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 24.0 in stage 1.0 (TID 25) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 24, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 23.0 in stage 1.0 (TID 24) in 9 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (24/50)
21/06/09 01:49:21 INFO Executor: Running task 24.0 in stage 1.0 (TID 25)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 24.0 in stage 1.0 (TID 25). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 25.0 in stage 1.0 (TID 26) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 25, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 24.0 in stage 1.0 (TID 25) in 9 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (25/50)
21/06/09 01:49:21 INFO Executor: Running task 25.0 in stage 1.0 (TID 26)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 25.0 in stage 1.0 (TID 26). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 26.0 in stage 1.0 (TID 27) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 26, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 25.0 in stage 1.0 (TID 26) in 9 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (26/50)
21/06/09 01:49:21 INFO Executor: Running task 26.0 in stage 1.0 (TID 27)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 26.0 in stage 1.0 (TID 27). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 27.0 in stage 1.0 (TID 28) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 27, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 26.0 in stage 1.0 (TID 27) in 10 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (27/50)
21/06/09 01:49:21 INFO Executor: Running task 27.0 in stage 1.0 (TID 28)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 27.0 in stage 1.0 (TID 28). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 28.0 in stage 1.0 (TID 29) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 28, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 27.0 in stage 1.0 (TID 28) in 9 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (28/50)
21/06/09 01:49:21 INFO Executor: Running task 28.0 in stage 1.0 (TID 29)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 28.0 in stage 1.0 (TID 29). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 29.0 in stage 1.0 (TID 30) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 29, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 28.0 in stage 1.0 (TID 29) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (29/50)
21/06/09 01:49:21 INFO Executor: Running task 29.0 in stage 1.0 (TID 30)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 29.0 in stage 1.0 (TID 30). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 30.0 in stage 1.0 (TID 31) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 30, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 29.0 in stage 1.0 (TID 30) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (30/50)
21/06/09 01:49:21 INFO Executor: Running task 30.0 in stage 1.0 (TID 31)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 30.0 in stage 1.0 (TID 31). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 31.0 in stage 1.0 (TID 32) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 31, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 30.0 in stage 1.0 (TID 31) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (31/50)
21/06/09 01:49:21 INFO Executor: Running task 31.0 in stage 1.0 (TID 32)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 31.0 in stage 1.0 (TID 32). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 32.0 in stage 1.0 (TID 33) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 32, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 31.0 in stage 1.0 (TID 32) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (32/50)
21/06/09 01:49:21 INFO Executor: Running task 32.0 in stage 1.0 (TID 33)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 32.0 in stage 1.0 (TID 33). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 33.0 in stage 1.0 (TID 34) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 33, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 32.0 in stage 1.0 (TID 33) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (33/50)
21/06/09 01:49:21 INFO Executor: Running task 33.0 in stage 1.0 (TID 34)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 33.0 in stage 1.0 (TID 34). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 34.0 in stage 1.0 (TID 35) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 34, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 33.0 in stage 1.0 (TID 34) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (34/50)
21/06/09 01:49:21 INFO Executor: Running task 34.0 in stage 1.0 (TID 35)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 34.0 in stage 1.0 (TID 35). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 35.0 in stage 1.0 (TID 36) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 35, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 34.0 in stage 1.0 (TID 35) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (35/50)
21/06/09 01:49:21 INFO Executor: Running task 35.0 in stage 1.0 (TID 36)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 35.0 in stage 1.0 (TID 36). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 36.0 in stage 1.0 (TID 37) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 36, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 35.0 in stage 1.0 (TID 36) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (36/50)
21/06/09 01:49:21 INFO Executor: Running task 36.0 in stage 1.0 (TID 37)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 36.0 in stage 1.0 (TID 37). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 37.0 in stage 1.0 (TID 38) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 37, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 36.0 in stage 1.0 (TID 37) in 9 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (37/50)
21/06/09 01:49:21 INFO Executor: Running task 37.0 in stage 1.0 (TID 38)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 37.0 in stage 1.0 (TID 38). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 38.0 in stage 1.0 (TID 39) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 38, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 37.0 in stage 1.0 (TID 38) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (38/50)
21/06/09 01:49:21 INFO Executor: Running task 38.0 in stage 1.0 (TID 39)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 38.0 in stage 1.0 (TID 39). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 39.0 in stage 1.0 (TID 40) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 39, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 38.0 in stage 1.0 (TID 39) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (39/50)
21/06/09 01:49:21 INFO Executor: Running task 39.0 in stage 1.0 (TID 40)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 39.0 in stage 1.0 (TID 40). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 40.0 in stage 1.0 (TID 41) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 40, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 39.0 in stage 1.0 (TID 40) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (40/50)
21/06/09 01:49:21 INFO Executor: Running task 40.0 in stage 1.0 (TID 41)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 40.0 in stage 1.0 (TID 41). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 41.0 in stage 1.0 (TID 42) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 41, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 40.0 in stage 1.0 (TID 41) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (41/50)
21/06/09 01:49:21 INFO Executor: Running task 41.0 in stage 1.0 (TID 42)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 41.0 in stage 1.0 (TID 42). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 42.0 in stage 1.0 (TID 43) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 42, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 41.0 in stage 1.0 (TID 42) in 8 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (42/50)
21/06/09 01:49:21 INFO Executor: Running task 42.0 in stage 1.0 (TID 43)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 42.0 in stage 1.0 (TID 43). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 43.0 in stage 1.0 (TID 44) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 43, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 42.0 in stage 1.0 (TID 43) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (43/50)
21/06/09 01:49:21 INFO Executor: Running task 43.0 in stage 1.0 (TID 44)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 43.0 in stage 1.0 (TID 44). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 44.0 in stage 1.0 (TID 45) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 44, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 43.0 in stage 1.0 (TID 44) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (44/50)
21/06/09 01:49:21 INFO Executor: Running task 44.0 in stage 1.0 (TID 45)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 44.0 in stage 1.0 (TID 45). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 45.0 in stage 1.0 (TID 46) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 45, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 44.0 in stage 1.0 (TID 45) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (45/50)
21/06/09 01:49:21 INFO Executor: Running task 45.0 in stage 1.0 (TID 46)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 45.0 in stage 1.0 (TID 46). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 46.0 in stage 1.0 (TID 47) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 46, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 45.0 in stage 1.0 (TID 46) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (46/50)
21/06/09 01:49:21 INFO Executor: Running task 46.0 in stage 1.0 (TID 47)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (14.8 KiB) non-empty blocks including 1 (14.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 46.0 in stage 1.0 (TID 47). 1068 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 47.0 in stage 1.0 (TID 48) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 47, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 46.0 in stage 1.0 (TID 47) in 6 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (47/50)
21/06/09 01:49:21 INFO Executor: Running task 47.0 in stage 1.0 (TID 48)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 47.0 in stage 1.0 (TID 48). 1068 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 48.0 in stage 1.0 (TID 49) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 48, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 47.0 in stage 1.0 (TID 48) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (48/50)
21/06/09 01:49:21 INFO Executor: Running task 48.0 in stage 1.0 (TID 49)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 48.0 in stage 1.0 (TID 49). 1068 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 49.0 in stage 1.0 (TID 50) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 49, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 48.0 in stage 1.0 (TID 49) in 6 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (49/50)
21/06/09 01:49:21 INFO Executor: Running task 49.0 in stage 1.0 (TID 50)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO Executor: Finished task 49.0 in stage 1.0 (TID 50). 1106 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Finished task 49.0 in stage 1.0 (TID 50) in 7 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (50/50)
21/06/09 01:49:21 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool default
21/06/09 01:49:21 INFO DAGScheduler: ShuffleMapStage 1 (map at Crawler.scala:235) finished in 0.681 s
21/06/09 01:49:21 INFO DAGScheduler: looking for newly runnable stages
21/06/09 01:49:21 INFO DAGScheduler: running: Set()
21/06/09 01:49:21 INFO DAGScheduler: waiting: Set(ResultStage 2)
21/06/09 01:49:21 INFO DAGScheduler: failed: Set()
21/06/09 01:49:21 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[9] at map at Crawler.scala:263), which has no missing parents
21/06/09 01:49:21 INFO DAGScheduler: Jars for session None: Map(dbfs:/FileStore/bcf/commons-compress-1.20.jar -> 1623203351666, dbfs:/FileStore/bcf/sparkler7.jar -> 1623203351666)
21/06/09 01:49:21 INFO DAGScheduler: Files for session None: Map()
21/06/09 01:49:21 INFO DAGScheduler: Archives for session None: Map()
21/06/09 01:49:21 INFO DAGScheduler: Submitting 50 missing tasks from ResultStage 2 (MapPartitionsRDD[9] at map at Crawler.scala:263) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
21/06/09 01:49:21 INFO TaskSchedulerImpl: Adding task set 2.0 with 50 tasks resource profile 0
21/06/09 01:49:21 INFO FairSchedulableBuilder: Added task set TaskSet_2.0 tasks to pool default
21/06/09 01:49:21 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 51) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 0, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 16.3 KiB, free 5.2 GiB)
21/06/09 01:49:21 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.8 KiB, free 5.2 GiB)
21/06/09 01:49:21 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.57.225.53:40357 (size: 7.8 KiB, free: 5.2 GiB)
21/06/09 01:49:21 INFO SparkContext: Created broadcast 2 from broadcast at TaskSetManager.scala:552
21/06/09 01:49:21 INFO Executor: Running task 0.0 in stage 2.0 (TID 51)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO MemoryStore: Block rdd_7_0 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:21 INFO BlockManagerInfo: Added rdd_7_0 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:21 INFO Executor: Finished task 0.0 in stage 2.0 (TID 51). 991 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 52) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 1, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 51) in 195 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (1/50)
21/06/09 01:49:21 INFO Executor: Running task 1.0 in stage 2.0 (TID 52)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO MemoryStore: Block rdd_7_1 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:21 INFO BlockManagerInfo: Added rdd_7_1 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:21 INFO Executor: Finished task 1.0 in stage 2.0 (TID 52). 991 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 53) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 2, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 52) in 155 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (2/50)
21/06/09 01:49:21 INFO Executor: Running task 2.0 in stage 2.0 (TID 53)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO MemoryStore: Block rdd_7_2 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:21 INFO BlockManagerInfo: Added rdd_7_2 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:21 INFO Executor: Finished task 2.0 in stage 2.0 (TID 53). 991 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 54) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 3, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 53) in 153 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (3/50)
21/06/09 01:49:21 INFO Executor: Running task 3.0 in stage 2.0 (TID 54)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO MemoryStore: Block rdd_7_3 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:21 INFO BlockManagerInfo: Added rdd_7_3 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:21 INFO Executor: Finished task 3.0 in stage 2.0 (TID 54). 991 bytes result sent to driver
21/06/09 01:49:21 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 55) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 4, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:21 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 54) in 156 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (4/50)
21/06/09 01:49:21 INFO Executor: Running task 4.0 in stage 2.0 (TID 55)
21/06/09 01:49:21 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:21 INFO MemoryStore: Block rdd_7_4 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:21 INFO BlockManagerInfo: Added rdd_7_4 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:22 INFO Executor: Finished task 4.0 in stage 2.0 (TID 55). 991 bytes result sent to driver
21/06/09 01:49:22 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 56) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 5, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:22 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 55) in 173 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (5/50)
21/06/09 01:49:22 INFO Executor: Running task 5.0 in stage 2.0 (TID 56)
21/06/09 01:49:22 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:22 INFO MemoryStore: Block rdd_7_5 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:22 INFO BlockManagerInfo: Added rdd_7_5 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:22 INFO Executor: Finished task 5.0 in stage 2.0 (TID 56). 991 bytes result sent to driver
21/06/09 01:49:22 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 57) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 6, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:22 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 56) in 152 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (6/50)
21/06/09 01:49:22 INFO Executor: Running task 6.0 in stage 2.0 (TID 57)
21/06/09 01:49:22 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:22 INFO MemoryStore: Block rdd_7_6 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:22 INFO BlockManagerInfo: Added rdd_7_6 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:22 INFO Executor: Finished task 6.0 in stage 2.0 (TID 57). 991 bytes result sent to driver
21/06/09 01:49:22 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 58) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 7, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:22 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 57) in 155 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (7/50)
21/06/09 01:49:22 INFO Executor: Running task 7.0 in stage 2.0 (TID 58)
21/06/09 01:49:22 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:22 INFO MemoryStore: Block rdd_7_7 stored as values in memory (estimated size 16.0 B, free 5.2 GiB)
21/06/09 01:49:22 INFO BlockManagerInfo: Added rdd_7_7 in memory on 10.57.225.53:40357 (size: 16.0 B, free: 5.2 GiB)
21/06/09 01:49:22 INFO Executor: Finished task 7.0 in stage 2.0 (TID 58). 991 bytes result sent to driver
21/06/09 01:49:22 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 59) (ip-10-57-225-53.us-west-2.compute.internal, executor driver, partition 8, PROCESS_LOCAL, taskResourceAssignments Map())
21/06/09 01:49:22 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 58) in 152 ms on ip-10-57-225-53.us-west-2.compute.internal (executor driver) (8/50)
21/06/09 01:49:22 INFO Executor: Running task 8.0 in stage 2.0 (TID 59)
21/06/09 01:49:22 WARN SparkConf: The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Getting 50 (525.1 KiB) non-empty blocks including 50 (525.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
21/06/09 01:49:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/09 01:49:22 INFO DefaultPluginStatusProvider: Enabled plugins: []
21/06/09 01:49:22 INFO DefaultPluginStatusProvider: Disabled plugins: []
21/06/09 01:49:22 INFO DefaultPluginManager: PF4J version 0.0.0 in 'deployment' mode
21/06/09 01:49:22 INFO PluginService$: Loading plugins...
21/06/09 01:49:23 INFO AbstractPluginManager: Plugin 'fetcher-chrome@0.2.2-SNAPSHOT' resolved
21/06/09 01:49:23 INFO AbstractPluginManager: Plugin 'urlfilter-regex@0.2.2-SNAPSHOT' resolved
21/06/09 01:49:23 INFO AbstractPluginManager: Plugin 'url-injector@0.2.2-SNAPSHOT' resolved
21/06/09 01:49:23 INFO AbstractPluginManager: Plugin 'urlfilter-samehost@0.2.2-SNAPSHOT' resolved
21/06/09 01:49:23 INFO PluginService$: 3 plugin(s) Active: [urlfilter-regex, urlfilter-samehost, fetcher-chrome]
21/06/09 01:49:23 WARN PluginService$: 1 extra plugin(s) available but not activated: Set(url-injector)
21/06/09 01:49:23 INFO AbstractPluginManager: Start plugin 'urlfilter-regex@0.2.2-SNAPSHOT'
21/06/09 01:49:23 INFO PluginService$: Extensions found: [edu.usc.irds.sparkler.plugin.RegexURLFilter@50ca765]
21/06/09 01:49:23 INFO PluginService$: Extensions lookup: PluginWrapper [descriptor=PluginDescriptor [pluginId=urlfilter-regex, pluginClass=edu.usc.irds.sparkler.plugin.RegexURLFilterActivator, version=0.2.2-SNAPSHOT, provider=edu.usc.irds.sparkler.plugin, dependencies=[], description=, requires=*, license=null], pluginPath=/dbfs/FileStore/bcf/plugins/urlfilter-regex-0.2.2-SNAPSHOT.jar].getPluginId
21/06/09 01:49:23 INFO PluginService$: Extensions id lookup: edu.usc.irds.sparkler.plugin.RegexURLFilter@50ca765.getClass.getName
21/06/09 01:49:23 INFO AbstractPluginManager: Start plugin 'urlfilter-samehost@0.2.2-SNAPSHOT'
21/06/09 01:49:23 INFO PluginService$: Extensions found: [edu.usc.irds.sparkler.plugin.UrlFilterSameHost@129c801b]
21/06/09 01:49:23 INFO PluginService$: Extensions lookup: PluginWrapper [descriptor=PluginDescriptor [pluginId=urlfilter-samehost, pluginClass=edu.usc.irds.sparkler.plugin.UrlFilterSameHostActivator, version=0.2.2-SNAPSHOT, provider=edu.usc.irds.sparkler.plugin, dependencies=[], description=, requires=*, license=null], pluginPath=/dbfs/FileStore/bcf/plugins/urlfilter-samehost-0.2.2-SNAPSHOT.jar].getPluginId
21/06/09 01:49:23 INFO PluginService$: Extensions id lookup: edu.usc.irds.sparkler.plugin.UrlFilterSameHost@129c801b.getClass.getName
21/06/09 01:49:23 INFO AbstractPluginManager: Start plugin 'fetcher-chrome@0.2.2-SNAPSHOT'
21/06/09 01:49:24 INFO PluginService$: Extensions found: [edu.usc.irds.sparkler.plugin.FetcherChrome@72a81440]
21/06/09 01:49:24 INFO PluginService$: Extensions lookup: PluginWrapper [descriptor=PluginDescriptor [pluginId=fetcher-chrome, pluginClass=edu.usc.irds.sparkler.plugin.FetcherChromeActivator, version=0.2.2-SNAPSHOT, provider=edu.usc.irds.sparkler.plugin, dependencies=[], description=, requires=*, license=null], pluginPath=/dbfs/FileStore/bcf/plugins/fetcher-chrome-0.2.2-SNAPSHOT.jar].getPluginId
21/06/09 01:49:24 INFO PluginService$: Extensions id lookup: edu.usc.irds.sparkler.plugin.FetcherChrome@72a81440.getClass.getName
21/06/09 01:49:24 INFO PluginService$: Recognised Plugins: Map(fetcher-chrome -> edu.usc.irds.sparkler.plugin.FetcherChrome, urlfilter-regex -> edu.usc.irds.sparkler.plugin.RegexURLFilter, urlfilter-samehost -> edu.usc.irds.sparkler.plugin.UrlFilterSameHost)
21/06/09 01:49:24 INFO PluginService$: Initialize class edu.usc.irds.sparkler.plugin.FetcherChrome as fetcher-chrome
21/06/09 01:49:25 INFO Utils: resolved command to be run: WrappedArray(getconf, PAGESIZE)
21/06/09 01:49:37 INFO Capabilities: Using `new ChromeOptions()` is preferred to `DesiredCapabilities.chrome()`
21/06/09 01:49:42 INFO ProtocolHandshake: Detected dialect: W3C
21/06/09 01:49:43 INFO Capabilities: Using `new ChromeOptions()` is preferred to `DesiredCapabilities.chrome()`
21/06/09 01:49:43 INFO ProtocolHandshake: Detected dialect: W3C
21/06/09 01:49:45 WARN PDFParser: J2KImageReader not loaded. JPEG2000 files will not be processed.
See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
for optional dependencies.
21/06/09 01:49:46 INFO ParseFunction$: PARSING https://www.cms.gov/medicare-coverage-database/indexes/article-list.aspx?Cntrctr=319&name=&DocType=Active&ContrVer=1&CntrctrSelected=319*1&bc=AAABAAIAAAAA&#ResultsAnchor
21/06/09 01:49:46 INFO PluginService$: Chaining [edu.usc.irds.sparkler.plugin.RegexURLFilter@657fbc1f, edu.usc.irds.sparkler.plugin.UrlFilterSameHost@6ef092d0] using class edu.usc.irds.sparkler.service.RejectingURLFilterChain
21/06/09 01:49:46 INFO PluginService$: Initialize class edu.usc.irds.sparkler.plugin.RegexURLFilter as urlfilter-regex
21/06/09 01:49:46 INFO PluginService$: Initialize class edu.usc.irds.sparkler.plugin.UrlFilterSameHost as urlfilter-samehost
21/06/09 01:49:46 INFO RejectingURLFilterChain: Initializing edu.usc.irds.sparkler.service.RejectingURLFilterChain with 2 extensions: [edu.usc.irds.sparkler.plugin.RegexURLFilter@657fbc1f, edu.usc.irds.sparkler.plugin.UrlFilterSameHost@6ef092d0]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment