Skip to content

Instantly share code, notes, and snippets.

View giaosudau's full-sized avatar

Chanh Le giaosudau

63 followers · 254 following
  • Vietnam
View GitHub Profile
@giaosudau
giaosudau / hadoop logs
Created August 15, 2016 04:14
hadoop logs
2016-08-03 10:05:47,121 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.ConnectException: Call From spark-dev-03/127.0.0.1 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.ConnectException: Call From spark-dev-03/127.0.0.1 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.ser
@giaosudau
giaosudau / logs parquet druid
Created August 12, 2016 03:22
2016-08-12T03:17:24,179 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2016-08-12T03:17:24,182 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties
2016-08-12T03:17:24,210 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2016-08-12T03:17:24,664 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='dist/druid/extensions', hadoopDependenciesDir='dist/druid/hadoop-dependencies', hadoopContainerDruidClasspath='null', loadList=[druid-datasketches, druid-avro-extensions, druid-parquet-extensions, postgresql-metadata-storage, druid-hdfs-storage, druid-histogram, druid-datasketches, druid-kafka-indexing-service]}]
2016-08-12T03:17:24,669 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [io.druid
@giaosudau
giaosudau / druid task log
Created August 11, 2016 08:48
2016-08-11T01:22:00,555 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2016-08-11T01:22:00,557 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties
2016-08-11T01:22:00,592 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2016-08-11T01:22:01,143 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='dist/druid/extensions', hadoopDependenciesDir='dist/druid/hadoop-dependencies', hadoopContainerDruidClasspath='null', loadList=[druid-datasketches, druid-avro-extensions, druid-parquet-extensions, postgresql-metadata-storage, druid-hdfs-storage]}]
2016-08-11T01:22:01,150 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [io.druid.cli.CliCommandCreator]
2016-08-11T01:22:01,161 INFO [main] io.drui
@giaosudau
giaosudau / gist:dd491faa14919e251a9b66363499257d
Created July 27, 2016 10:58
16/07/27 17:56:42 WARN CsvRelation$: Dropping malformed line: 2016-07-25-16,-1,-1,-1,1,2,1,1,24,-1,2,24608,532112262,24828,532385205,548281807,29707,,21895362,1,0,0,0,0.0,0.0,0.0
16/07/27 17:56:42 WARN CsvRelation$: Dropping malformed line: 2016-07-25-16,36,-1,-1,1,1,1,1,29,-1,1,10507,545808095,114,545808096,545809079,15150,,557802881,0,1,0,0,0.0,0.0,0.0
16/07/27 17:56:42 WARN CsvRelation$: Dropping malformed line: 2016-07-25-16,-1,-1,-1,1,2,1,1,57,-1,3,24608,532112262,24821,532152715,548281807,29707,,2402811497,1,0,0,0,0.0,0.0,0.0
16/07/27 17:56:42 WARN CsvRelation$: Dropping malformed line: 2016-07-25-16,-1,-1,-1,1,1,1,1,24,-1,3,24608,535429652,26513,535429657,535587269,26461,,2360615763,2,0,0,0,0.0,0.0,0.0
16/07/27 17:56:42 WARN CsvRelation$: Dropping malformed line: 2016-07-25-16,-1,-1,-1,1,1,1,1,24,-1,3,24608,532112262,24821,532152715,579014381,42491,,629474068,1,0,0,0,0.0,0.0,0.0
16/07/27 17:56:42 WARN CsvRelation$: Dropping malformed line: 2016-07-25-16,-1,-1,-1,1,1,1,1,24,-1,3,24608,532112262,25072,53
@giaosudau
giaosudau / logs
Created July 19, 2016 07:15
k_id=10507/part-r-00000-7bc39af2-2631-4f94-aedd-7a9eb871326d.snappy.parquet start: 0 end: 35842250 length: 35842250 hosts: []}
16/07/19 14:12:25 INFO type: getFileStatus(alluxio://master1:19998/AD_COOKIE_REPORT/time=2016-07-17-23/network_id=10507/part-r-00000-7bc39af2-2631-4f94-aedd-7a9eb871326d.snappy.parquet)
16/07/19 14:12:25 INFO type: open(alluxio://master1:19998/AD_COOKIE_REPORT/time=2016-07-17-23/network_id=10507/part-r-00000-7bc39af2-2631-4f94-aedd-7a9eb871326d.snappy.parquet, 65536)
16/07/19 14:12:25 INFO type: Connecting to remote worker @ master2/10.197.0.4:29998
16/07/19 14:12:26 INFO type: Connected to remote machine master2/10.197.0.4:29999
16/07/19 14:12:26 INFO type: Data 3760361308160 from remote machine master2/10.197.0.4:29999 received
16/07/19 14:12:26 INFO type: Connected to remote machine master2/10.197.0.4:29999
16/07/19 14:12:26 INFO type: Data 3760361308160 from remote machine master2/10.197.0.4:29999 received
16/07/19 14:12:26 INFO type: Connected to remote machine master2/10.197.0.4
@giaosudau
giaosudau / build.sbt
Created June 16, 2016 14:28
Spark SBT Build File Example
name := "ImpressionMetric"
version := "1.0"
scalaVersion := "2.10.6"
// Spark Dependencies
libraryDependencies ++= Seq(
@giaosudau
giaosudau / log file
Created June 10, 2016 07:43
spark job server error dynamic allocation
[2016-06-10 14:42:22,137] INFO ocalContextSupervisorActor [] [akka://JobServer/user/context-supervisor] - Creating a SparkContext named sql_context-1
[2016-06-10 14:42:22,138] INFO .jobserver.JobManagerActor [] [akka://JobServer/user/context-supervisor/sql_context-1] - Starting actor spark.jobserver.JobManagerActor
[2016-06-10 14:42:22,140] INFO k.jobserver.JobStatusActor [] [akka://JobServer/user/context-supervisor/sql_context-1/$a] - Starting actor spark.jobserver.JobStatusActor
[2016-06-10 14:42:22,140] INFO k.jobserver.JobResultActor [] [akka://JobServer/user/context-supervisor/sql_context-1/$b] - Starting actor spark.jobserver.JobResultActor
[2016-06-10 14:42:22,144] WARN .apache.spark.SparkContext [] [akka://JobServer/user/context-supervisor/sql_context-1] - Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243). The other SparkContext was created at:
org.apache.spark.
@giaosudau
giaosudau / Log error
Created May 27, 2016 06:52
Log Spark error
java.io.IOException: org.apache.spark.storage.BlockFetchException: Failed to fetch block from 1 locations. Most recent failure cause:
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:65)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
@giaosudau
giaosudau / private.xml
Created March 26, 2016 15:27
Karabiner for M545
<?xml version="1.0"?>
<root>
<item>
<name>Logitech M545 Mouse</name>
<devicevendordef>
<vendorname>LOGITECH</vendorname>
<vendorid>0x046d</vendorid>
</devicevendordef>
<deviceproductdef>