Russell Jurney rjurney
rjurney
/ beaconingFeature.scala
Created
April 13, 2014 21:19
How to return an eventTime sorted array of values for each unique grouping of (periodSeconds, cIp, csHost, requestMethod, userAgent)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.securityx.modelfeature.resources | |
| import javax.ws.rs.{QueryParam, GET, Produces, Path} | |
| import scala.Array | |
| import javax.ws.rs.core.{Response, MediaType} | |
| import org.slf4j.{LoggerFactory, Logger} | |
| import org.joda.time.format.{ISODateTimeFormat, DateTimeFormatter} | |
| import org.joda.time.DateTimeZone | |
| import com.securityx.modelfeature.dao.{FeatureDao, BeaconActivityDao} |
rjurney
/ BeaconActivityFeature.scala
Created
April 15, 2014 19:04
Scala for time series controller - how can I improve/speed this up?
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def fillZeros(startDateStr : String, endDateStr : String, periodSeconds : Int, jsonKey : scala.collection.immutable.Map[String,Any], group : ListBuffer[collection.mutable.Map[String, Any]]): scala.collection.mutable.ListBuffer[scala.collection.mutable.Map[String,Any]] = { | |
| var finalTimeSeries = ListBuffer[collection.mutable.Map[String, Any]]() | |
| var startDate = MutableDateTime.parse(startDateStr) | |
| val endDate = MutableDateTime.parse(endDateStr) | |
| while(startDate.isBefore(endDate)) { | |
| println(startDate.toString + " is before " + endDate.toString) | |
| // If our results have an entry for this timestamp, append it to the final array | |
| val searchResult = group.filter(x => x.get("eventTime") == Some(startDate.toString)) | |
| if (searchResult.length > 0) { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| scala> val avroRdd = sc.newAPIHadoopFile("hdfs://hivecluster2/securityx/web_proxy_mef/2014/05/27/19/*", classOf[AvroKeyInputFormat[GenericRecord]], classOf[AvroKey[GenericRecord]], classOf[NullWritable]) | |
| 14/05/27 17:02:49 INFO storage.MemoryStore: ensureFreeSpace(167954) called with curMem=0, maxMem=308713881 | |
| 14/05/27 17:02:49 INFO storage.MemoryStore: Block broadcast_0 stored as values to memory (estimated size 164.0 KB, free 294.3 MB) | |
| avroRdd: org.apache.spark.rdd.RDD[(org.apache.avro.mapred.AvroKey[org.apache.avro.generic.GenericRecord], org.apache.hadoop.io.NullWritable)] = NewHadoopRDD[0] at newAPIHadoopFile at <console>:23 | |
| scala> avroRdd.take(1) | |
| 14/05/27 17:03:05 INFO input.FileInputFormat: Total input paths to process : 21 | |
| 14/05/27 17:03:05 INFO spark.SparkContext: Starting job: take at <console>:26 | |
| 14/05/27 17:03:05 INFO scheduler.DAGScheduler: Got job 0 (take at <console>:26) with 1 output partitions (allowLocal=true) | |
| 14/05/27 17:03:05 INFO scheduler.DAGScheduler: Final stage: Stage 0 (take at <cons |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.SparkContext | |
| import org.apache.spark.SparkContext._ | |
| import org.apache.avro.generic.GenericRecord | |
| import org.apache.avro.mapred.AvroKey | |
| import org.apache.avro.mapreduce.AvroKeyInputFormat | |
| import org.apache.hadoop.io.NullWritable | |
| import org.apache.commons.lang.StringEscapeUtils.escapeCsv | |
| import org.apache.avro.file.DataFileStream |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.SparkContext | |
| import org.apache.spark.SparkContext._ | |
| import org.apache.avro.generic.GenericRecord | |
| import org.apache.avro.mapred.AvroKey | |
| import org.apache.avro.mapreduce.AvroKeyInputFormat | |
| import org.apache.hadoop.io.NullWritable | |
| import org.apache.commons.lang.StringEscapeUtils.escapeCsv | |
| val file = sc.textFile("hdfs://hivecluster2/securityx/beaconing_activity.txt/2014/05/12/14/hour") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| scala> val avroRdd = sc.newAPIHadoopFile("hdfs://hivecluster2/securityx/web_proxy_mef/2014/05/27/19/part-m-00019.avro", | |
| | classOf[AvroKeyInputFormat[GenericRecord]], | |
| | classOf[AvroKey[GenericRecord]], | |
| | classOf[NullWritable]) | |
| 14/05/27 19:44:58 INFO storage.MemoryStore: ensureFreeSpace(167562) called with curMem=369637, maxMem=308713881 | |
| 14/05/27 19:44:58 INFO storage.MemoryStore: Block broadcast_3 stored as values to memory (estimated size 163.6 KB, free 293.9 MB) | |
| avroRdd: org.apache.spark.rdd.RDD[(org.apache.avro.mapred.AvroKey[org.apache.avro.generic.GenericRecord], org.apache.hadoop.io.NullWritable)] = NewHadoopRDD[7] at newAPIHadoopFile at <console>:41 | |
| scala> val genericRecords = avroRdd.map{case (ak, _) => ak.datum()} | |
| genericRecords: org.apache.spark.rdd.RDD[org.apache.avro.generic.GenericRecord] = MappedRDD[8] at map at <console>:43 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.SparkContext | |
| import org.apache.spark.SparkContext._ | |
| import org.apache.spark.SparkConf | |
| import org.apache.avro.generic.GenericRecord | |
| import org.apache.avro.mapred.AvroKey | |
| import org.apache.avro.mapred.AvroInputFormat | |
| import org.apache.avro.mapreduce.AvroKeyInputFormat | |
| import org.apache.hadoop.io.NullWritable | |
| import org.apache.commons.lang.StringEscapeUtils.escapeCsv |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.avro.generic.GenericRecord | |
| import org.apache.avro.mapred.AvroKey | |
| import org.apache.avro.mapred.AvroInputFormat | |
| import org.apache.avro.mapreduce.AvroKeyInputFormat | |
| import org.apache.hadoop.io.NullWritable | |
| import org.apache.commons.lang.StringEscapeUtils.escapeCsv | |
| import org.apache.hadoop.fs.FileSystem | |
| import org.apache.hadoop.fs.Path | |
| import org.apache.hadoop.conf.Configuration |
rjurney
/ stdout
Created
June 6, 2014 04:22
Results of attempt to use https://gist.github.com/massie/7224868
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <console>:43: error: type mismatch; | |
| found : Class[T] | |
| required: org.apache.avro.Schema | |
| val reader = new GenericDatumReader[T](classManifest[T].erasure.asInstanceOf[Class[T]]) | |
| ^ | |
| <console>:44: error: type mismatch; | |
| found : Class[T] | |
| required: org.apache.avro.Schema | |
| val writer = new GenericDatumWriter[T](classManifest[T].erasure.asInstanceOf[Class[T]]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Russells-MacBook-Pro:azkaban2 rjurney$ git remote -v | |
| origin https://github.com/azkaban/azkaban2.git (fetch) | |
| origin https://github.com/azkaban/azkaban2.git (push) | |
| rjurney git@github.com:rjurney/azkaban2.git (fetch) | |
| rjurney git@github.com:rjurney/azkaban2.git (push) | |
| Russells-MacBook-Pro:azkaban2 rjurney$ git branch -v | |
| master cb88bf7 [ahead 60] Merge branch 'master' of https://github.com/azkaban/azkaban2 | |
| * remotes/origin/release-2.5 963a85f All my hacks to Azkaban. | |
| v cb88bf7 Merge branch 'master' of https://github.com/azkaban/azkaban2 |
OlderNewer