This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import math._ | |
object DistanceMetrics { | |
/** | |
* distance method helps get the Euclidean Distance between two vectors | |
* Euclidean Distance tests the space betweeen two Vectors and the result is | |
* between 0 and 2 where: | |
* 0 means the space between them is 0 | |
* 2 meanns the space between the two vector is at maximum | |
* @param xs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//Extracting Values from Optional in Scala | |
//a. This works well if you have a defined value for your else | |
val value = Some("Data") | |
value.getOrElse("No Value") | |
//b Using match you can extract the value and do anything you want with it | |
value match { | |
case Some(i) => println(s"There is a value $i") | |
case _ => println("There is nothing in value") | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def decodeImageString(src: String): Unit = { | |
//Decode Image String from base64 | |
val imageByte = new ByteArrayInputStream(java.util.Base64.getDecoder.decode(src)) | |
val image = ImageIO.read(imageByte) | |
val outputFile = new File("Image.jpeg") | |
ImageIO.write(image, "jpeg", outputFile) | |
} | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.mlserver.reactivefacenet.modelserver | |
import java.io.InputStream | |
import com.mlserver.reactivefacenet.utility.ResourceUtils | |
import org.dmg.pmml.{FieldName, PMML} | |
import org.jpmml.evaluator.{Computable, FieldValue, ModelEvaluatorFactory} | |
import org.jpmml.model.PMMLUtil | |
import org.jpmml.evaluator.visitors._ | |
import sun.misc.BASE64Decoder |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//Create new Kafka topic | |
def createTopic(topic: String, client: AdminClient) = { | |
val newTopic = new NewTopic(topic, 6, 3.toShort) | |
try { | |
val topic = client.createTopics(List(newTopic).asJavaCollection) | |
topic.values().get(topic).get() //Not Scala design here | |
} catch { | |
case e: InterruptedException with ExecutionException=> { | |
case d: TopicExistsException => throw new RuntimeException(e.getMessage, e) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.sparkserver.sparkstream | |
import org.apache.spark.mllib.clustering.StreamingKMeansModel | |
import org.apache.spark.streaming.{ Seconds, StreamingContext } | |
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent | |
import org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe | |
//import org.apache.spark.{ SparkConf, SparkContext } | |
import org.apache.spark.streaming.kafka010._ | |
import sun.misc.BASE64Decoder | |
import org.apache.spark.mllib.clustering.StreamingKMeans |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import org.apache.kafka.clients.producer.KafkaProducer | |
import org.apache.kafka.streams.{StreamBuilder, StreamsConfig} | |
import java.util.Properties | |
val config: Properties = { | |
val p = new Properties() | |
p.put(StreamsConfig.APPLICATION_ID_CONFIG, "simple-confluent-stream") | |
p.put(StreamsConfig.CLIENT_ID_CONFIG, "simple-confluent-stream-client") | |
p.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//Given an incoming message to train a Streaming Machine Learning Model | |
val signaturesToTrainModelsOn = 1 to max | |
val trainTopic = "signatures" | |
signaturesToTrainModelsOn.foreach { n => | |
streams.send(new ProducerRecord(trainTopic, "sigs", s"$n")) | |
} | |
//Given an incoming message to predict | |
val signaturesToPredictOn = maths.random() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
object SparkStreamingOnEmbeddings extends App { | |
val spark = SparkSession.builder | |
.master("local[*]") | |
.appName("StreamEmbeddings") | |
.getOrCreate() | |
val sc = spark.sparkContext | |
val ssc = new StreamingContext(sc, Seconds(10)) | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import org.apache.kafka.clients.consumer.ConsumerConfig | |
import org.apache.kafka.common.serialization.StringDeserializer | |
object KafkaSupport { | |
def getKafkaConsumerConfig(brokers: String, groupID: String): Map[String, String] = { | |
Map[String, String]( | |
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> brokers, | |
ConsumerConfig.GROUP_ID_CONFIG -> groupID, |
OlderNewer