View DataAnalysisIntro7.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
val osAovCount = data.filter { x => interestedBrowsers.contains(x.osFamily)} | |
.filter (_.categoryId == 128) | |
.map { x => x.osFamily } | |
.countByValue | |
.toSeq | |
draw(column(osAovCount.toList)) |
View DataAnalysisIntro6.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//CODE of MIC | |
import data.VarPairData | |
import mine.core.MineParameters | |
import analysis.Analysis | |
import analysis.results.BriefResult | |
import scala.util.Random | |
//Code a discrete value by randomly changing the order of the "codes | |
def encode(col: Array[String]): Array[Double] = { |
View DataAnalysisIntro5.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//CODE | |
import org.ddahl.rscala._ | |
import ru.retailrocket.ispark._ | |
def connect() = RClient("R", false) | |
@transient | |
val r = connect() | |
R.plot(r, "barplot(cs, names.arg=names, col = 'red' )", Map("cs"->os.Aov.map(_._2), "names" -> osAov.map(_._1)) ) |
View DataAnalysisIntro4.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//CODE of Highcharts | |
import com.quantifind.charts.Highcharts._ | |
import ru.retailrocket.ispark.wisp._ | |
draw(column(osAov.toList)) |
View DataAnalysisIntro3.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//CODE: | |
val interestedBrowsers = List("Android", "OS X", "iOS", "Linux", "Windows") | |
val osAov = dataAov.filter(x => interestedBrowsers.contains(x.osFamily)) //we leave only the desired OS | |
.filter(_.categoryId == 128) // filter categories | |
.map(x => (x.osFamily, (x.aov, 1.0))) // need to calculate average purchase amount | |
.reduceByKey((x, y) => (x._1 + y._1, x._2 + y._2)) | |
.map{ case(osFamily, (revenue, orders)) => (osFamily, revenue/orders) } | |
.collect() | |
//OUT |
View DataAnalysisIntro2.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//CODE: | |
//The most popular category. | |
dataAov.map { x => x.categoryId } // select the categoryId field | |
.countByValue() // calculate how often each categoryId appears | |
.toSeq | |
.sortBy( - _._2) // sort by frequency in descending order | |
.take(10) //take the top 10 records | |
//OUT: | |
//format: (categoryId, count) |
View DataAnalysisIntro1.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//CODE: | |
import org.apache.spark.rdd.RDD | |
import org.apache.spark.sql._ | |
import org.tribbloid.ispark.display.dsl._ | |
import scala.util.Try | |
val sqlContext = new org.apache.spark.sql.SQLContext(sc) | |
import sqlContext.implicits._ |
View DataAnalysisIntro.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//CODE: | |
import org.apache.spark.rdd.RDD | |
import org.apache.spark.sql._ | |
import org.tribbloid.ispark.display.dsl._ | |
import scala.util.Try | |
val sqlContext = new org.apache.spark.sql.SQLContext(sc) | |
import sqlContext.implicits._ |
View Nd4j_lstm.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import org.nd4j.linalg.api.ndarray.INDArray | |
import org.nd4j.linalg.api.ops.impl.broadcast.BroadcastAddOp | |
import org.nd4j.linalg.api.rng.distribution.impl.UniformDistribution | |
import org.nd4j.linalg.factory.Nd4j | |
import scala.util.Random | |
object Xavier { | |
def init(shape: Array[Int]): INDArray = { | |
var n = shape.dropRight(1).product |
View gist:d5aa7e065203a2a7221ce1bbf663f511
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package org.deeplearning4j.learn | |
import java.util.Arrays | |
import org.nd4j.api.linalg.DSL._ | |
import org.nd4j.linalg.api.ndarray.INDArray | |
import org.nd4j.linalg.factory.Nd4j | |
import org.nd4j.linalg.indexing.{NDArrayIndex, BooleanIndexing} | |
import org.nd4j.linalg.indexing.conditions.Conditions | |
import org.nd4j.linalg.indexing.functions.Value |
NewerOlder