pedroduartecosta

val holdout = model.transform(test).select("prediction", "DelayOutputVar")

val rm = new RegressionMetrics(holdout.rdd.map(x =>
  (x(0).asInstanceOf[Double], x(1).asInstanceOf[Double])))

println("sqrt(MSE): " + Math.sqrt(rm.meanSquaredError))
println("mean absolute error: " + 	rm.meanAbsoluteError)
println("R Squared: " + rm.r2)                         
println("Explained Variance: " + rm.explainedVariance + "\n")

pedroduartecosta

val lr = new LinearRegression()
  .setLabelCol("DelayOutputVar")
  .setFeaturesCol("features")
val paramGrid = new ParamGridBuilder()
  .addGrid(lr.regParam, Array(0.1, 0.01))
  .addGrid(lr.fitIntercept)
  .addGrid(lr.elasticNetParam, Array(0.0, 1.0))
  .build()

val steps:Array[org.apache.spark.ml.PipelineStage] = if(useCategorical){

pedroduartecosta

val categoricalVariables = if(useCategorical){
                            Array("Origin", "Dest")
                          }else{
                            null
                          }

val categoricalIndexers = if(useCategorical){
                            categoricalVariables.map(i => new StringIndexer().setInputCol(i).setOutputCol(i+"Index").setHandleInvalid("skip"))
                          }else{
                            null

pedroduartecosta

val conf = new SparkConf().setAppName("predictor")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)

val rawData = sqlContext.read.format("com.databricks.spark.csv")
            .option("header", "true")
            .option("inferSchema", "true")
            .load(dataPath)
            .withColumn("DelayOutputVar", col("ArrDelay").cast("double"))
            .withColumn("DepDelayDouble", col("DepDelay").cast("double"))

pedroduartecosta

Workshop instructions

• Go to dynamo db

  • Create dynamoDB table
    • Table Name: posts
    • Primary key: id
    • Click create

• Go to S3

• Create s3 bucket

pedroduartecosta

How to set up a NodeJS website with Authentication and Back office (and Blog)

In this tutorial we're going to build a NodeJS website with the following features:

• User authentication
• Back office
• Blog (optional)

The first to parts of the tutorial are meant to quickly set you up and running. The third part is a more in depth explanation of some points I find important, so feel free to read them later as you need.

pedroduartecosta

# Sprint1

#### Start-date: 30/03/18
#### End-date: 06/04/18

# Development Team
- Duarte Brandão
- Pedro Costa
- ...

pedroduartecosta

• Implement CI/CD pipelines
  • Gitlab CI
    • Use Runners
  • Github
    • Use TravisCI or others
  • Necessary stages
    • Build Code
    • Run Tests
      • Code quality
• Unit testing

pedroduartecosta

tensorflow for poets:

https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/

Virtual Box -> Network -> port forward -> 6006 - 6006 and 8888 - 8888

docker run -it -p 6006:6006 -p 8888:8888 -v $HOME/tf_files:/tf_files gcr.io/tensorflow/tensorflow:latest-devel

python /tensorflow/tensorflow/models/image/imagenet/classify_image.py

pedroduartecosta

Data Connection Protocol

We used a script "port.sh" that can virtual emulate a connection through a serial port, you can execute that script by running:

sh port.sh

It's recomended to install socat to use this script.