Skip to content

Instantly share code, notes, and snippets.

Pedro Costa pedroduartecosta

  • Porto
View GitHub Profile
View keybase.md

Keybase proof

I hereby claim:

  • I am pedroduartecosta on github.
  • I am pedroc (https://keybase.io/pedroc) on keybase.
  • I have a public key ASAzPvCsscwUOE487ulXNKdyfk-Vfu-9gwYaWGmGJq1f-wo

To claim this, I am signing this object:

View metrics.scala
val holdout = model.transform(test).select("prediction", "DelayOutputVar")
val rm = new RegressionMetrics(holdout.rdd.map(x =>
(x(0).asInstanceOf[Double], x(1).asInstanceOf[Double])))
println("sqrt(MSE): " + Math.sqrt(rm.meanSquaredError))
println("mean absolute error: " + rm.meanAbsoluteError)
println("R Squared: " + rm.r2)
println("Explained Variance: " + rm.explainedVariance + "\n")
View creating-features-vector.scala
val categoricalVariables = if(useCategorical){
Array("Origin", "Dest")
}else{
null
}
val categoricalIndexers = if(useCategorical){
categoricalVariables.map(i => new StringIndexer().setInputCol(i).setOutputCol(i+"Index").setHandleInvalid("skip"))
}else{
null
View linear-regression-implementatio.scala
val lr = new LinearRegression()
.setLabelCol("DelayOutputVar")
.setFeaturesCol("features")
val paramGrid = new ParamGridBuilder()
.addGrid(lr.regParam, Array(0.1, 0.01))
.addGrid(lr.fitIntercept)
.addGrid(lr.elasticNetParam, Array(0.0, 1.0))
.build()
val steps:Array[org.apache.spark.ml.PipelineStage] = if(useCategorical){
View reading-data.scala
val conf = new SparkConf().setAppName("predictor")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val rawData = sqlContext.read.format("com.databricks.spark.csv")
.option("header", "true")
.option("inferSchema", "true")
.load(dataPath)
.withColumn("DelayOutputVar", col("ArrDelay").cast("double"))
.withColumn("DepDelayDouble", col("DepDelay").cast("double"))
View SINf_Workshop.md

Workshop instructions

  • Go to dynamo db

    • Create dynamoDB table
      • Table Name: posts
      • Primary key: id
      • Click create
  • Go to S3

    • Create s3 bucket
View Wiki sprints
# Sprint1
#### Start-date: 30/03/18
#### End-date: 06/04/18
# Development Team
- Duarte Brandão
- Pedro Costa
- ...
@pedroduartecosta
pedroduartecosta / bestpractices.md
Last active Mar 31, 2018
Project Development Best Practices
View bestpractices.md
  • Implement CI/CD pipelines
    • Gitlab CI
      • Use Runners
    • Github
      • Use TravisCI or others
    • Necessary stages
      • Build Code
      • Run Tests
        • Code quality
        • Unit testing
View How to set up a NodeJS website with Authentication and Back office (and Blog).md

How to set up a NodeJS website with Authentication and Back office (and Blog)

In this tutorial we're going to build a NodeJS website with the following features:

  • User authentication
  • Back office
  • Blog (optional)

The first to parts of the tutorial are meant to quickly set you up and running. The third part is a more in depth explanation of some points I find important, so feel free to read them later as you need.

You can’t perform that action at this time.