Skip to content

Instantly share code, notes, and snippets.

@velotiotech

velotiotech/Analysis_Metrics.scala

Last active January 27, 2021 07:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save velotiotech/21d701e7bc2e1e1a84a4af59a153da0f to your computer and use it in GitHub Desktop.
Save velotiotech/21d701e7bc2e1e1a84a4af59a153da0f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Analysis_Metrics.scala
val runAnalyzer: AnalyzerContext = { AnalysisRunner
.onData(data)
.addAnalyzer(Size())
.addAnalyzer(Completeness("averageRating"))
.addAnalyzer(Uniqueness("tconst"))
.addAnalyzer(Mean("averageRating"))
.addAnalyzer(StandardDeviation("averageRating"))
.addAnalyzer(Compliance("top rating", "averageRating >= 7.0"))
.addAnalyzer(Correlation("numVotes", "averageRating"))
.addAnalyzer(Distinctness("tconst"))
.addAnalyzer(Maximum("averageRating"))
.addAnalyzer(Minimum("averageRating"))
.run()
}
val metricsResult = successMetricsAsDataFrame(spark, runAnalyzer)
metricsResult.show(false)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment