Skip to content

Instantly share code, notes, and snippets.

View Habitats's full-sized avatar
🚀
What's happening?

Patrick Skjennum Habitats

🚀
What's happening?
View GitHub Profile
@Habitats
Habitats / settings.json
Created March 2, 2021 10:12
Windows Terminal tmux-friendly Settings
{
"$schema": "https://aka.ms/terminal-profiles-schema",
// Add custom actions and keybindings to this array.
// To unbind a key combination from your defaults.json, set the command to "unbound".
// To learn more about actions and keybindings, visit https://aka.ms/terminal-keybindings
"actions":
[
// Copy and paste are bound to Ctrl+Shift+C and Ctrl+Shift+V in your defaults.json.
// These two lines additionally bind them to Ctrl+C and Ctrl+V.
// To learn more about selection, visit https://aka.ms/terminal-selection
case class Field(field: String, value: Any)
case class Fields(fields: List[Field], score: Double, siteId: String, id: String)
def hello: Unit = {
val rdd = sc.textFile("hello.json").map { json =>
import org.json4s.jackson.JsonMethods._
implicit val jsonFormats = DefaultFormats
val fields = parse(json).extract[Fields]
for {
field <- fields.fields if field.field == "body"
# Norweigan
## Shopping
- [aHR0cDovL3d3dy5hdmlzYS1zdC5uby9ueWhldGVyLzIwMTcvMTEvMjIvTiVDMyVBNS1lci1kZXQtZiVDMyVBNS1kYWdlci10aWwtJUMzJUE1cmV0cy12aWxsZXN0ZS1oYW5kbGVkYWctMTU2Mzg5OTQuZWNl](http://omni.mito.ai/news/aHR0cDovL3d3dy5hdmlzYS1zdC5uby9ueWhldGVyLzIwMTcvMTEvMjIvTiVDMyVBNS1lci1kZXQtZiVDMyVBNS1kYWdlci10aWwtJUMzJUE1cmV0cy12aWxsZXN0ZS1oYW5kbGVkYWctMTU2Mzg5OTQuZWNl/annotate)
- [aHR0cHM6Ly93d3cuaWZpbm5tYXJrLm5vL2FsdGEvb3Nsby90cm9tc28vZGV0dGUtYmlsZGV0LWVyLXRhdHQtcGEtbm9yd2VnaWFucy1mbHktbWVsbG9tLWFsdGEtb2ctb3Nsby10aWwtdHJvbXNvLXZhci1kZXQtc21la2tmdWxsdC9vLzUtODEtNjM1MTg2](http://omni.mito.ai/news/aHR0cHM6Ly93d3cuaWZpbm5tYXJrLm5vL2FsdGEvb3Nsby90cm9tc28vZGV0dGUtYmlsZGV0LWVyLXRhdHQtcGEtbm9yd2VnaWFucy1mbHktbWVsbG9tLWFsdGEtb2ctb3Nsby10aWwtdHJvbXNvLXZhci1kZXQtc21la2tmdWxsdC9vLzUtODEtNjM1MTg2/annotate)
## News
- [aHR0cDovL3d3dy5hZHJlc3NhLm5vL255aGV0ZXIvdHJvbmRlbGFnLzIwMTcvMTEvMjQvQnJhbm4taS1iaWwtaS1Pcmthbmdlci1pLW5hdHQtMTU2NTAwNzUuZWNl](http://omni.mito.ai/news/aHR0cDovL3d3dy5hZHJlc3NhLm5vL255aGV0ZXIvdHJvbmRlbGFnLzIwMTc
build.gradle:
plugins {
id 'idea'
}
subprojects{
apply plugin: 'scala'
repositories {
INFO [2016-03-04 14:03:57,816] org.eclipse.jetty.util.log: Logging initialized @4270ms
INFO [2016-03-04 14:03:57,903] io.dropwizard.assets.AssetsBundle: Registering AssetBundle with name: assets for path /assets/*
port: 0
port: 0
WARN [2016-03-04 14:03:57,977] org.glassfish.jersey.internal.Errors: The following warnings have been detected: WARNING: Cannot create new registration for component type class org.deeplearning4j.ui.exception.GenericExceptionMapper: Existing previous registration found for the type.
INFO [2016-03-04 14:03:57,986] io.dropwizard.server.ServerFactory: Starting UiServer
WARN [2016-03-04 14:03:58,017] org.glassfish.jersey.internal.Errors: The following warnings have been detected: WARNING: Cannot create new registration for component type class io.dropwizard.jersey.jackson.JsonProcessingExceptionMapper: Existing previous registration found for the type.
INFO [2016-03-04 14:03:58,071] org.eclipse.jetty.setuid.SetUIDListener: Opened application@2f79f192{HTTP/1.1}{0.0.0.0
log4j.appender.myConsoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.myConsoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.myConsoleAppender.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
log4j.appender.RollingAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppender.File=/var/log/spark.log
log4j.appender.RollingAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppender.layout.ConversionPattern=[%p] %d %c %M - %m%n
For a binary classification with two outputs, eihter [1,0] or [0,1],
and the Evalution object, the following outputs are generated ...
The following confusion matrix:
Predicted Class
0 1 Total
Actual Class 0 1500 2 1502
1 68 131 199
Total 1568 133
import java.util
import org.deeplearning4j.datasets.iterator.DataSetIterator
import org.nd4j.linalg.dataset.DataSet
import org.nd4j.linalg.dataset.api.DataSetPreProcessor
import org.nd4j.linalg.factory.Nd4j
import scala.collection.JavaConverters._
import scala.util.Random
import java.util
import org.deeplearning4j.datasets.iterator.DataSetIterator
import org.nd4j.linalg.dataset.DataSet
import org.nd4j.linalg.dataset.api.DataSetPreProcessor
import org.nd4j.linalg.factory.Nd4j
import scala.collection.JavaConverters._
import scala.util.Random