Skip to content

Instantly share code, notes, and snippets.

Nicola Ferraro nicolaferraro

Block or report user

Report or block nicolaferraro

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View todo-swagger.json
---
swagger: "2.0"
info:
title: "Todo App API"
description: "Example Todo Application API"
version: "1.0.0"
license:
name: "Apache 2.0"
url: "http://www.apache.org/licenses/LICENSE-2.0.html"
host: "hostname"
@nicolaferraro
nicolaferraro / TryMapJava.java
Created Sep 4, 2016
Spark try-map functionality for Java
View TryMapJava.java
package it.test;
import java.io.Serializable;
import java.util.HashSet;
import java.util.LinkedList;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import org.apache.spark.Accumulator;
View stop-words.txt
a
about
above
across
after
afterwards
again
against
all
almost
View streaming print
def setStreamingLogLevels() {
val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements
if (!log4jInitialized) {
// We first log something to initialize Spark's default logging, then we override the
// logging level.
logInfo("Setting log level to [WARN] for streaming example." +
" To override add a custom log4j.properties to the classpath.")
Logger.getRootLogger.setLevel(Level.WARN)
}
}
View hbase pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>it.eng.spark</groupId>
<artifactId>spark-streaming</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
View log4j.properties
log4j.rootLogger=INFO, stdout
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
View hbase pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>it.eng.hbase</groupId>
<artifactId>hbase-connect</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>hbase-connect</name>
View hbase conf
You can use a dockerized version if you want just to try the shell. Just type:
docker run -it -p 2181:2181 -p 60000:60000 -p 60010:60010 -p 60020:60020 -p 60030:60030 -h hbase nerdammer/hbase-shell
Then add “hbase” to your “/etc/hosts” file and point it to boot2docker (192.168.59.103) address.
If you want to install it:
Download: http://archive.eu.apache.org/dist/hbase/hbase-1.0.0/hbase-1.0.0-bin.tar.gz
Untar
Run: bin/start-hbase.sh
For the shell: bin/hbase shell
If you want to install it on Windows:
View CrawlerService.scala
package it.eng.scala.crawl
import org.jsoup.Jsoup
import scala.collection.JavaConversions._
object CrawlerService {
val AbsolutePrefix = "http://en.wikipedia.org/wiki/"
val RelativePrefix = "/wiki/"
val AbsolutePath = "http://en.wikipedia.org"
View jsoup
<dependency>
<groupId>net.databinder.dispatch</groupId>
<artifactId>dispatch-jsoup_2.10</artifactId>
<version>0.11.2</version>
</dependency>
def retrieveLinks(address: String) =
Jsoup.
connect(address)
.get
You can’t perform that action at this time.