Skip to content

Instantly share code, notes, and snippets.

asmaier

Block or report user

Report or block asmaier

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@asmaier
asmaier / python_type_hints.md
Last active Jan 16, 2018
Minimal overview of Python Type Hints
View python_type_hints.md

Python Type Hints

  • Type after variable
  • Problem: IDE cannot suggest variable name based on type

Primitives

From

// primitive function
@asmaier
asmaier / load_parquet_s3.py
Last active Jun 21, 2019
Pyspark script for downloading a single parquet file from Amazon S3 via the s3a protocol. It also reads the credentials from the "~/.aws/credentials", so we don't need to hardcode them. See also https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html .
View load_parquet_s3.py
#
# Some constants
#
aws_profile = "your_profile"
aws_region = "your_region"
s3_bucket = "your_bucket"
#
# Reading environment variables from aws credential file
#
View start_cloudera.sh
#!/bin/bash
docker run --name cdh --hostname "quickstart.cloudera" --privileged=true -t -i -d \
-p 80:80 \
-p 7180:7180 \
-p 8888:8888 \
cloudera/quickstart /usr/bin/docker-quickstart
docker exec -ti cdh /home/cloudera/cloudera-manager --express
View hdp26-spark2-livy-cluster.json
{
"host_groups": [
{
"name": "host_group_client_1",
"configurations": [],
"components": [
{
"name": "ZOOKEEPER_CLIENT"
},
{
@asmaier
asmaier / wir-sind-bild.ipynb
Last active Aug 29, 2015
Unsmoothed Maximum Likelihood Character Level Language Model to create headlines for the german Bild-Zeitung
View wir-sind-bild.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@asmaier
asmaier / ShowSystemProperties.java
Created Apr 8, 2014
Small java program showing a sorted list of all java system properties. Useful for debugging, e.g. encoding issues. Compile and run with "javac ShowSystemProperties.java; java ShowSystemProperties".
View ShowSystemProperties.java
import java.util.Map;
import java.util.Properties;
import java.util.TreeMap;
/**
* Shows a sorted list of all system properties
*/
public class ShowSystemProperties {
public static void main(String[] argv) {
@asmaier
asmaier / KafkaProducerIT.java
Last active Dec 22, 2018
Simple java junit test of an apache kafka producer (works with Kafka 0.11.0.2) (see also https://github.com/asmaier/mini-kafka)
View KafkaProducerIT.java
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.util.Arrays;
import java.util.Iterator;
import java.util.Properties;
import org.I0Itec.zkclient.ZkClient;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
You can’t perform that action at this time.