Skip to content

Instantly share code, notes, and snippets.

View jrask's full-sized avatar

Johan Rask jrask

  • RemotiveLabs
  • Lomma, Sweden
View GitHub Profile
@jrask
jrask / FlinkHdfs2hdfs.java
Last active November 17, 2019 08:31
Flink hdfs -> hdfs does to "commit" files, they stay as ".in-progress" files
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.enableCheckpointing(TimeUnit.MINUTES.toMillis(1), CheckpointingMode.EXACTLY_ONCE);
env.setStateBackend(new FsStateBackend("hdfs://<server><dir>/checkpoints",true));
env.getCheckpointConfig().setTolerableCheckpointFailureNumber(1);
StreamingFileSink<JsonObject> hdfsSink = StreamingFileSink
.<JsonObject>forRowFormat(new Path("hdfs://<server>/<dir>"), new SimpleStringEncoder<>("UTF-8"))
@jrask
jrask / TimerMax.java
Created December 21, 2018 09:03
Micrometer Timer.max rotation?
package se.flapsdown.micrometer.bugs;
import io.micrometer.core.instrument.Clock;
import io.micrometer.core.instrument.Meter;
import io.micrometer.core.instrument.Timer;
import io.micrometer.core.instrument.step.StepMeterRegistry;
import io.micrometer.core.instrument.step.StepRegistryConfig;
import java.time.Duration;
import java.util.Random;
@jrask
jrask / PrintHistograms.java
Created December 14, 2018 14:52
Micrometer Histogram does not match timer.count
package se.flapsdown.micrometer.histograms.bug;
import io.micrometer.core.instrument.Clock;
import io.micrometer.core.instrument.Meter;
import io.micrometer.core.instrument.Timer;
import io.micrometer.core.instrument.distribution.CountAtBucket;
import io.micrometer.core.instrument.distribution.HistogramSnapshot;
import io.micrometer.core.instrument.step.StepMeterRegistry;
import io.micrometer.core.instrument.step.StepRegistryConfig;
@jrask
jrask / lumbermill-grok-sample.groovy
Created June 17, 2016 04:55
Sample of running lumbermill locally and testing grok
package lumbermill
import lumbermill.api.Codecs
import rx.Observable
import static lumbermill.Core.*
// Simply replace with correct contents and play around
Observable.just(Codecs.TEXT_TO_JSON.from('Hello World'))
.flatMap (
@jrask
jrask / gist:49e12284460400b4104d
Last active August 29, 2015 14:22
Adding a Transformer to get kinesis to work with bunyan
var kinesis = require('kinesis')
var stream = require('stream')
var crypto = require('crypto')
// Make sure to set your AWS_PROFILE / AWS_DEFAULT_PROFILE unless you run IAM
var options = {
region: 'eu-west-1',
name: 'streamName'
}
@jrask
jrask / proxy_api.js
Created October 2, 2012 10:40
mongo-proxy-api
mongodb.request.get({
baseUri:'http://localhost:8000',
database:'demo',
collection:'demo',
method:'_find',
criteria:{name:'johan'},
sort:{age:mongodb.sort.ASC},
user:'demo',
passwd:'demo123',
error:function (data) {
@jrask
jrask / auth.js
Created October 1, 2012 20:56
auth
// Basic-auth validation and url (database) access validation
function authenticate(req, res, callback) {
// Extract authorization header and decode
var header = req.headers['authorization'] || '', // get the header
token = header.split(/\s+/).pop() || '', // and the encoded auth token
auth = new Buffer(token, 'base64').toString(), // convert from base64
parts = auth.split(/:/), // split on colon
user = parts[0],
passwd = parts[1];
@jrask
jrask / cors.js
Created October 1, 2012 20:39
CORS
function handleCors(req, res, callback) {
res.setHeader('Access-Control-Allow-Origin', '*')
res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE,OPTIONS');
res.setHeader('Access-Control-Allow-Headers', 'Authorization');
// CORS OPTIONS request, simply return 200
if (req.method == 'OPTIONS') {
res.statusCode = 200;
res.end();