Skip to content

Instantly share code, notes, and snippets.

View roger-link's full-sized avatar

roger link roger-link

View GitHub Profile
@roger-link
roger-link / pycallgraph_middleware.py
Created March 20, 2014 19:48
Add pycallgraph to django middleware
import time
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput
class CallgraphMiddleware(object):
def process_view(self, request, callback, callback_args, callback_kwargs):
if settings.DEBUG and 'graph' in request.GET:
pycallgraph = PyCallGraph(output=GraphvizOutput(output_file='callgraph-' + str(time.time()) + '.png'))
import pyocr
import pyocr.builders
import glob
import Image
tools = pyocr.get_available_tools()
tool = tools[0]
langs = tool.get_available_languages()
lang = langs[0]
IMAGE_PATH = '/home/ubuntu/images/*'
@roger-link
roger-link / Beam-Pipeline.java
Last active June 15, 2020 17:34
Beam-Pipeline.java
package com.redacted.redacted.processing.beam.errorrate;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.kafka.KafkaRecord;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.extensions.jackson.ParseJsons;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.SerializableFunction;
/usr/local/Cellar/apache-flink/1.8.0/bin/flink run ./processing/beam/errorrate/target/redacted-errorrate-1.0-SNAPSHOT.jar --runner=FlinkRunner --awsRegion=us-west-2 --kafkaTopic=redacted-Topic --kafkaBrokers="b-2.redacted-processing-rog.nrpwiy.c2.kafka.us-west-2.amazonaws.com:9092,b-1.redacted-processing-rog.nrpwiy.c2.kafka.us-west-2.amazonaws.com:9092,b-4.redacted-processing-rog.nrpwiy.c2.kafka.us-west-2.amazonaws.com:9092" --kinesisStream=redacted-processing
Starting execution of program
java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
org/apache/beam/runners/flink/FlinkStreamingTransformTranslators$CombinePerKeyTranslator.translateNode(Lorg/apache/beam/sdk/transforms/PTransform;Lorg/apache/beam/runners/flink/FlinkStreamingTranslationContext;)V @467: invokespecial
Reason:
Type 'org/apache/flink/streaming/api/transformations/TwoInputTransformation' (current frame, stack[4]) is not assignable to 'org/apache/flink/api/dag/Transformation'
Current Frame:
bci: @467
@roger-link
roger-link / pom.xml
Created June 15, 2020 17:35
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.redacted.redacted.processing</groupId>
<artifactId>processing-parent</artifactId>
<version>${revision}</version>
<relativePath>../pom.xml</relativePath>
</parent>