Skip to content

Instantly share code, notes, and snippets.

@echeran
echeran / pom.xml
Last active Dec 27, 2015
mvn-lein-test - pom.xml
View pom.xml
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1</version>
<repositories>
<repository>
<id>clojars</id>
<name>Clojars</name>
@echeran
echeran / project.clj
Last active Dec 27, 2015
mvn-lein-test - project.clj
View project.clj
(defproject mvn-lein-test "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.7.0"]
[medley "0.7.0"]
[leiningen "2.5.3"]]
:main mvnleintest.core
:aot :all
@echeran
echeran / stacktrace.txt
Created Dec 27, 2015
stacktrace from running java -jar on Maven-built uberjar
View stacktrace.txt
2015-12-27 00:22:56,112 DEBUG com.google.cloud.dataflow.sdk.options.PipelineOptionsFactory - Provided Arguments: {}
Exception in thread "main" java.lang.IllegalArgumentException: interface TestWordCountOptions is not visible from class loader
at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:581)
at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:557)
at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:230)
at java.lang.reflect.WeakCache.get(WeakCache.java:127)
at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:419)
at java.lang.reflect.Proxy.getProxyClass(Proxy.java:371)
at com.google.cloud.dataflow.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:587)
at com.google.cloud.dataflow.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1242)
@echeran
echeran / async_util.clj
Last active Jul 2, 2018
Utility functions for core.async 0.1.346.0-17112a-alpha to create a linear pipeline of go-routines in a Storm-esque way.
View async_util.clj
(ns your.namespace.async-util
(:require
[clojure.core.async :as a :refer [>! <! >!! <!! go chan buffer
close! thread alts! alts!! timeout]]
[clojure.core.match :as match :refer [match]]))
;;
;; core.async util fns
;;
View LangUtil.scala
/*
Copyright 2018 Google LLC.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
View SeqUtil.scala
/**
Copyright 2018 Google LLC.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
@echeran
echeran / transpiler-veterans-quesitons.md
Last active Apr 11, 2020
Questions for Transpiler Veterans
View transpiler-veterans-quesitons.md
  1. For your transpiler, how did you decide which source language and target language(s) to focus on?
  2. Were there any existing transpiler solutions that solved part of all of the problem surface you were addressing? If so, which factors led to you choosing to create your own transpiler instead of reusing existing solutions?
  3. Who were your stakeholders when starting the project?
  4. What were obstacles that you faced when starting the project?
  5. What were obstacles that you faced during the implementation of the project?
  6. What factors led to the success of the project?
  7. How much time has been spent on the project since the beginning?
  8. Are there any remaining features that you would still like to fix in the transpiling project?
  9. What are the technical considerations / challenges for anyone considering a transpiling approach? Are there unwritten pearls of wisdom about transpilers / compilers?
  10. What is your opinion on why there are few (no?) “universal” general purpose one-to-all transpilers? Ex:
@echeran
echeran / clojure-multithreaded-spark.clj
Created Mar 13, 2021
Mulit-threaded Spark jobs via Clojure with Claypoole (custom pmap) + mapPartitions glue code
View clojure-multithreaded-spark.clj
(require '[claypoole.core :as tpool])
;;
;; helper fns
;;
(defn your-multi-threaded-spark-fn
"Create a parallelized version of your-fn using a fixed-size threadpool.
This fn is created to fit the signature of .mapPartitions in Spark."
[extra-args-to-set-up-your-fn]
(fn
@echeran
echeran / clojure-serialized-fns-flambo-spark.clj
Last active Mar 13, 2021
Reusing your fns as Clojure serialized fns to use in Flambo (Spark wrapper for Clojure)
View clojure-serialized-fns-flambo-spark.clj
(require '[clj-time.core :as t])
(require '[flambo.api :as f])
(defn date-filter-fn
"Given a LocalDate, returns a serializable fn that takes a DateTime
object and returns true if the DateTime fields match those of the
provided LocalDate, else returns false."
[local-date]
;; returning a serializable fn to be used with Flambo/Spark's filter operation
(f/fn