Skip to content

Instantly share code, notes, and snippets.

View geoHeil's full-sized avatar

geoHeil geoHeil

95 followers · 83 following
View GitHub Profile
@geoHeil
geoHeil / dbt_materialize_redpanda.yml
Last active August 8, 2022 15:12
dbt_materialize_redpanda.yml
version: '3.7'
services:
redpanda:
image: docker.vectorized.io/vectorized/redpanda:v22.1.7
container_name: redpanda
hostname: redpanda
command:
- redpanda start
- --overprovisioned
- --smp 1
@geoHeil
geoHeil / sensor.py
Created July 25, 2022 20:51
sensor not resetting example
def make_date_file_sensor_for_asset(j: JobDefinition, resource_defs_for_ssh):
@sensor(
job=j,
name=j.name + "_sensor",
default_status=DefaultSensorStatus.RUNNING,
)
def date_file_sensor(context):
with build_resources(resource_defs_for_ssh) as resources:
ssh = resources.ssh
sftp = ssh.open_sftp()
@geoHeil
geoHeil / docker-compose.yaml
Last active July 14, 2022 07:32
readpanda materialize automated CI setup with user auth
version: '3.7'
# based on
# 1. https://redpandacommunity.slack.com/archives/C01AJDUT88N/p1657529113002319?thread_ts=1657527079.070669&cid=C01AJDUT88N
# 2. https://redpandacommunity.slack.com/archives/C01AJDUT88N/p1652213796745369?thread_ts=1652195821.668589&cid=C01AJDUT88N
services:
redpanda:
image: docker.vectorized.io/vectorized/redpanda:v22.1.4
container_name: redpanda
hostname: redpanda
command:
@geoHeil
geoHeil / stacktrace-geomesa-geospark.scala
Last active February 10, 2022 16:18
geomesa-geospark scala
```
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function geospark_rs_array replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function geospark_rs_normalize replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function geospark_st_union_aggr replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function geospark_st_envelope_aggr replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function geospark_st_intersection_aggr replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function st_boundary replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function st_coorddim replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionRegistry: The function st_dimension replaced a previously registered function.
22/02/10 17:07:54 WARN SimpleFunctionReg
@geoHeil
geoHeil / foo.py
Last active November 9, 2020 13:34
matrixprofile / stumpy / mulltiple time-series handling / pandas UDF per group (no aggregation to scalar
sure:
%pylab inline
import stumpy
import pandas as pd
import numpy as np
import random
random_seed = 47
@geoHeil
geoHeil / foo.py
Created October 15, 2020 13:20
df['device_id'] = df['device_id'].astype(str).apply(lambda x: x.zfill(2))
df = df.sort_values(['device_id'])
d_r = df.reset_index()
a4_dims = (11.7, 8.27)
fig, ax = pyplot.subplots(figsize=a4_dims)
ax = sns.lineplot(x='index', y='metrik_0', style='device_id', hue='device_id', data=d_r, ax=ax)
import matplotlib.dates as mdates
ax.axvspan(*mdates.datestr2num(['2020-01-04 05:00:00', '2020-01-05 16:00:00']), color='gray', alpha=0.2)
ax.axvspan(*mdates.datestr2num(['2020-01-04 15:00:00', '2020-01-05 00:00:00']), color='gray', alpha=0.2)
@geoHeil
geoHeil / rapids_15_3.8.txt
Created October 1, 2020 12:20
# packages in environment at ~/.conda/envs/rapidstest:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge /conda-forge
_openmp_mutex 4.5 1_gnu /conda-forge
abseil-cpp 20200225.2 he1b5a44_2 /conda-forge
aiohttp 3.6.2 py38h516909a_0 /conda-forge
appdirs 1.4.3 py_1 /conda-forge
argon2-cffi 20.1.0 py38h1e0a361_1 /conda-forge
arrow-cpp 0.17.1 py38h1234567_11_cpu /conda-forge
@geoHeil
geoHeil / gist:3b71999901af4ca44c14d6d6dc5eac8d
Created October 1, 2020 11:26
conda_list.txt
```
(rapidstest) > $ conda list [±4-econ-graph ●●]
# packages in environment at ~/.conda/envs/rapidstest:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge /conda-forge
_openmp_mutex 4.5 1_gnu /conda-forge
aiohttp 3.6.2 py37h516909a_0 /conda-forge
appdirs 1.4.3 py_1 /conda-forge
@geoHeil
geoHeil / Foo.scala
Last active July 23, 2020 08:44
failing
// setup in bash
wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka_2.11/1.11.1/flink-connector-kafka_2.11-1.11.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka-base_2.11/1.11.1/flink-connector-kafka-base_2.11-1.11.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/flink-avro-confluent-registry/1.11.1/flink-avro-confluent-registry-1.11.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/flink-avro/1.11.1/flink-avro-1.11.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/force-shading/1.11.1/force-shading-1.11.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2.jar -P lib/
wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.11.1/jackson-core-2.11.1.jar -P lib/
wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-d
@geoHeil
geoHeil / Tweet.scala
Created June 29, 2020 12:16
new specific type class
/** MACHINE-GENERATED FROM AVRO SCHEMA. DO NOT EDIT DIRECTLY */
package com.github.geoheil.streamingreference
import scala.annotation.switch
final case class Tweet(var tweet_id: Option[String], var text: Option[String], var source: Option[String], var geo: Option[String], var place: Option[String], var lang: Option[String], var created_at: Option[String], var timestamp_ms: Option[String], var coordinates: Option[String], var user_id: Option[Long], var user_name: Option[String], var screen_name: Option[String], var user_created_at: Option[String], var followers_count: Option[Long], var friends_count: Option[Long], var user_lang: Option[String], var user_location: Option[String], var hashtags: Option[Seq[String]]) extends org.apache.avro.specific.SpecificRecordBase {
def this() = this(None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None)
def get(field$: Int): AnyRef = {
(field$: @switch) match {
case 0 => {