xtrmstep

DO
$do$
declare
	r record;
	query_cmd text;
begin
	for r in select table_name from information_schema.tables where table_schema = 'public' and table_name like 'prefix%'
	loop
		query_cmd := format('delete from %s where CONDITION', r.table_name);
-- 		raise notice '%', query_cmd;

xtrmstep

// usage:
// 'http://www.website.com/'.urlQueryParameter('id', 2) => http://www.website.com/?id=2
// 'http://www.website.com/?type=1'.urlQueryParameter('id', 2) => http://www.website.com/?type=1&id=2

String.prototype.isString = true;
String.prototype.urlQueryParameter = function(key, value) {
    var uri = this;
    if (uri.isString) {
        var regEx = new RegExp("([?|&])" + key + "=.*?(&|$)", "i");
        var separator = uri.indexOf('?') !== -1 ? "&" : "?";

xtrmstep

function odump(object, depth, max) {
    depth = depth || 0;
    max = max || 2;
    if (depth > max) return false;
    var indent = "";
    for (var i = 0; i < depth; i++) indent += " ";
    var output = "";
    for (var key in object) {
        output += "n" + indent + key + ": ";
        switch (typeof object[key]) {

xtrmstep

files = [
  "file://path"
]
df = spark.read.json(files)
catalyst_plan = df._jdf.queryExecution().logical()
df_size_read = spark._jsparkSession.sessionState().executePlan(catalyst_plan).optimizedPlan().stats().sizeInBytes()

xtrmstep

# source: https://stackoverflow.com/a/50156142/2833774

def flatten(schema, prefix=None):
    fields = []
    for field in schema.fields:
        name = prefix + '.' + field.name if prefix else field.name
        alias_name = name.replace(".", "__")
        dtype = field.dataType
        if isinstance(dtype, pst.ArrayType):
            dtype = dtype.elementType

xtrmstep

EMOJI CHEAT SHEET

Emoji emoticons listed on this page are supported on Campfire, GitHub, Basecamp, Redbooth, Trac, Flowdock, Sprint.ly, Kandan, Textbox.io, Kippt, Redmine, JabbR, Trello, Hall, plug.dj, Qiita, Zendesk, Ruby China, Grove, Idobata, NodeBB Forums, Slack, Streamup, OrganisedMinds, Hackpad, Cryptbin, Kato, Reportedly, Cheerful Ghost, IRCCloud, Dashcube, MyVideoGameList, Subrosa, Sococo, Quip, And Bang, Bonusly, Discourse, Ello, and Twemoji Awesome. However some of the emoji codes are not super easy to remember, so here is a little cheat sheet. ✈ Got flash enabled? Click the emoji code and it will be copied to your clipboard.

People

😄

xtrmstep

SENDING METRICS TO STATSD WITH A SHELL ONE-LINER

NOTE: From here

Etsy’s great post about tracking every release and has some good tips about tracking releases with statsd and graphite (including some essential graphite config tweaks). I was wondering how to do this from within a shell script, and I had to dig through lots of StatsD code and examples to find this snippet. I forget where I eventually found it, and thought it’d make it easier to find.

Deploy scripts are just one place where a concise and safe way to record a metric/event in important. From Etsy’s blog, using vertical lines to represent distinct events (code deployments) to give more context to the login trends:

Sending a metric from the command line, with netcat or curl, is just one bit of ‘glue’ that is essential for pulling t

xtrmstep

Custom Logs in GCP Logging

Logs can be sent to GCP logging very simple.

Basic Usage

You need to install the package google-cloud-logging.

Then the code will looks somehting like this:

xtrmstep

I accumulate here the list of articles which have interesting information about Apache Airflow. Subjects of the articles covers topics from installation and management to implementation of DAGs and best practices.

Best Practices

• Airflow: Lesser Known Tips, Tricks, and Best Practises

Comparison

• Airbnb’s Airflow Versus Spotify’s Luigi
• Airflow vs. Luigi vs. Argo vs. MLFlow vs. KubeFlow

xtrmstep

Here I gathered answers to small pitfalls which from time to time appears because you’re not doing some actions often. Such actions include using of .dockerignore, mounting files, SSL certificates for web servers in containers, etc.

Building of Docker images

1. The file want be applied if it’s not located in the same directory where you run docker build command.
2. Building of an image will fail if you use specific file names which you are ignoring. For example, if you write in Dockerfile COPY filetoignore.txt . and you are ignoring such file(s) in .dockerignore, you’ll get an error
3. Paths inside Dockerfile are relative to the folder where you’re running command docker build and not to the one where your Dockerfile is located.
4. Relative paths should use only current or child folders, since the parent one (or through it) is out of Docker context during building.

Docker-compose