Skip to content

Instantly share code, notes, and snippets.

Avatar

Herminio Vazquez canimus

View GitHub Profile
@canimus
canimus / pyspark_timestamp_cast.py
Last active May 9, 2021
Pyspark Cast 100 columns
View pyspark_timestamp_cast.py
from functools import reduce, methodcaller
import pyspark.sql.functions as F
_ts = lambda dataFrame, col: methodcaller('withColumn', f'{col}TimeStamp', F.to_timestamp(F.col(col)/1000))(dataFrame)
reduce(lambda a,b: _ts(a,b), ['start','stop'], df).select('startTimeStamp', 'stopTimeStamp').show(truncate=False)
@canimus
canimus / AWS Swarm cluster.md
Created Mar 29, 2021 — forked from ghoranyi/AWS Swarm cluster.md
Create a Docker 1.12 Swarm cluster on AWS
View AWS Swarm cluster.md

This gist will drive you through creating a Docker 1.12 Swarm cluster (with Swarm mode) on AWS infrastructure.

Prerequisites

You need a few things already prepared in order to get started. You need at least Docker 1.12 set up. I was using the stable version of Docker for mac for preparing this guide.

$ docker --version
Docker version 1.12.0, build 8eab29e

You also need Docker machine installed.

@canimus
canimus / parse.py
Created Mar 16, 2021
IBM Web Server Log Parse
View parse.py
from hashlib import md5 as xx
from collections import namedtuple
import os
import re
# Environment parametes
COLUMN_SEPARATOR = chr(os.getenv('SEPARATOR', 449))
FILE_NAME = os.getenv('FILE', 'hashed.csv')
# Regular expression to capture JSESSIONID
@canimus
canimus / pie.py
Created Feb 15, 2021
Pie Chart Matplotlib
View pie.py
import matplotlib.pyplot as plt
# create data
names='Failed: 20', 'Passed: 80',
size=[11.8, 98.2]
# Create a circle for the center of the plot
my_circle=plt.Circle( (0,0), 0.7, color='white')
plt.pie(size, labels=names, colors=['#f56262', '#23d993'])
View git-author.sh
find . -type f -exec git log --reverse --format="{} %cn" -1 {} \; | cut -d" " -f2- | sort | uniq -c | sort -n -k1
View wait_seconds.sh
rg ".*I wait for (\d+) secon.*$" -N -r '$1' | rg "\d+" | awk -F: '{s[$1]+=$2}END{for (i in s) print i,s[i]}' | sort -n -k2 | awk '{s+=$2}END{print s}'
@canimus
canimus / tag_count.js
Created Feb 1, 2021
Count tags in web page with webdriverio
View tag_count.js
browser.execute(
() => {
return Array.from(document.querySelectorAll("*")).map(e=>e.tagName.toLowerCase())
})
.reduce((a,b) => {a[b]=(a[b] || 0)+1; return a}, {})
@canimus
canimus / parser.py
Created Jan 7, 2021
termgraph test cases
View parser.py
#!/home/herminio/py385/bin/python
import glob
import json
import chalk
import asyncio
import aiofiles
import tabulate
import pandas as pd
import itertools as it
@canimus
canimus / pyspark-json.py
Created Jan 4, 2021
PySpark Union JSON
View pyspark-json.py
df = reduce(lambda a,b: a.union(b), [spark.read.json(f'{x}/*result.json') for x in range(195, 202)])
@canimus
canimus / testcase.ipynb
Last active Oct 28, 2020
Test Case - Rules
View testcase.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.