Skip to content

Instantly share code, notes, and snippets.

@joocer
joocer / setup_env.bat
Last active July 11, 2020 15:13
A simple script to create and activate a virtual env, install requirements and running a main.py script
@echo off
set "VIRTUAL_ENV=%CD%\temp"
set "PATH=%VIRTUAL_ENV%\Scripts;%PATH%"
set "PROMPT=(VIRTUAL) $P$G%"
python -m venv temp
pip install -r requirements.txt
python main.py
@joocer
joocer / gist:b64b3c82e805ec50f3d2d073b1bed4b5
Last active August 11, 2020 08:56
qualys to off sample (note to jsonl)
import pandas as pd
import xmltodict
import json
import off
FINDINGS_FILE = r'qualys\qualys_hosts.xml'
SEVERITIES = ['Low', 'Low', 'Medium', 'Medium', 'High']
CONFIDENCES = { 'Potential': 'Medium', 'Confirmed': 'High' }
OFF_FILE = r'offl.json'
@joocer
joocer / read_large_file.py
Last active October 24, 2020 09:48
read an arbitrary long file in python, line by line
def read_file(filename, chunk_size=1024*1024, delimiter='\n'):
with open(filename, 'r', encoding="utf8") as f:
carry_forward = ''
chunk = 'INITIALIZED'
while len(chunk) > 0:
chunk = f.read(chunk_size)
augmented_chunk = carry_forward + chunk
lines = augmented_chunk.split(delimiter)
carry_forward = lines.pop()
yield from lines
@joocer
joocer / gcs_handlers.py
Last active October 22, 2020 14:18
simplify the handling of GCS blobs in Python
import datetime, re
from google.cloud import storage
from functools import lru_cache
try:
import ujson as json
except ImportError:
import json
def select_dictionary_values(dictionary, values):
@joocer
joocer / gcs_threaded_reader.py
Created October 24, 2020 09:46
Reads a set of blobs parallel to speed up reading. Blobs are read line by line, the usecase this was written for was jsonl files, but any record-per-line format where you don't care about the order of the lines should not have issues.
"""
Threaded Google Cloud Storage Blob reader.
Reads a set of blobs parallel to speed up reading. Blobs are
read line by line, the usecase this was written for was jsonl
files, but any record-per-line format where you don't care
about the order of the lines should not have issues.
Limited performance testing reading a set of eight blobs,
4x 19Mb, 4x 5.4Mb in four threads ran in 20.1% of the time.
@joocer
joocer / bplustree.py
Last active April 25, 2021 19:34 — forked from savarin/bplustree.py
Python implementation of a B+ tree
"""
B+Tree Code Adapted From:
https://gist.github.com/savarin/69acd246302567395f65ad6b97ee503d
No explicit license when accessed on 2nd March 2020.
Other code:
(C) 2021 Justin Joyce.
@joocer
joocer / histogram
Created March 8, 2021 20:37
single char high histogram
bar_chars = (' ', '▁', '▂', '▃', '▄', '▅', '▆', '▇', '█')
def _draw_histogram(bins):
mx = max([v for k,v in bins.items()])
bar_height = (mx / 7)
if mx == 0:
return ' ' * len(bins)
histogram = ''
for k,v in bins.items():
@joocer
joocer / bollinger.py
Created June 15, 2021 20:35
python bollinger bands without pandas
from matplotlib import pyplot as plt # type:ignore
from mabel.data.formats import dictset
import numpy as np # type:ignore
def bollinger_bands(series, length=20, sigmas=2):
index = 0
window = series[0:length]
num_vals = len(window)
@joocer
joocer / boolparser.py
Last active August 2, 2021 13:40 — forked from leehsueh/boolparser.py
Python Boolean Expression Parser/Evaluator
"""
Grammar:
========
Expression --> AndTerm { OR AndTerm}+
AndTerm --> Condition { AND Condition}+
Condition --> Terminal (>,<,>=,<=,==) Terminal | (Expression)
Terminal --> Number or String or Variable
Usage:
======
@joocer
joocer / color-gradient.js
Last active January 26, 2022 17:43 — forked from gskema/color-gradient.js
Generate gradient color from an arbitrary number of colors
function colorGradient(colors, fadeFraction) {
if (fadeFraction >= 1) {
return colors[colors.length - 1]
} else if (fadeFraction <= 0) {
return colors[0]
}
let fade = fadeFraction * (colors.length - 1);
let interval = Math.trunc(fade);
fade = fade - interval