Skip to content

Instantly share code, notes, and snippets.

View AndrewFarley's full-sized avatar

Farley Farley (yes, really) AndrewFarley

  • New Plymouth, New Zealand
View GitHub Profile
@AndrewFarley
AndrewFarley / generate-random-1gb-csv.py
Last active March 28, 2024 14:27
This simple Python file generates a random massive CSV file efficiently taking almost no RAM while doing so streaming data into your CSV file. I've pre-done the calculation of the rows/columns to the file-size, so you can easily add or remove zeroes from "rows" variable to increase or decrease the size of the file generated
import csv
import random
# 1000000 and 52 == roughly 1GB (WARNING TAKES a while, 30s+)
rows = 1000000
columns = 52
print_after_rows = 100000
def generate_random_row(col):
a = []
@AndrewFarley
AndrewFarley / RDS Replicate to External MySQL Server Howto
Created March 9, 2018 17:03
RDS Replicate to External MySQL Server Howto
### Overview ###
In order to replicate from an AWS RDS database to an external server, you need 3 components, and to keep two things in mind:
**Components**
* RDS Master DB - `rds-master`
* RDS Read-Only Slave - `rds-slave`
* External DB Server - `external-db`
**Two Things to Keep in Mind**
* This process is fairly brittle and not fully supported by AWS except for temporary data extraction.
@AndrewFarley
AndrewFarley / Gitlab Squasher
Created February 5, 2019 21:21
This simple Python code force squashes, if you call this via a webhook automatically via gitlab, then "magic" all merges will squash
import os
import flask
import requests
BASE_URL = 'https://YOURGITLABURL/api/v4'
TOKEN = 'YOURADMINAPITOKEN'
app = flask.Flask(__name__)
@AndrewFarley
AndrewFarley / backup-gitlab-to-s3.sh
Created September 5, 2018 11:57
A Shell Script to Backup Gitlab to S3 (with slack notifications)
#!/bin/bash
LOGFILE=/tmp/backup-gitlab-to-s3.log
GITLAB_BACKUP_FOLDER=/var/opt/gitlab/backups
S3_FILE_PREFIX=gitlab
S3_BUCKET_PATH=bucket-name-goes-here/gitlab-rotating-backups
SLACK_USERNAME="Backup Gitlab Daily - `hostname`"
SLACK_CHANNEL="#farley-testing"
# SLACK_CHANNEL="#monitoring"
SLACK_ICON="https://s3.amazonaws.com/kudelabs-archives/harddrive256.png"
@AndrewFarley
AndrewFarley / gitlab-ci.yml
Created January 4, 2023 22:11
Gitlab CI Template example
# Include our pre-engineered Gitlab-CI Templates
include:
# This is our global base and templates, always should be included together
- project: 'devops/gitlab-ci-templates'
ref: master
file: '/base.yml'
- project: 'devops/gitlab-ci-templates'
ref: master
file: '/global-templates.yml'
- project: 'devops/gitlab-ci-templates'
@AndrewFarley
AndrewFarley / gist:a17f9fdf9a22a93a67faa638532c72d6
Last active December 19, 2022 19:38
Setup AWS on Linux, Amazon Linux, Docker, FSX bash script helper. A scratch pad, I always keep randomly looking this stuff up
#!/bin/bash
# If not AWS Amazon Linux, install awscli...
# https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
# Make sure to install curl groff less, eg:
apt-get install curl groff less
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install
@AndrewFarley
AndrewFarley / annotation-to-add-into-helm-chart.sh
Created May 5, 2022 22:46
create annotations to import resource into helm chart installation
export KIND=PriorityClass
export NAME=workflow-controller
export RELEASE=argo-workflows
export NAMESPACE=argo
kubectl annotate $KIND $NAME meta.helm.sh/release-name=$RELEASE
kubectl label $KIND $NAME app.kubernetes.io/managed-by=Helm
kubectl annotate --overwrite $KIND $NAME meta.helm.sh/release-namespace=$NAMESPACE
@AndrewFarley
AndrewFarley / gist:3cdb7bb2f604f50827bff45910fcd345
Created April 8, 2022 00:38
Sample EKSCTL with YAML Aliases
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: master
region: us-east-1
version: "1.21"
iam:
withOIDC: true
@AndrewFarley
AndrewFarley / usb_hid_keys.h
Created January 13, 2022 10:25 — forked from MightyPork/usb_hid_keys.h
USB HID Keyboard scan codes
/**
* USB HID Keyboard scan codes as per USB spec 1.11
* plus some additional codes
*
* Created by MightyPork, 2016
* Public domain
*
* Adapted from:
* https://source.android.com/devices/input/keyboard-devices.html
*/