Skip to content

Instantly share code, notes, and snippets.

@gabefair
gabefair / atwarrior.md
Created April 27, 2023 21:03 — forked from catleeball/atwarrior.md
Tutorial on running an Archive Team Warrior container with Docker Compose

You can run an Archive Team container on your computer! It'll help crawl the web and putting stuff into archive.org, based on the current projects.

@gabefair
gabefair / git.migrate
Created May 2, 2020 00:49 — forked from niksumeiko/git.migrate
Moving git repository and all its branches, tags to a new remote repository keeping commits history
#!/bin/bash
# Sometimes you need to move your existing git repository
# to a new remote repository (/new remote origin).
# Here are a simple and quick steps that does exactly this.
#
# Let's assume we call "old repo" the repository you wish
# to move, and "new repo" the one you wish to move to.
#
### Step 1. Make sure you have a local copy of all "old repo"
### branches and tags.
@gabefair
gabefair / progress.py
Created September 2, 2018 16:19
How I monitor progress in python
import sys, os, datetime
from time import time
def update_progress(total_imported, skipped_files, current_location):
text = str(total_imported) + "/" + str(total_imported+skipped_files) + " imported. Looking in folder: " + current_location
sys.stdout.write('\r' + text)
sys.stdout.flush()
def main():
start_time = datetime.datetime.fromtimestamp(time())
@gabefair
gabefair / split_json-lines_block.py
Created August 30, 2018 22:00
Splits a giant jsonL file into multipble jsonL files.
# Gabriel Fair
# Please suggest any improvements: http://keybase.io/gabefair
import mmap
import re
import argparse
import sys
import progressbar
from time import sleep
import os
from time import clock
@gabefair
gabefair / to_dataframe.py
Last active November 2, 2020 02:49
Mongodb collection to pandas dataframe
import pandas as pd
from pymongo import MongoClient
client = MongoClient("mongodb://localhost:27017")
db = client.database_Name
collection_conn = db['collection_name']
collection_cursor = collection_conn.find()
collection_pandas_df = pd.DataFrame(list(collection_cursor))
@gabefair
gabefair / cool_name.sh
Created August 18, 2018 17:30
Cool terminal text
#!/bin/bash
cat << EOF
███████╗███████╗████████╗██╗ ██╗
██╔════╝██╔════╝╚══██╔══╝██║ ██║ by Adrian Vollmer
███████╗█████╗ ██║ ███████║ seth@vollmer.syss.de
╚════██║██╔══╝ ██║ ██╔══██║ SySS GmbH, 2017
███████║███████╗ ██║ ██║ ██║ https://www.syss.de
╚══════╝╚══════╝ ╚═╝ ╚═╝ ╚═╝
EOF
@gabefair
gabefair / setup.py
Created August 9, 2018 19:54
merge rows of two csv files based on a column-key and place one of them ahead infront of the other file's rows after the merge
import sys, os
try:
import pandas as pd
except ImportError:
print("Please run `pip install pandas` from a command line")
exit()
def get_file_head(file_name):
@gabefair
gabefair / restore_packages.R
Created August 5, 2018 02:31 — forked from arne-cl/restore_packages.R
save/load/install a list of your currently installed R packages
# restore_packages.R
#
# installs each package from the stored list of packages
# source: http://hlplab.wordpress.com/2012/06/01/transferring-installed-packages-between-different-installations-of-r/
load("~/installed_packages.rda")
for (count in 1:length(installedpackages)) {
install.packages(installedpackages[count])
}
@gabefair
gabefair / v2_import.py
Last active September 25, 2018 05:34
Json to Mongo file importer
# pylint: disable=unsubscriptable-object
import sys, os, json, datetime, platform, argparse, traceback, mechanize
from time import time
try:
import mechanize
except ImportError:
print("Please run `pip install mechanize` from a command line")
exit()
@gabefair
gabefair / Tilestache-guide.md
Created December 15, 2017 14:48
Installing Tilestache to serve custom ArcGIS layers using Postgres 10 on Ubuntu 16 (Xenial Xerus)

Installing Tilestache to serve custom ArcGIS layers using Postgres 10 on Ubuntu 16 (Xenial Xerus)

Here I share my work notes to get custom layers that were once in ArcGIS to serve on a Ubuntu web host.

  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo apt install curl
  • curl -O -L https://bootstrap.pypa.io/get-pip.py
  • sudo python get-pip.py
  • sudo pip install -U pillow modestmaps simplejson uuid tilestache