Skip to content

Instantly share code, notes, and snippets.

@dbernheisel
dbernheisel / batch-folders.sh
Last active August 29, 2015 14:15
Bash script to move files into batched folders for easier processing. For example if there were 50k files and you needed them in 200 file chunks, you would call this to move the 50k into folders of 200 files each.
#!/bin/bash
# Usage: batch-folders.sh /Where/To/Look 5000
# This will create 5000-large batch folders within /Where/To/Look
FolderPath="$1"
HowManyToBatch="$2"
BatchFolder="Batch_"
# check if number of arguments is right
if [ ! $# == 2 ]; then
@dbernheisel
dbernheisel / country-math.py
Created February 21, 2015 04:47
This is a Python script to do some country code math when I was working with book metadata. It seemed that Amazon didn't recognize the newer country codes so they have an exception case. This was helpful in case someone expressed their sales rights as "Nowhere but US", but I needed to express it as "US only" (invert the expression).
#!/bin/python
import argparse
# Usage: country-math.py -a "US CA SS EY"
# Usage: country-math.py "US CA SS EY" -o "US CA SS EY GB"
parser = argparse.ArgumentParser(description="Intersect two ISO country code lists. Remove countries found in REMOVE_COUNTRIES from ORIGINAL_COUNTRIES or the world.")
parser.add_argument("countries", help="List of undelimited country codes surrounded by doublequotes that need to be removed", metavar="REMOVE_COUNTRIES")
parser.add_argument("-a", "--amazon", help="Remove country codes that are not supported by Amazon.", action='store_true')
parser.add_argument("-o", "--originallist", dest="originalcountries", help="List of undelimited country codes that represent original list. This list needs to be surrounded by doublequotes. If this flag is not provided, then program will default to all country codes", metavar="ORIGINAL_COUNTRIES")
parser.set_defaults(amazon=False)
@dbernheisel
dbernheisel / flatten-folders.sh
Created February 21, 2015 04:51
Flatten a folder hierarchy.
#!/bin/bash
# Usage: execute flatten-folders.sh while in the directory you want to flatten.
# This will move all content in subdirectories down to present directory
CURRENTDIR="$( cd "$( dirname "$0" )" && pwd)"
for f in "$CURRENTDIR"
do
parentdir="$(dirname "$f")"
find "$f/" -iname '*.*' -exec mv -v \{\} "$parentdir/" \;
rmdir -v "$f"
@dbernheisel
dbernheisel / isbn-folders.sh
Created February 21, 2015 04:52
Separate files into new folders by ISBN13 (no dashes)
#/bin/bash
# Separate ISBNs to folders named by ISBN
for f in 978*.*; do
isbn=${f:0:13}
if [ ! -d $isbn ]; then
echo Creating folder for $isbn
mkdir $isbn
fi
mv -v $f $isbn
@dbernheisel
dbernheisel / limit-onix3-sales-rights.py
Created February 21, 2015 05:07
Some ONIX 3.0 files have complete sales rights expressed, but a contract may need to override what's in the ONIX. For example, the publisher may have world rights, but the given contract limits their ability to sell in the US.
#!/usr/local/bin/python3
from lxml import etree
import sys
import os
import logging
import shutil
import argparse
# This is designed to take an ONIX 3.0 file, analyze the Sales Rights, enforce the contract limitations, and then give pass it on for ingestion.
# USAGE: python3 limit-sales-rights.py -f $FILE -c US CA
@dbernheisel
dbernheisel / extract-onix-records.py
Created February 21, 2015 05:14
Sometimes I was given 10k+ ONIX files, but I needed to correct a couple of records (by ISBN13). This script helps extract only those records I need so I can correct them and re-ingest only those records.
#!/usr/local/bin/python3
from lxml import etree
import sys
import os
import re
import logging
import shutil
import argparse
from itertools import islice
import chardet
@dbernheisel
dbernheisel / backup-rsync-nas-external.sh
Last active August 29, 2015 14:15
This script will sync an external drive with a NAS.
#!/bin/bash
remote="/Volumes/Volume_1"
local="/Volumes/Storage"
LOCK=~/Desktop/Synching
logging=~/backup-rsync-log.txt
sleeptime=20
maxthreads=1
set -e
@dbernheisel
dbernheisel / batch-folders.rb
Last active August 31, 2015 23:31
Batch files into folders according to a batch size provided by the user.
#!/usr/bin/env ruby
# This program accepts a file path and a number of your choice for how many
# files to split into folders.
#
# eg: If you have 1000 files and need 100 files per folder, then enter 100 as
# the number, and this program will create 10 folders and move 100 files into
# each.
require "Pathname"
require "FileUtils"
@dbernheisel
dbernheisel / input_statistics.rb
Last active August 31, 2015 19:13
Iron Yard Homework, Week 1
#!/usr/bin/env ruby
# Need the ability to determine if input text is really just a number. This
# will return a simple true/false
def is_number?(f)
begin
Float(f)
return true
rescue
return false
@dbernheisel
dbernheisel / rails_resources.md
Last active September 6, 2015 17:20 — forked from jookyboi/rails_resources.md
Rails-related Gems and guides to accelerate your web project.

Gems

  • Bundler - Bundler maintains a consistent environment for ruby applications. It tracks an application's code and the rubygems it needs to run, so that an application will always have the exact gems (and versions) that it needs to run.
  • rabl - General ruby templating with json, bson, xml, plist and msgpack support
  • Thin - Very fast and lightweight Ruby web server
  • Unicorn - Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels.
  • SimpleCov - SimpleCov is a code coverage analysis tool for Ruby 1.9.
  • Zeus - Zeus preloads your Rails app so that your normal development tasks such as console, server, generate, and specs/tests take less than one second.
  • [factory_girl](h