Skip to content

Instantly share code, notes, and snippets.

Make me a new CentOS linode

Notes:

  • Instructions written for CentOS 6.3
  • Change, me, myhost, myip etc. to your username, hostname, ip address and so on.
  • Run all commands as root unless otherwise directed.
  • You might want to look at mounting /var and /home on separate partitions.
  • I've just allowed all members of the wheel group to operate as root. This is the height of laziness and highlights the fact that I'm just a developer that's stolen a sysadmin's
@iaingray
iaingray / gist:10389383
Created April 10, 2014 14:39
Hive base64 decoder UDF
#base64 decoder UDF
import base64
import sys
line = sys.stdin.readline();
while line:
padded = line[:-1]
print base64.b64decode(padded)
line = sys.stdin.readline();
sys.exit(0)
@iaingray
iaingray / gist:10389969
Created April 10, 2014 14:45
hive sha1 hasher UDF for emails etc
#sha1 hasher for emails etc
import hashlib
import sys
line = sys.stdin.readline();
while line:
m = hashlib.sha1()
m.update(line[:-1].lower())
print m.hexdigest()
line = sys.stdin.readline();
@iaingray
iaingray / gist:c93f2fec991b575ff075
Created June 1, 2014 11:41
ruby convert to clean strings to utf-8
# Converting ASCII-8BIT to UTF-8 based domain-specific guesses
if new_value.is_a? String
begin
# Try it as UTF-8 directly
cleaned = new_value.dup.force_encoding('UTF-8')
unless cleaned.valid_encoding?
# Some of it might be old Windows code page
cleaned = new_value.encode( 'UTF-8', 'Windows-1252' )
end
new_value = cleaned
@iaingray
iaingray / decode_cids.rb
Last active August 29, 2015 14:02
Quick and dirty script to decode url-encoded CIDs from redshift/pg database
#db settings contains params for @conn
require_relative 'db_settings.rb'
require 'pg'
require 'open-uri'
conn = PG::Connection.new(:dbname => @database, :host => @host , :port => @port, :user => @user, :password => @password )
decoded_cids = {}
sql = {
@iaingray
iaingray / remove_s3_underscores.rb
Created June 4, 2014 23:50
Script to remove leading underscores from s3 files, to allow them to be used in Hive
#!/bin/ruby
require 'fog'
require 'inifile'
INI_FILE = ENV['HOME']+'/.aws/config'
#details from command line
location = /^s3:\/\/(?<bucket>[a-zA-Z0-9-]+)\/(?<folder>[\w-]*)\/?$/.match(ARGV[0])
@iaingray
iaingray / convert_ip_to_bigint.sql
Created June 5, 2014 01:12
Convert ip address to bigint in redshift
( LEFT(ip_address, STRPOS(ip_address, '.')-1) * 16777216) + (LEFT(SUBSTRING(ip_address, LEN(LEFT(ip_address, STRPOS(ip_address, '.')+1)), LEN(ip_address) - LEN(LEFT(ip_address, STRPOS(ip_address, '.')-1)) - LEN(LEFT(REVERSE(ip_address), STRPOS(REVERSE(ip_address), '.')-1)) - 2), STRPOS( SUBSTRING(ip_address, LEN(LEFT(ip_address, STRPOS(ip_address, '.')+1)), LEN(ip_address) - LEN(LEFT(ip_address, STRPOS(ip_address, '.')-1)) - LEN(LEFT(REVERSE(ip_address), STRPOS(REVERSE(ip_address), '.')-1)) - 2), '.')-1) * 65536) + (RIGHT( SUBSTRING(ip_address, LEN(LEFT(ip_address, STRPOS(ip_address, '.')+1)), LEN(ip_address) - LEN(LEFT(ip_address, STRPOS(ip_address, '.')-1)) - LEN(LEFT(REVERSE(ip_address), STRPOS(REVERSE(ip_address), '.')-1)) - 2), LEN(SUBSTRING(ip_address, LEN(LEFT(ip_address, STRPOS(ip_address, '.')+1)), LEN(ip_address) - LEN(LEFT(ip_address, STRPOS(ip_address, '.')-1)) - LEN(LEFT(REVERSE(ip_address), STRPOS(REVERSE(ip_address), '.')-1)) - 2)) - STRPOS(SUBSTRING(ip_address, LEN(LEFT(ip_address, STRPOS(ip_a
/**
* Extends an object with addtional namespace nodes e.g. animals.birds.ducks.mallard
* @param base {Object} base object to be extended
* @param nodes {array} nodes to add
*/
extendObject : function(base, nodes){
if(typeof base !== 'object' || !(nodes instanceof Array)){
throw new Error ('Incorrect parameters passed to extendObject');
}
@iaingray
iaingray / hashpii.js
Last active August 29, 2015 14:06
hash PII
/**
* hashpii.js
*
* @description Function to standardise and hash PII strings
* @author Various (see below)
* @license MIT / GPL v2
*
* Modifications:
* - Various (see below)
*
@iaingray
iaingray / tunnel-ssh
Created November 11, 2014 00:38
tunneling postgress over SSH in Ruby
#tunnel pg through ssh
#pg test
require 'pg'
require 'net/ssh/gateway'
gate = Net::SSH::Gateway.new(
'BASTION_HOST_ADDR',
'BASTION_USER',
{port: 22, key_data: ['PRIVATE_KEY( as string)'], keys_only: true}