Skip to content

Instantly share code, notes, and snippets.

@ScotterC
ScotterC / chunkify.py
Created February 28, 2023 16:25
Chunk up text using lanchain's text splitters
#!/usr/bin/env python3
# Note: if spacy isn't working you may need to download english model python -m spacy download en
import argparse
import json
from langchain.text_splitter import NLTKTextSplitter, CharacterTextSplitter, RecursiveCharacterTextSplitter, SpacyTextSplitter, TokenTextSplitter
def chunkify(text, chunk_size, chunk_overlap, method):
if method == 'nltk':
text_splitter = NLTKTextSplitter.from_tiktoken_encoder(
@ScotterC
ScotterC / blackhole.rb
Created March 23, 2017 17:03
blackhole.rb
#! /usr/bin/ruby
require 'fileutils'
path = "/etc/hosts"
sites = %w(
news.ycombinator.com
reddit.com
www.reddit.com
@ScotterC
ScotterC / gist:b738721015de6c56573f
Created November 4, 2015 20:58
Basic vagrant error output
~/vagrant-setup (master): vagrant plugin install vagrant-hostsupdater
Installing the 'vagrant-hostsupdater' plugin. This can take a few minutes...
Bundler, the underlying system Vagrant uses to install plugins,
reported an error. The error is shown below. These errors are usually
caused by misconfigured plugin installations or transient network
issues. The error from Bundler is:
An error occurred while installing json (1.8.1), and Bundler cannot continue.
Make sure that `gem install json -v '1.8.1'` succeeds before bundling.
@ScotterC
ScotterC / gist:4255cd9755ea7c04f160
Created September 6, 2015 17:53
Basic Mac Ruby Setup
@ScotterC
ScotterC / pre-push.sh
Created September 2, 2015 16:46
Git Hook PrePush protection for force pushing
#!/bin/sh
# Called by "git push" after it has checked the remote status,
# but before anything has been pushed.
#
# If this script exits with a non-zero status nothing will be pushed.
#
# Steps to install, from the root directory of your repo...
# 1. git config --global init.templatedir '~/.git-templates'
# 2. mkdir -p ~/.git-templates/hooks
#!/usr/bin/env bash
for branch in `git checkout --quiet master && git branch -r --merged | grep origin/ | sed 's/origin\///' | sed '1d;$d' | grep --invert-match master`; do echo -e `git push origin --delete $branch` \\t$branch; done
#!/bin/sh
# Called by "git push" after it has checked the remote status,
# but before anything has been pushed.
#
# If this script exits with a non-zero status nothing will be pushed.
#
# Steps to install, from the root directory of your repo...
# 1. git config --global init.templatedir '~/.git-templates'
# 2. mkdir -p ~/.git-templates/hooks
@ScotterC
ScotterC / bas
Created January 14, 2015 22:03
Git branch in command line
parse_git_branch() {
git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'
}
PS1="\[\033[0m\]\w\[\033[34m\] \$(parse_git_branch)\[\033[01;36m\]:\[\033[00m\] "
@ScotterC
ScotterC / gist:8826026
Created February 5, 2014 15:27
Find the largest prime factor of a number
def largest_prime_factor(number)
factors = factors_of(number)
factors.select do |f|
is_prime?(f)
end.max
end
def factors_of(number)
@ScotterC
ScotterC / gist:7604440
Last active December 29, 2015 02:59
Unicorn initialization script
#!/bin/sh
set -e
# Example init script, this can be used with nginx, too,
# since nginx and unicorn accept the same signals
# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/x/my_app/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="/usr/bin/unicorn -D -c $APP_ROOT/config/unicorn.rb"