Skip to content

Instantly share code, notes, and snippets.

View machawk1's full-sized avatar

Mat Kelly machawk1

View GitHub Profile
@edsu
edsu / wget-warc.txt
Created December 2, 2011 10:34
example of using wget's warc functionality
.-(ed@curry 05:24:32) ~
`-->wget --recursive --warc-file=c4lj.warc.gz http://journal.code4lib.org
FINISHED --2011-12-02 05:17:11--
Total wall clock time: 19m 24s
Downloaded: 1524 files, 99M in 4m 17s (395 KB/s)
.-(ed@curry 05:45:22) ~
`-->ls -lh c4lj.warc.gz.warc.gz
-rw-rw-r-- 1 ed ed 85M 2011-12-02 05:17 c4lj.warc.gz.warc.gz
@willurd
willurd / web-servers.md
Last active July 25, 2024 21:14
Big list of http static server one-liners

Each of these commands will run an ad hoc http static server in your current (or specified) directory, available at http://localhost:8000. Use this power wisely.

Discussion on reddit.

Python 2.x

$ python -m SimpleHTTPServer 8000
@Asparagirl
Asparagirl / gist:6206247
Last active July 19, 2024 15:13
Have a WARC that you would like to upload to the Internet Archive so that it can eventually be included in their Wayback Machine? Here's how to upload it from the command line.

Do you have a WARC file of a website all downloaded and ready to be added to the Internet Archive? Great! You can do that with the Internet Archive's web-based uploader, but it's not ideal and it can't handle really big uploads. Here's how you can upload your WARC files to the IA from the command line, and without worrying about a size restriction.

First, you need to get your Access Key and Secret Key from the Internet Archive for the S3-like API. Here's where you can get that for your IA account: http://archive.org/account/s3.php Don't share those with other people!

Here's their documentation file about how to use it, if you need some extra help: http://archive.org/help/abouts3.txt

Next, you should copy the following files to a text file and edit them as needed:

export IA_S3_ACCESS_KEY="YOUR-ACCESS-KEY-FROM-THE-IA-GOES-HERE"
@Zearin
Zearin / python_decorator_guide.md
Last active July 24, 2024 03:06
The best explanation of Python decorators I’ve ever seen. (An archived answer from StackOverflow.)

NOTE: This is a question I found on StackOverflow which I’ve archived here, because the answer is so effing phenomenal.


Q: How can I make a chain of function decorators in Python?


If you are not into long explanations, see [Paolo Bergantino’s answer][2].

@anjackson
anjackson / sha1b32.sh
Created November 18, 2015 22:03
Calculate the Base32-encoded SHA-1 digest of a file at the command line.
openssl dgst -sha1 -binary $1 | python -c "import base64,sys; print base64.b32encode(sys.stdin.read())"

InterPlanetary Version Control (call it IPVC?)

IPLD-based Version History

This is just a sketch of a possibility. If we just want a git-style toolchain with git version graph, it might be better to just put an ipfs storage adapter behind go-git -- basically putting IPFS unixfs trees where git usually creates git tree objects. In that case you would have regular git commit objects, not IPLD objects. That would be less reusable beyond the git context but it would fit better with existing git-based tooling.

Keep in mind: it will be really useful to be able to use IPFS files api to manipulate these trees, allowing you to do things like modify part of a large dataset and commit the changes without ever pulling the whole dataset -- you can just pull the parts that you're changing.

Features

@atomotic
atomotic / readme.md
Last active October 19, 2017 04:29
poor man's WARC viewer

install solr and create a core (books)

brew install solr
solr start
solr create -c books -d /usr/local/Cellar/solr/7.2.1/example/files/conf

index a pdf

post -c books /tmp/gabriella-giannachi-archive-everything-mapping-the-everyday.pdf
@N0taN3rd
N0taN3rd / README.md
Last active April 23, 2018 20:44
Selectively Compile Latex Documents

The subfile package is a much more efficient way to bring in the sub parts of a large latex document.

Each part of the complete document that is split up into its own text file brought is added to the document from the main tex file via

\subfile{file} 

Each subfile has the following structure

\documentclass[main.tex]{subfiles}
@N0taN3rd
N0taN3rd / check_margins.sh
Created April 27, 2018 08:16
Appease the margin police at all costs
#!/usr/bin/env bash
####
# Original version of this file was created by Dr. Chuck Cartledge
# as a make file and can be found in his WS-DL blog post:
# https://ws-dl.blogspot.com/2014/07/2014-0-7-2-ode-to-margin-police-or-how.html
####
#### Set these variables to match your setup
CHECK="msThesis"