Skip to content

Instantly share code, notes, and snippets.

View hrchu's full-sized avatar
:octocat:
Have an Octotastic day!

petertc hrchu

:octocat:
Have an Octotastic day!
View GitHub Profile
@hrchu
hrchu / gist:c964f18d60b44f0bb9af
Last active August 29, 2015 14:08
Emulate large file in python, with file-like object interface and without memory overload
file_size=5*2**30 #5GB
class FakeFile:
def __init__(self):
self.remain_size=file_size
def read(self, size):
if self.remain_size == 0:
return None
elif self.remain_size >= size:
self.remain_size-=size

Example: You have a branch refactor that is quite different from master. You can't merge all of the commits, or even every hunk in any single commit or master will break, but you have made a lot of improvements there that you would like to bring over to master.

Note: This will not preserve the original change authors. Only use if necessary, or if you don't mind losing that information, or if you are only merging your own work.

On master:

> git co -b temp
@hrchu
hrchu / 0_reuse_code.js
Last active August 29, 2015 14:10
Here are some things you can do with Gists in GistBox.
// Use Gists to store code you would like to remember later on
console.log(window); // log the "window" object to the console
@hrchu
hrchu / python_resources.md
Created December 4, 2014 01:43 — forked from jookyboi/python_resources.md
Python-related modules and guides.

Packages

  • lxml - Pythonic binding for the C libraries libxml2 and libxslt.
  • boto - Python interface to Amazon Web Services
  • Django - Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
  • Fabric - Library and command-line tool for streamlining the use of SSH for application deployment or systems administration task.
  • PyMongo - Tools for working with MongoDB, and is the recommended way to work with MongoDB from Python.
  • Celery - Task queue to distribute work across threads or machines.
  • pytz - pytz brings the Olson tz database into Python. This library allows accurate and cross platform timezone calculations using Python 2.4 or higher.

Guides

@hrchu
hrchu / gist:33bbd35a4b95288ad695
Created March 18, 2015 09:18
yet another s3 policy example
{
"Version": "2008-10-17",
"Id": "Policy1426504761539",
"Statement": [
{
"Sid": "Stmt1426504756244",
"Effect": "Deny",
"Principal": {
"AWS": "*"
},
@hrchu
hrchu / MogileOperator.java
Created July 21, 2015 05:45
Test Moji concurrency behavior
import fm.last.moji.MojiFile;
/**
* Created by developer on 4/30/15.
*/
class MogileOperator implements Runnable {
@Override
public void run() {
MojiFile rickRoll = TestMojiConcurrency.moji.getFile("rick-astley");
@hrchu
hrchu / xstartup
Created October 22, 2015 02:24
~/.vnc/xstartup for ubuntu 14.04 (fix 3D causing problem)
#!/bin/sh
# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
@hrchu
hrchu / mogstat.py
Created March 14, 2016 08:14
The mogstat collector of python-diamond collects utilization info from the mogilefs storage system.
import diamond.collector
import telnetlib
import time
class MogstatsCollector(diamond.collector.Collector):
def get_default_config_help(self):
config_help = super(MogstatsCollector, self).get_default_config_help()
config_help.update({
})
@hrchu
hrchu / V2sign.java
Created May 10, 2016 02:44
aws java sdk force v2 signing for s3 compatible storage
public static void main(String[] args) {
ClientConfiguration clientConf = new ClientConfiguration();
clientConf.withSignerOverride("S3SignerType");
AWSCredentials myCredentials = new BasicAWSCredentials("XX", "DD");
client = new AmazonS3Client(myCredentials, clientConf);
client.setEndpoint("http://s3.com");
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("application/octet-stream");
@hrchu
hrchu / complete_mpu.py
Created November 12, 2014 05:13
Complete unfinished multipart upload by boto
import boto
from client import conn
bucket_name = '[YOUR BUCKET NAME]'
object_key = '[YOUR OBJECT KEY]'
upload_id = '[UPLOAD ID], can get it by s3cmd multipart s3://[YOUR BUCKET NAME]'
def main():
bucket = conn.lookup(bucket_name)
complete_part_upload(bucket, object_key, upload_id)