Skip to content

Instantly share code, notes, and snippets.

View fjxmlzn's full-sized avatar

Zinan Lin fjxmlzn

View GitHub Profile
@nikhilweee
nikhilweee / dummy_gpu.py
Created April 24, 2022 20:28
Run dummy GPU job whenever usage drops below 5%
import torch
import subprocess
import time
import logging
# Takes about 8GB
ndim = 25_000
logging.basicConfig(format='[%(asctime)s] %(filename)s [%(levelname).1s] %(message)s', level=logging.DEBUG)
def get_gpu_usage():
@henrik242
henrik242 / airtag-to-gpx-sync.sh
Last active May 25, 2024 13:24
Read AirTag data from the FindMy.app cache and convert to GPX
#!/usr/bin/env bash
#
# Reads AirTag data from the FindMy.app cache and converts it to a daily GPX file
#
# Rsyncs the data to a web accessible folder that can be displayed with e.g.
# https://gist.github.com/henrik242/84ad80dd2170385fe819df1d40224cc4
#
# This should typically be run as a cron job
#
@antoinebrl
antoinebrl / README.md
Last active June 15, 2024 14:00
Prepare ImageNet

Preparation of ImageNet (ILSVRC2012)

The dataset can be found on the official website if you are affiliated with a research organization. It is also available on Academic torrents.

This script extracts all the images and group them so that folders contain images that belong to the same class.

  1. Download the ILSVRC2012_img_train.tar and ILSVRC2012_img_val.tar
  2. Download the script wget https://gist.githubusercontent.com/antoinebrl/7d00d5cb6c95ef194c737392ef7e476a/raw/dc53ad5fcb69dcde2b3e0b9d6f8f99d000ead696/prepare.sh
  3. Run it ./prepare.sh
@machuu
machuu / WSL2_VPN_Workaround_Instructions.md
Last active June 15, 2024 16:00
Workaround for WSL2 network broken on VPN

Overview

Internet connection and DNS routing are broken from WSL2 instances, when some VPNs are active.

The root cause seems to be that WSL2 and the VPN use the same IP address block, and the VPN routing clobbers WSL2's network routing.

This problem is tracked in multiple microsoft/WSL issues including, but not limited to:

@TengdaHan
TengdaHan / ddp_notes.md
Last active June 7, 2024 06:23
Multi-node-training on slurm with PyTorch

Multi-node-training on slurm with PyTorch

What's this?

  • A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
  • Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
  • Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
  • Warning: might need to re-factor your own code.
  • Warning: might be secretly condemned by your colleagues because using too many GPUs.
import requests
def format_bytes(size):
# 2**10 = 1024
power = 2 ** 10
n = 0
power_labels = {0: '', 1: 'K', 2: 'M', 3: 'G', 4: 'T'}
while size > power:
size /= power
@kkew3
kkew3 / incrwrite_npz.py
Created April 10, 2019 02:56
Write incrementally to `npz` file to save memory
import zipfile
import io
import typing
import numpy as np
class IncrementalNpzWriter:
"""
Write data to npz file incrementally rather than compute all and write
@nfekete
nfekete / wsl-fix-resolvconf.sh
Created August 8, 2018 03:05
Fix resolv.conf in Windows Subsystem for Linux, when WSL doesn't correctly generate it.
#!/bin/bash
TMP=`mktemp`
trap ctrlC INT
removeTempFiles() {
rm -f $TMP
}
ctrlC() {
import argparse
import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
import torchvision
import torchvision.transforms as T
from torchvision.datasets import ImageFolder
@batzner
batzner / tensorflow_rename_variables.py
Last active May 25, 2023 06:15
Small python script to rename variables in a TensorFlow checkpoint
import sys, getopt
import tensorflow as tf
usage_str = 'python tensorflow_rename_variables.py --checkpoint_dir=path/to/dir/ ' \
'--replace_from=substr --replace_to=substr --add_prefix=abc --dry_run'
def rename(checkpoint_dir, replace_from, replace_to, add_prefix, dry_run):
checkpoint = tf.train.get_checkpoint_state(checkpoint_dir)