nikhilweee

import torch
import subprocess
import time
import logging

# Takes about 8GB
ndim = 25_000
logging.basicConfig(format='[%(asctime)s] %(filename)s [%(levelname).1s] %(message)s', level=logging.DEBUG)

def get_gpu_usage():

henrik242

#!/usr/bin/env bash

# 
# Reads AirTag data from the FindMy.app cache and converts it to a daily GPX file
#
# Rsyncs the data to a web accessible folder that can be displayed with e.g. 
# https://gist.github.com/henrik242/84ad80dd2170385fe819df1d40224cc4
#
# This should typically be run as a cron job
#

antoinebrl

Preparation of ImageNet (ILSVRC2012)

The dataset can be found on the official website if you are affiliated with a research organization. It is also available on Academic torrents.

This script extracts all the images and group them so that folders contain images that belong to the same class.

1. Download the ILSVRC2012_img_train.tar and ILSVRC2012_img_val.tar
2. Download the script wget https://gist.githubusercontent.com/antoinebrl/7d00d5cb6c95ef194c737392ef7e476a/raw/dc53ad5fcb69dcde2b3e0b9d6f8f99d000ead696/prepare.sh
3. Run it ./prepare.sh

machuu

Overview

Internet connection and DNS routing are broken from WSL2 instances, when some VPNs are active.

The root cause seems to be that WSL2 and the VPN use the same IP address block, and the VPN routing clobbers WSL2's network routing.

This problem is tracked in multiple microsoft/WSL issues including, but not limited to:

• microsoft/WSL#5068
• microsoft/WSL#4277

TengdaHan

Multi-node-training on slurm with PyTorch

What's this?

• A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
• Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
• Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
• Warning: might need to re-factor your own code.
• Warning: might be secretly condemned by your colleagues because using too many GPUs.

ypwhs

import requests


def format_bytes(size):
    # 2**10 = 1024
    power = 2 ** 10
    n = 0
    power_labels = {0: '', 1: 'K', 2: 'M', 3: 'G', 4: 'T'}
    while size > power:
        size /= power

kkew3

import zipfile
import io
import typing

import numpy as np


class IncrementalNpzWriter:
    """
    Write data to npz file incrementally rather than compute all and write

nfekete

#!/bin/bash

TMP=`mktemp`
trap ctrlC INT

removeTempFiles() {
	rm -f $TMP
}

ctrlC() {

jcjohnson

import argparse
import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.utils.data import DataLoader

import torchvision
import torchvision.transforms as T
from torchvision.datasets import ImageFolder

batzner

import sys, getopt

import tensorflow as tf

usage_str = 'python tensorflow_rename_variables.py --checkpoint_dir=path/to/dir/ ' \
            '--replace_from=substr --replace_to=substr --add_prefix=abc --dry_run'


def rename(checkpoint_dir, replace_from, replace_to, add_prefix, dry_run):
    checkpoint = tf.train.get_checkpoint_state(checkpoint_dir)