Skip to content

Instantly share code, notes, and snippets.

View vinjn's full-sized avatar

Jing Zhang vinjn

View GitHub Profile
@TengdaHan
TengdaHan / ddp_notes.md
Last active May 16, 2024 21:48
Multi-node-training on slurm with PyTorch

Multi-node-training on slurm with PyTorch

What's this?

  • A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
  • Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
  • Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
  • Warning: might need to re-factor your own code.
  • Warning: might be secretly condemned by your colleagues because using too many GPUs.
@banyudu
banyudu / free-docker-space-in-wsl2.blog.md
Last active May 11, 2024 02:11
WSL2 Docker释放磁盘空间

WSL2 Docker释放磁盘空间

Docker使用久了,会残留很多中间生成的镜像,占用磁盘空间。

要清理Docker镜像,本来是挺简单的,一条命令就搞定了:

@mcarilli
mcarilli / nsight.sh
Last active May 10, 2024 08:48
Favorite nsight systems profiling commands for Pytorch scripts
# This isn't supposed to run as a bash script, i named it with ".sh" for syntax highlighting.
# https://developer.nvidia.com/nsight-systems
# https://docs.nvidia.com/nsight-systems/profiling/index.html
# My preferred nsys (command line executable used to create profiles) commands
#
# In your script, write
# torch.cuda.nvtx.range_push("region name")
# ...
@ocornut
ocornut / imgui_node_graph_test.cpp
Last active April 23, 2024 19:02
Node graph editor basic demo for ImGui
// Creating a node graph editor for Dear ImGui
// Quick sample, not production code!
// This is quick demo I crafted in a few hours in 2015 showcasing how to use Dear ImGui to create custom stuff,
// which ended up feeding a thread full of better experiments.
// See https://github.com/ocornut/imgui/issues/306 for details
// Fast forward to 2023, see e.g. https://github.com/ocornut/imgui/wiki/Useful-Extensions#node-editors
// Changelog
// - v0.05 (2023-03): fixed for renamed api: AddBezierCurve()->AddBezierCubic().
@kylehounslow
kylehounslow / client.py
Last active April 23, 2024 10:58
Send and receive images using Flask, Numpy and OpenCV
from __future__ import print_function
import requests
import json
import cv2
addr = 'http://localhost:5000'
test_url = addr + '/api/test'
# prepare headers for http request
content_type = 'image/jpeg'
@Mxhmovd
Mxhmovd / cap.py
Created October 22, 2018 20:36
Read Youtube videos with OpenCV without downloading it locally.
#pip install pafy
#sudo pip install --upgrade youtube_dl
import cv2, pafy
url = "https://www.youtube.com/watch______"
video = pafy.new(url)
best = video.getbest(preftype="webm")
#documentation: https://pypi.org/project/pafy/
capture = cv2.VideoCapture(best.url)
@goncalossilva
goncalossilva / gifenc.sh
Last active December 12, 2023 22:31
Encode mp4, webm and gif using ffmpeg (2.6 or above). Audio is discarded on all of these!
#!/bin/sh
# sh gifenc.sh input.mp4 output.gif
# Optionally accepts width / height (one or both).
palette="/tmp/palette.png"
filters="fps=15"
if [ -n "$3" ]; then
if [ -n "$4" ]; then
filters="$filters,scale=$3:$4"
commit 6fac225de0575b18550f9421196f90a2279c44af
Author: Tiago Koji Castro Shibata <tiago.shibata@gmail.com>
Date: Fri Dec 28 23:13:14 2018 -0200
Fix compilation with latest OpenCV
C APIs were removed and must be replaced with C++ calls
diff --git a/Makefile b/Makefile
index 63e15e6..c148d4b 100644