akaanirban

Edit 04/11/2021: This gist is quite old now. The current version of PyTorch is 1.8.1, which is miles ahead of version 1.0.1 that I was trying to install when I wrote this gist. Therefore some of the instructions may not apply, or some dependencies may have changed or bugs taken care of. I do not currently have a Raspberry Pi to verify unfortunately. Please proceed with caution. Further, there are may others who have shared their fixes, and direct links to their wheels down in the comments. Cheers !

Install the prerequisites (the last one for numpy):

sudo apt install libopenblas-dev libblas-dev m4 cmake cython python3-yaml libatlas-base-dev

Increase the swap size:

• Stop the swap : sudo dphys-swapfile swapoff
• Modify the size of the swap by editing as root the following file : /etc/dphys-swapfile. Modify the valiable CONF_SWAPSIZE and change its value to CONF_SWAPSIZE=2048

akaanirban

Basically this error is due to the fact that libcudnn*.so files are not present in the installation path of cuda. It appears that on installing the cuda toolkit, the cudnn files are not by default installed. Now the installation path of cuda varies in different machines/installations for example it can be /usr/local/cuda/lib64 , but also in some cases can be /usr/lib/cuda/lib64/ etc.

1. Anyway, where ever the installation path maybe, you can find it out by doing whereis cuda.
2. Once that is done, then you need to download the specific cuda version of cuDNN Library for Linux from Nvidias website: https://developer.nvidia.com/cudnn . Here in the following I download and install cuDNN for cuda 10.1.

Version 7 (libcudnn.so.7) can be directly downloaded as

wget https://developer.download.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.1_20191031/cudnn-10.1-linux-x64-v7.6.5.32.tgz?m2hqDRAYQiO6rrilch-TIU9xtf877-R5k-aUI4GXEShM3FY3Sap9Ua4wTj5LUxyQAED2lrSf2NSWhuEn-tQCvEDghZ4f

akaanirban

What if you saved some loss values / accuracy values as a list of pytorch tensor in a system with cuda and then trying to plot the losses in a system with no GPU?

With some googling I found that the following code from (pytorch/pytorch#16797 (comment)) works fine! You just need to define the custome unpickler and use it in place of pickle.load!

import io
import torch
class CPU_Unpickler(pickle.Unpickler):
    def find_class(self, module, name):
 if module == 'torch.storage' and name == '_load_from_bytes':

akaanirban

A small script to get the colors in a specific cmap as lists and then you can use them in your code. Main code was here: https://matplotlib.org/stable/gallery/color/colormap_reference.html Colormaps "Dark2" and "Set2" looks very good.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Sat Mar 13 16:08:06 2021

akaanirban

The following shows how to setup Kind locally with multiple nodes and connect to it from a remote computer.

WARNING: DO NOT DO THIS UNLESS YOU KNOW AHT YOU ARE DOING. OR UNLESS YOU ARE IN A SUBNET. KIND HAS VERY LITTLE SECURITY AND EXPOSING IT TO OUTSIDE MAY COMPROMISE YOUR SYSTEM!

Step 1:

• Install Kind in the local computer. Lets assume the ip of the local computer is a.b.c.d and you want the kubernetes control plane to run on port 4321.
• Lets further suppose you want a kind deployment with 1 master node and 3 worker node. Some of this is taken from kubernetes-sigs/kind#873 (comment) .
• Make a file kind_node_config and paste the following in it

  # four node (three workers) cluster config
  

kind: Cluster

akaanirban

import dask
import dask.dataframe as dd
import pandas as pd
import pandas as pd
import numpy as np
from pandas.tseries.holiday import USFederalHolidayCalendar
import os
import time
import pyarrow.dataset as ds

akaanirban

#!/bin.bash

set -e

# More details on other OS in https://cloud.google.com/compute/docs/gpus/install-drivers-gpu

# install docker 
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" 

akaanirban

import re
from typing import List

import spacy
from spacy.tokens import Doc
from tqdm import tqdm


class SpacyPreprocessor:
    def __init__(

akaanirban

akaanirban

##+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
## Created by: Hang Zhang, Rutgers University, Email: zhang.hang@rutgers.edu
## Modified by Thomas Wolf, HuggingFace Inc., Email: thomas@huggingface.co
## Copyright (c) 2017-2018
##
## This source code is licensed under the MIT-style license found in the
## LICENSE file in the root directory of this source tree
##+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

"""Encoding Data Parallel"""