Randl

% Fixed extra right bracket
%
% Evgenii Zheltonozhskii, 09/28/2020, zheltonozhskiy@gmail.com
%
% ---------------------------------------------------------------
% Modified CVPR ieee_fullname.bst to support natbib
%
% Evgenii Zheltonozhskii, 03/10/2019, zheltonozhskiy@gmail.com
%
% ---------------------------------------------------------------

ZijiaLewisLu

In most of deep learning projects, the training scripts always start with lines to load in data, which can easily take a handful minutes. Only after data ready can start testing my buggy code. It is so frustratingly often that I wait for ten minutes just to find I made a stupid typo, then I have to restart and wait for another ten minutes hoping no other typos are made.

In order to make my life easy, I devote lots of effort to reduce the overhead of I/O loading. Here I list some useful tricks I found and hope they also save you some time.

1. use Numpy Memmap to load array and say goodbye to HDF5.

   I used to relay on HDF5 to read/write data, especially when loading only sub-part of all data. Yet that was before I realized how fast and charming Numpy Memmapfile is. In short, Memmapfile does not load in the whole array at open, and only later "lazily" load in the parts that are required for real operations.

Sometimes I may want to copy the full array to memory at once, as it makes later operations

bmmalone

\documentclass[a4paper,10pt]{article}

% make writing commands easier
\usepackage{xparse}

% colored text
\usepackage{color}

% include eps, pdf graphics
\usepackage{graphicx}

luckydev

# maximum capability of system
user@ubuntu:~$ cat /proc/sys/fs/file-max
708444

# available limit
user@ubuntu:~$ ulimit -n
1024

# To increase the available limit to say 200000
user@ubuntu:~$ sudo vim /etc/sysctl.conf

alexlee-gk

This was tested on a ThinkPad P70 laptop with an Intel integrated graphics and an NVIDIA GPU:

lspci | egrep 'VGA|3D'
00:02.0 VGA compatible controller: Intel Corporation Device 191b (rev 06)
01:00.0 VGA compatible controller: NVIDIA Corporation GM204GLM [Quadro M3000M] (rev a1)

A reason to use the integrated graphics for display is if installing the NVIDIA drivers causes the display to stop working properly. In my case, Ubuntu would get stuck in a login loop after installing the NVIDIA drivers. This happened regardless if I installed the drivers from the "Additional Drivers" tab in "System Settings" or the ppa:graphics-drivers/ppa in the command-line.