harusametime

PySPQRをインストールしたい

• ScipyにはSparse matrix向けのQR分解がありません．．．
• Dense matrix向けはあります！が，todense()するとメモリで死んでしまいます．
• SuiteSparseを使うとできるけどpythonで使えない，
• Python wrapperのPySPQRを使う．

環境

• Windows 10
• Microsoft Visual Studio 14.0 (SuiteSparseのコンパイルにいる）

harusametime

harusametime

harusametime

Problem

When passing big arrays to joblib.Parallel in docker container, parallel processing does not start immediately.

Why?

Joblib uses the folder specified by "JOBLIB_TEMP_FOLDER" for memmap of the arrays. Without specifying, JOBLIB_TEMP_FOLDER is set to /dev/shm, which usually has small size and is not enough for the big arrays.

https://pythonhosted.org/joblib/generated/joblib.Parallel.html

Solution

• Set JOBLIB_TEMP_FOLDER

harusametime

import boto3

redshift = boto3.client('redshift')
credentials = redshift.get_cluster_credentials(
    DbUser='demouser', 
    DbName='dev', 
    ClusterIdentifier='redshiftml', 
    DurationSeconds=3600,
    AutoCreate=False
)

harusametime

harusametime

harusametime

Numpyで複数のndarrayを保存するとき，npzを使うのは便利だが

import numpy as np

X = np.zeros((100,100))
y = np.zeros(10)

file_path = /home/samejima/data.npz
np.savez(file_path, x = X, y=y) 

harusametime

import os
import argparse
import tensorflow as tf

# global variables
width = 28
height = 28
n_class = 10

# tfrecord parser

harusametime