Skip to content

Instantly share code, notes, and snippets.

View lazykyama's full-sized avatar

Kazuhiro Yamasaki lazykyama

View GitHub Profile
@lazykyama
lazykyama / convert.py
Created August 17, 2020 06:15
Reproduction code to verify an issue related to Spark 3.0 and Yosegi Spark.
#
# This script is a part of the example below.
# https://github.com/NVIDIA/spark-xgboost-examples/blob/51df0f3d77d6d6d5d0a8a3329258376b1d59eda5/datasets/ETL/MortgageETL.ipynb
# For convenience, this script is created from original notebook, separetely.
# The purpose of this script is to confirm a bug related to yosegi-spark (https://github.com/yahoojapan/yosegi-spark).
#
import datetime
import time
@lazykyama
lazykyama / convert.scala
Created July 13, 2020 04:21
Spark 3.0 with GPU issue repro code.
// This code is based on the example below.
// https://github.com/rapidsai/spark-examples/blob/master/datasets/ETL/MortgageETL.ipynb
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.{DataFrame, SparkSession}
object ReadPerformanceCsv {
def apply(spark: SparkSession, path: String): DataFrame = {
val performanceSchema = StructType(Array(
@lazykyama
lazykyama / Dockerfile
Created April 16, 2020 11:56
Setup kocr environment based on Theano container.
# Dockerfile for kocr based on NGC's Theano container.
FROM nvcr.io/nvidia/theano:18.06
LABEL maintainer="lazykyama"
# Install dependencies.
RUN apt-get update && \
apt-get install -y openssl \
libssl-dev \
libtool-bin \
libcv-dev \
@lazykyama
lazykyama / Dockerfile
Last active April 10, 2020 14:05
Dockerfile for NEUTRINO on GPU
# Dockerfile for NEUTRINO.
FROM nvcr.io/nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04
LABEL maintainer="lazykyama"
# Install dependencies.
RUN apt update && \
apt install -y python3 python3-pip curl unzip && \
apt clean && \
rm -rf /var/lib/apt/lists/*
# Not good from here...
@lazykyama
lazykyama / Dockerfile
Last active September 15, 2019 08:14
Dockerfile to build a Chainer container with OpenMPI and NVIDIA DALI.
# To make a container including chainer, OpenMPI and DALI.
FROM nvcr.io/nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
# Install global dependencies.
RUN apt-get update && apt-get install -y vim git wget python3-pip
# Install OpenMPI, mpi4py and DALI.
WORKDIR /opt
RUN wget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.4.tar.gz \
&& tar xf openmpi-3.1.4.tar.gz
#!/usr/bin/env python
# -*- coding: utf-8-unix -*-
import os
import logging
import re
from PIL import Image
INFILENAME_PATTERN=re.compile(r'([^_]+)_img.png')
@lazykyama
lazykyama / char_imggen.py
Last active October 18, 2015 14:46
generate single character image following the given json file.
#!/usr/bin/env python
# -*- coding: utf-8-unix -*-
"""generate single character image.
Usage:
char_imggen.py [-i INPUT] [-o OUTDIR] [-v]
char_imggen.py -h|--help
char_imggen.py --version
@lazykyama
lazykyama / Vagrantfile
Last active August 29, 2015 14:27
Vagrantfiles for NLP env. 予告なく削除することがあるので注意。
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure(2) do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
@lazykyama
lazykyama / _readme.md
Last active August 29, 2015 14:22
「ソフトウェアエンジニアならば1時間以内に解けなければいけない5つの問題」の解答。The answer of "Five programming problems every Software Engineer should be able to solve in less than 1 hour".
@lazykyama
lazykyama / Dockerfile
Last active August 29, 2015 14:18
Vagrantfile and Dockerfile for Sirius, open-source end-to-end standalone intelligent personal assistant (IPA) service. Dockerfile and start-qa-server.patch MUST be moved ./docker directory.
FROM ubuntu:trusty
# forked from https://github.com/jhauswald/sirius/pull/55
MAINTAINER kyama <yamasaki.k.1101@gmail.com>
# for Ubuntu 14.04
RUN apt-get update && apt-get install -y \
software-properties-common
RUN apt-get update && apt-get install -y \