jdye64

#!/usr/bin/python

print "Converting all of the .dav files in this current directory into .mp4 files using ffmpeg"	
import os
from subprocess import call

files = [f for f in os.listdir('.') if os.path.isfile(f)]
for f in files:
    ext = f.split(".")[-1]
    if ext == "dav" or ext == "DAV":

jdye64

import pandas as pd
import dask.dataframe as dd

# Create an empty dask.dataframe
df = dd.from_pandas(pd.DataFrame(), npartitions=1)
mappings = {'a': 1}

# Assign the new columns and data
df = df.assign(**mappings)

jdye64

git clone https://github.com/dask-contrib/dask-sql.git
cd dask-sql
git checkout datafusion-sql-planner
conda env create -n dask-datafusion -f ./continuous_integration/environment-3.9-dev.yaml
conda activate dask-datafusion
python ./setup.py install

jdye64

Dask-SQL SegFault notes and observations

Notes

• Occurs regardless of LocalCUDACluster transport specified. Ex: UCX, TCP, etc
• Only occurs when ucx-py is installed in the Anaconda environment AND LocalCUDACluster is used instead of standard Distributed.Client
• Any environment without UCX and issues cannot be reproduced

Results

I have provided 2 test cases. One with ucx and one without. The tests are as close as possible (some imports had to be removed) to demonstrate the failures.

jdye64

#!/bin/bash
# These are examples for the stable V1. They will not be broken by any future V1 releases.

# Global Variables
USER="admin"
PASS="admin"
AMBARI_HOST="localhost"
AMBARI_PORT=8080

# -------- BEGIN HDP Stack Admin Actions --------

jdye64

from custreamz import kafka

# How to connect to Kafka, brokers, partitions, security, etc ...
# Full list of configurations can be found at: 
# https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
kafka_configs = {
	"metadata.broker.list": "localhost:9092",
	"group.id": "custreamz-client",
}

jdye64

#!/bin/bash

# Global Arguments
BUILD_ROOT=~/Development
CMAKE_ROOT=$BUILD_ROOT/cmake
LLVM_ROOT=$BUILD_ROOT/llvm
LIBCUDF_BUILD_ROOT=$BUILD_ROOT/cudf/cpp/build
CUDF_BUILD_ROOT=$BUILD_ROOT/cudf/python/cudf

echo "======== RapidsAI Xavier Installation Script ========"

jdye64

1) clone the cudf repo, cloned from my repo, made cudf_xavier branch and added upstream to rapidsai/cudf expecting there might be some code changes I need to make to cudf and can capture those changes in this branch
2) Installed cmake via sudo apt-get install cmake since build.sh wouldn't work without cmake installed
3) That caused problems because the cmake version is installed was 3.10.2 and cudf needs >= 3.12 .... lets try something else. SKIP THIS STEP!
4) I wanted to do this without conda but I'm going to install conda and use the cmake that it installs.
5) Of course Anaconda does not seem to officially support ARM64 so that route is not going to work ... something else
6) I could build the latest version of cmake from source ... lets try that. Cmake does not offer binaries for ARM64 directly without building them.
7) cd /tmp && wget https://github.com/Kitware/CMake/releases/download/v3.16.2/cmake-3.16.2.tar.gz && tar -xzvf ./cmake-3.16.2.tar.gz && cd cmake-3.16.2 && ./bootstrap && make && make install
7

jdye64

sudo su && yum install -y wget
cd /opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz"
tar xzf jdk-8u131-linux-x64.tar.gz
cd /opt/jdk1.8.0_131/
alternatives --install /usr/bin/java java /opt/jdk1.8.0_131/bin/java 2
alternatives --config java
alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_131/bin/jar 2
alternatives --install /usr/bin/javac javac /opt/jdk1.8.0_131/bin/javac 2
alternatives --set jar /opt/jdk1.8.0_131/bin/jar

jdye64

#!/bin/bash

#### CHANGE DOCKER ROOT DIRECTORY ####
# For Azure VM edit vi /lib/systemd/system/docker.service to add the -g flag to the directory that you want be the Docker root directory
# systemctl daemon-reload
# systemctl restart docker

#### REDHAT 7 STOPPING FIREWALLD ####
# service firewalld stop
# systemctl disable firewalld