Skip to content

Instantly share code, notes, and snippets.

from pyspark.sql.types import *
from pyspark.sql.functions import *
#Flatten array of structs and structs
def flatten(df):
# compute Complex Fields (Lists and Structs) in Schema
complex_fields = dict([(field.name, field.dataType)
for field in df.schema.fields
if type(field.dataType) == ArrayType or type(field.dataType) == StructType])
while len(complex_fields)!=0:
@lotusk
lotusk / nvidia-centos-update.md
Created June 21, 2019 09:40 — forked from frengky/nvidia-centos-update.md
Update Nvidia driver on CentOS 7.x after kernel update

Update Nvidia driver on CentOS 7.x

Download the latest Nvidia driver on http://www.nvidia.com/drivers

Update the kernel to the latest version

$ yum update

Change to runlevel 3 - multi user mode for the next reboot

@lotusk
lotusk / cuspsolve.py
Created September 17, 2018 14:12 — forked from mrkwjc/cuspsolve.py
CUDA vs. CPU sparse solver in Python
# ### Interface cuSOLVER PyCUDA
import pycuda.gpuarray as gpuarray
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np
import scipy.sparse as sp
import ctypes
## Wrap the cuSOLVER cusolverSpDcsrlsvqr() using ctypes
@lotusk
lotusk / HostSizeFilter.scala
Last active January 3, 2016 08:19
反射就错啊
反射就错啊反射就错啊
@lotusk
lotusk / gist:6051962
Created July 22, 2013 07:35
这里的 jedisPool 如果不先搞成父类就无法调用 returnBrokenResource方法. 为什么,,why 类似这样的情况有无其他方法
package com.zhisuo.redis
import redis.clients.jedis.Jedis
import redis.clients.jedis.JedisPool
import redis.clients.jedis.BinaryJedis
import redis.clients.util.Pool
/**
* 基本redis访问
*/