Skip to content

Instantly share code, notes, and snippets.

View kk17's full-sized avatar

Kyle Zhike Chen kk17

View GitHub Profile
@kk17
kk17 / train_on_ray2.ipynb
Created September 7, 2022 02:19
train_on_ray2 Tuner Error
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@kk17
kk17 / acme-cert-dump-all.py
Last active November 27, 2020 03:37 — forked from mikifus/acme-cert-dump-all.py
Dump all certificates from Traefik's acme.json to .pem files
#!/usr/bin/env python
import argparse
import base64
import json
import os
def main():
parser = argparse.ArgumentParser(
description="Dump all certificates out of Traefik's acme.json file")
@kk17
kk17 / .env
Last active September 18, 2019 08:32
Deploying Spark on Spark Standalone Cluster using Jupyter Docker Image
#2.2.0
# SPARK_NOTEBOOK_VERSION=400c69639ea5
SPARK_NOTEBOOK_VERSION=latest
SPARK_HOME=/usr/local/spark
@kk17
kk17 / CatalogUtil.scala
Created April 27, 2019 08:49
Pitfalls in Using Spark Catalog API
package org.apache.spark.sql.catalog
import org.apache.spark.sql.catalyst.catalog.{CatalogTable, CatalogTableType, CatalogUtils => SparkCatalogUtils}
import org.apache.spark.sql.execution.datasources.{DataSource, HadoopFsRelation}
import org.apache.spark.sql.sources.BaseRelation
import org.apache.spark.sql.types.{StructField, StructType}
import org.apache.spark.sql.{DataFrame, SparkSession}
/**
* Util class for creating an external hive metastore table with user specified schema path.
@kk17
kk17 / Dockerfile
Created March 28, 2019 07:30
Spark-SQL-Ranger
FROM centos:7
RUN yum -y update
RUN yum -y install wget gcc openldap-devel
RUN yum -y install https://centos7.iuscommunity.org/ius-release.rpm
RUN yum -y install -y python36u python36u-libs python36u-devel python36u-pip
# JAVA
# /dev/urandom is used as random source, which is prefectly safe
# according to http://www.2uo.de/myths-about-urandom/
@kk17
kk17 / SpringBeanMethodCommandLineRunner.java
Created May 21, 2018 13:07
在命令行参数中指定spring BeanName,方法和方法参数即可运行对应的方法
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
@kk17
kk17 / docker-cleanup-resources.md
Created November 27, 2017 02:50 — forked from bastman/docker-cleanup-resources.md
docker cleanup guide: containers, images, volumes, networks

Docker - How to cleanup (unused) resources

Once in a while, you may need to cleanup resources (containers, volumes, images, networks) ...

delete volumes

// see: https://github.com/chadoe/docker-cleanup-volumes

$ docker volume rm $(docker volume ls -qf dangling=true)

$ docker volume ls -qf dangling=true | xargs -r docker volume rm

{eleveldb, [
{total_leveldb_mem_percent, 30}
]},
{riak_kv, [
%% Other configs
{add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-2.1.1/ebin"]},
{storage_backend, riak_cs_kv_multi_backend},
{multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
{multi_backend_default, be_default},
{multi_backend, [
@kk17
kk17 / DynamicCompileClassLoader.java
Created March 10, 2017 10:57
A Java ClassLoader that can load classes from Java source file. This ClassLoader can't compile recursively.
package test;
import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;