Skip to content

Instantly share code, notes, and snippets.


Harshavardhana harshavardhana

View GitHub Profile
harshavardhana / generate_hive_schema.scala
Created Oct 10, 2021 — forked from spektom/generate_hive_schema.scala
Generate Hive schema from Spark Dataframe
View generate_hive_schema.scala
import org.apache.spark.sql.DataFrame
def dataFrameToDDL(dataFrame: DataFrame, tableName: String): String = {
val columns = { field =>
" " + + " " + field.dataType.simpleString.toUpperCase
s"CREATE TABLE $tableName (\n${columns.mkString(",\n")}\n)"
import spark.sqlContext.implicits._
View gist:89fdd38b069a2195886cd9bf8bdfa141
# Resource script for Minio
# Description: Manages Minio as an OCF resource in
# an Active-Passive High Availability setup.
# Author: Ricardo Branco <> : Initial script for minio server
# License: GNU General Public License (GPL)

Erasure code sizing guide

Toy Setup

Capacity constrainted environments, works but not recommended for production.

servers drives stripe_size parity
1 1 1 0
1 4 4 2
View instance.yaml
apiVersion: v1
kind: Secret
name: test-minio-creds-secret
type: Opaque
accesskey: bWluaW8= # base 64 encoded "minio" (echo -n 'minio' | base64)
secretkey: bWluaW8xMjM= # based 64 encoded "minio123" (echo -n 'minio123' | base64)
# This version makes a GET request and passes the signature
# in the Authorization header.
import sys, os, base64, datetime, hashlib, hmac, urllib
import requests
import logging
from minio.signer import presign_v4
from minio.helpers import get_target_url
from minio.credentials import Credentials, Static
View kubespray-calico.yaml
# You can set MTU value here. If left undefined or empty, it will
# not be specified in calico CNI config, so Calico will use built-in
# defaults. The value should be a number, not a string.
calico_mtu: 1500
# Configure the MTU to use for workload interfaces and tunnels.
# - If Wireguard is enabled, set to your network MTU - 60
# - Otherwise, if VXLAN or BPF mode is enabled, set to your network MTU - 50
# - Otherwise, if IPIP is enabled, set to your network MTU - 20
# - Otherwise, if not using any encapsulation, set to your network MTU.

Creating PersistentVolumes

Slack Docker Pulls

We recommend using local Persistent Volumes as the backing storage for MinIO StatefulSets. MinIO creates a distributed, resilient, failure tolerant storage system on local PVs. There is no explicit dependency on external distributed file systems or CSI vendors.

This document explains how to create local PVs. Once you create these PVs, MinIOInstance PVCs should automatically bind to these PVs.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,

Start Dex

~ ./bin/dex serve dex.yaml
time="2020-07-12T20:45:50Z" level=info msg="config issuer:"
time="2020-07-12T20:45:50Z" level=info msg="config storage: sqlite3"
time="2020-07-12T20:45:50Z" level=info msg="config static client: Example App"
time="2020-07-12T20:45:50Z" level=info msg="config connector: mock"
time="2020-07-12T20:45:50Z" level=info msg="config connector: local passwords enabled"
time="2020-07-12T20:45:50Z" level=info msg="config response types accepted: [code token id_token]"
View functional_tests.go
// +build ignore
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at