Skip to content

Instantly share code, notes, and snippets.

View harshavardhana's full-sized avatar
🌚
I may be slow to respond.

Harshavardhana harshavardhana

🌚
I may be slow to respond.
View GitHub Profile
API AWS S3 MinIO GCS (S3 compatibility)
ListObjectVersions ✔️ ✔️

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc ls --versions gcs/harshavardhana
mc: <ERROR> Unable to list folder. unrecognized option:Marker
@harshavardhana
harshavardhana / generate_hive_schema.scala
Created October 10, 2021 21:58 — forked from spektom/generate_hive_schema.scala
Generate Hive schema from Spark Dataframe
import org.apache.spark.sql.DataFrame
def dataFrameToDDL(dataFrame: DataFrame, tableName: String): String = {
val columns = dataFrame.schema.map { field =>
" " + field.name + " " + field.dataType.simpleString.toUpperCase
}
s"CREATE TABLE $tableName (\n${columns.mkString(",\n")}\n)"
}
import spark.sqlContext.implicits._
#!/bin/sh
#
# Resource script for Minio
#
# Description: Manages Minio as an OCF resource in
# an Active-Passive High Availability setup.
#
# Author: Ricardo Branco <tsmgeek@gmail.com> : Initial script for minio server
# License: GNU General Public License (GPL)
#

Erasure code sizing guide

Toy Setup

Capacity constrainted environments, works but not recommended for production.

servers drives stripe_size parity
1 1 1 0
1 4 4 2
apiVersion: v1
kind: Secret
metadata:
name: test-minio-creds-secret
type: Opaque
data:
accesskey: bWluaW8= # base 64 encoded "minio" (echo -n 'minio' | base64)
secretkey: bWluaW8xMjM= # based 64 encoded "minio123" (echo -n 'minio123' | base64)
---
apiVersion: minio.min.io/v1
# This version makes a GET request and passes the signature
# in the Authorization header.
import sys, os, base64, datetime, hashlib, hmac, urllib
import requests
import logging
from minio.signer import presign_v4
from minio.helpers import get_target_url
from minio.credentials import Credentials, Static
# You can set MTU value here. If left undefined or empty, it will
# not be specified in calico CNI config, so Calico will use built-in
# defaults. The value should be a number, not a string.
calico_mtu: 1500
# Configure the MTU to use for workload interfaces and tunnels.
# - If Wireguard is enabled, set to your network MTU - 60
# - Otherwise, if VXLAN or BPF mode is enabled, set to your network MTU - 50
# - Otherwise, if IPIP is enabled, set to your network MTU - 20
# - Otherwise, if not using any encapsulation, set to your network MTU.

Creating PersistentVolumes

Slack Docker Pulls

We recommend using local Persistent Volumes as the backing storage for MinIO StatefulSets. MinIO creates a distributed, resilient, failure tolerant storage system on local PVs. There is no explicit dependency on external distributed file systems or CSI vendors.

This document explains how to create local PVs. Once you create these PVs, MinIOInstance PVCs should automatically bind to these PVs.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,

Start Dex

~ ./bin/dex serve dex.yaml
time="2020-07-12T20:45:50Z" level=info msg="config issuer: http://127.0.0.1:5556/dex"
time="2020-07-12T20:45:50Z" level=info msg="config storage: sqlite3"
time="2020-07-12T20:45:50Z" level=info msg="config static client: Example App"
time="2020-07-12T20:45:50Z" level=info msg="config connector: mock"
time="2020-07-12T20:45:50Z" level=info msg="config connector: local passwords enabled"
time="2020-07-12T20:45:50Z" level=info msg="config response types accepted: [code token id_token]"