timrobertson100

Based on ideas here, expanded to enable Hive support
https://www.linkedin.com/pulse/running-spark-2xx-cloudera-hadoop-distro-cdh-deenar-toraskar-cfa/

wget https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-without-hadoop.tgz

tar -xvzf spark-2.4.8-bin-without-hadoop.tgz
cd spark-2.4.8-bin-without-hadoop

cp -R /etc/spark2/conf/* conf/
cp /etc/hive/conf/hive-site.xml conf/

asdaraujo

# Requirements: kafka-python gssapi krbticket
import os
import time
from kafka import KafkaConsumer, KafkaProducer
from krbticket import KrbConfig, KrbCommand

try:
    os.environ['KRB5CCNAME'] = '/tmp/krb5cc_<myusername>'
    kconfig = KrbConfig(principal='araujo', keytab='/path/to/<myusername>.keytab')
    KrbCommand.kinit(kconfig)

mrpeardotnet

HP Smart Storage Admin CLI (ssacli) installation and usage on Proxmox PVE (6.x)

Why use HP Smart Storage Admin CLI?

You can use ssacli (smart storage administrator command line interface) tool to manage any of supported HP Smart Array Controllers in your Proxmox host without need to reboot your server to access Smart Storage Administrator in BIOS. That means no host downtime when managing your storage.

CLI is not as convenient as GUI interface provided by BIOS or desktop utilities, but still allows you to fully manage your controller, physical disks and logical drives on the fly with no Proxmox host downtime.

ssacli replaces older hpssacli, but shares the same syntax and adds support for newer servers and controllers.

Installation

achintya-kumar

Kafka console consumer with Kerberos

1. Create a jaas.conf file with the following contents:

KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="keytabFile.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"

wagnerjgoncalves

# -*- coding: utf-8 -*-

"""
    Example of Python Data Frame with SparkSession.
"""
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *

croxton

Generate ssl certificates with Subject Alt Names on OSX

Open ssl.conf in a text editor.

Edit the domain(s) listed under the [alt_names] section so that they match the local domain name you want to use for your project, e.g.

DNS.1   = my-project.dev

Additional FQDNs can be added if required:

linar-jether

from flask import Flask, request, jsonify, json, abort
from flask_cors import CORS, cross_origin

import pandas as pd

app = Flask(__name__)

cors = CORS(app)
app.config['CORS_HEADERS'] = 'Content-Type'

TonyWuLihu

import sys
from datetime import date,datetime,timedelta
import datetime
import string
from pexpect import *
from os import remove,listdir
import os
import pprint

def chunks(l, n):

aseigneurin

Spark - High availability

Components in play

As a reminder, here are the components in play to run an application:

• The cluster:
  • Spark Master: coordinates the resources
  • Spark Workers: offer resources to run the applications
• The application:

rajkrrsingh

su - hdfs 
wget https://github.com/hortonworks/hive-testbench/archive/hive14.zip
unzip hive14.zip
cd hive-testbench-hive14/
vi hive-testbench-hive14/settings/load-partitioned.sql -- remove G1GC and use Parrallel GC scheme
cd hive-testbench-hive14/
yum install gcc
echo 'export JAVA_HOME=/usr/jdk64/jdk1.8.0_77' >> ~/.bashrc
echo 'PATH=$PATH:$JAVA_HOME/bin' >> ~/.bashrc