Skip to content

Instantly share code, notes, and snippets.

View akki's full-sized avatar
🇮🇳
.

Akshesh Doshi akki

🇮🇳
.
View GitHub Profile
@akki
akki / benchmark-spark-magic-committer.py
Last active March 28, 2024 14:22
Spark app with magic committer
# These extra-configs enable the magic committer.
# A _SUCCESS file with data is generated if these extra configs are passed. The file would otherwise be zero-byte in size, indicating that magic commiter did NOT get enabled.
extra_configs = {
"spark.sql.sources.commitProtocolClass": "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol",
"spark.sql.parquet.output.committer.class": "org.apache.hadoop.mapreduce.lib.output.BindingPathOutputCommitter",
"spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a": "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory",
"spark.hadoop.fs.s3a.committer.name": "magic",
"spark.hadoop.fs.s3a.committer.magic.enabled": "true",
}
@akki
akki / stack.yml
Last active March 2, 2020 04:08
Docker-Swarm-Operator-for-Airflow-blog
version: '3'
services:
postgres:
image: postgres:9.6-alpine
environment:
- POSTGRES_USER=test
- POSTGRES_PASSWORD=test
- POSTGRES_DB=test
pod:
[Unit]
Requires=zookeeper.service
After=zookeeper.service
[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
@akki
akki / mongo-hadoop-core-2.0.2.jar
Created June 1, 2018 11:55
JAR files for MongoDB-Hadoop connector
" zip.vim version v27
" Browsing zipfile /home/hadoopuser/hadoop/lib/mongo-hadoop-core-2.0.2.jar
" Select a file with cursor and press ENTER
META-INF/
META-INF/MANIFEST.MF
com/
com/mongodb/
com/mongodb/hadoop/
com/mongodb/hadoop/MongoOutputFormat.class
https://qph.ec.quoracdn.net/main-qimg-b1a6103989e261ac377e821fbd192393-c
___ _ __ _ _ ___ ___ ___
/ \__ _| |_ __ _ / _(_) | ___ / \/ __\ / _ \
/ /\ / _` | __/ _` | |_| | |/ _ \ / /\ /__\// / /_\/
/ /_// (_| | || (_| | _| | | __/ / /_// \/ \/ /_\\
/___,' \__,_|\__\__,_|_| |_|_|\___| /___,'\_____/\____/
Collection to check:
Prints details (Y/N/full)? Checking collection #8:
-------------------------------------------------------------------
mkdir /var/lock/subsys
touch /var/lock/subsys/listener
sudo rm -rf /dev/shm
sudo mkdir /dev/shm
sudo mount -t tmpfs shmfs -o size=2048m /dev/shm
sudo service oracle-xe restart
Desired output:
relname | array | indisunique | indisprimary | order | type (can skip this if it is difficult to get this column)
---------------------------------+-------+-------------+--------------+-------------+-----------------------------------------------------------
ticket26180_indexes_spamin_pkey | {id} | t | t | {ASC} | btree
ticket26180_a_6fe9a5_idx | {a,b} | f | f | {ASC, ASC} | btree
test26180_indexes_spamin_atpata | {a,b} | f | f | {ASC, DESC} | btree
test26180_indexes_spamin_b_hsh | {b} | f | f | | hash
test26180_indexes_spamin_a_hsh | {a} | f | f | | hash
@akki
akki / gsoc-report.md
Last active January 13, 2019 15:48
My work submission report for GSoC 2016

INDEX STATEMENTS

POSTGRESQL

  • CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] idx_name ON table_name [ USING method ] ( { col_name | ( expression ) } [ opclass ] [ ASC | DESC ]
  • DROP INDEX [ IF EXISTS ] idx_name [, idx_name1, ...]