ryanpedersen42

version: 2.1

setup: true

orbs:
  path-filtering: circleci/path-filtering@0.0.2

workflows:
  setup-workflow:
    jobs:

vncsna

set -e, -u, -o, -x pipefail

The set lines

• These lines deliberately cause your script to fail. Wait, what? Believe me, this is a good thing.
• With these settings, certain common errors will cause the script to immediately fail, explicitly and loudly. Otherwise, you can get hidden bugs that are discovered only when they blow up in production.

• set -euxo pipefail is short for:

set -e
set -u

thsutton

import logging
import time
from contextlib import AbstractContextManager
from logging import Handler, LogRecord
from typing import Any, List, Optional

from pyspark.sql import SparkSession


logging.basicConfig(level=logging.INFO)

cartershanklin

#!/usr/bin/env python

# This script executes Example 1 of the OCI Data Flow Tutorial
# https://docs.cloud.oracle.com/en-us/iaas/data-flow/data-flow-tutorial/tutorial/dfs_tut_etl_java.htm#etl_with_java

import argparse
import oci
import sys
import time

cartershanklin

#!/usr/bin/env bash

# Copyright (c) 2020, Oracle and/or its affiliates. All rights reserved.
#
# This script executes Example 1 of the Data Flow Tutorial
# https://docs.cloud.oracle.com/en-us/iaas/data-flow/data-flow-tutorial/tutorial/dfs_tut_etl_java.htm#etl_with_java
#
# For more help with specific Data Flow commands, see:
#   oci data-flow -h
#

drorata

• Ansible
  • AWS and boto3
    • Change tags of AWS-Glue crawler
• Athena
  • Create partitioned external table
    • Assumptions
    • Create the table
    • Fix the table
    • Query the table
• Docker

CarlosDomingues

Create a new project

poetry new <project-name>

Add a new lib

poetry add <library>

Remove a lib

dimaspivak

# Install clusterdock.
pip3 install clusterdock

# Clone the Apache Kafka topology for clusterdock.
git clone https://github.com/clusterdock/topology_apache_kafka.git

# Start Apache Kafka (defaults to node-1.cluster, node-2.cluster, node-3.cluster)
clusterdock -v start topology_apache_kafka --brokers node-1 node-2 node-3 --kafka-version 1.0.0

# Start StreamSets Data Collector on the same cluster network.

paulochf

from s3fs.core import S3FileSystem

s3 = S3FileSystem(anon=False, profile_name="your-profile-name")

print(s3.ls("s3://your-bucket/some-folder"))

ruanbekker

My Elasticsearch cheatsheet with example usage via rest api (still a work-in-progress)

Shortlinks:

• Cluster Health
  • Index Level
  • Shard Level
• Nodes Overview
• Indices Overview
• Cluster Maintenance