timvw

resource "snowflake_unsafe_execute" "volume" {
  execute = <<EOT
CREATE OR REPLACE EXTERNAL VOLUME ${var.external_volume_name}
   STORAGE_LOCATIONS =
      (
         (
            NAME = '${var.external_volume_bucket_name}'
            STORAGE_PROVIDER = 'S3'
            STORAGE_BASE_URL = 's3://${var.external_volume_bucket_name}'
            STORAGE_AWS_ROLE_ARN = '${var.snowflake_iam_role_arn}'

noamsdahan

thsutton

import logging
import time
from contextlib import AbstractContextManager
from logging import Handler, LogRecord
from typing import Any, List, Optional

from pyspark.sql import SparkSession


logging.basicConfig(level=logging.INFO)

smartnose

Spark internals through code

Nothing gives you more detail about spark internals than actually reading it source code. In addition, you get to learn many design techniques and improve your scala coding skills. These are the random notes I make while reading the spark code. The best way to comprehend the notes is to load spark code into an IDE, e.g. IntelliJ, and navigate the code on the side.

Genesis - creation of a spark cluster

The scripts for creating a spark cluster are: start-master.sh and start-slave.sh. Read them carefully, and you can see that both scripts are very similar except the values for $CLASS variable. For start-master.sh, the value is CLASS="org.apache.spark.deploy.master.Master", while the value for start-slave.sh is shown below with more context.

# NOTE: This exact class name is matched downstream by SparkSubmit.

remmelt

#!/usr/bin/python

"""

Create a stage in your project, make it the last stage.
Make a task in the stage with this inline script:

#! /bin/bash

/some/path/bamboo-to-slack.py "${bamboo.planKey}" "${bamboo.buildPlanName}" "${bamboo.buildResultsUrl}"

bitemyapp

The Learning Haskell guide has been moved to a git repo to enable outside contribution

Find it here: https://github.com/bitemyapp/learnhaskell

simonmorley

server {
  listen   8443 default ssl;
  server_name  www.yourdomain.com;
  ssl                  on;
  ssl_certificate      /etc/nginx/conf.d/cert.pem;
  ssl_certificate_key  /etc/nginx/conf.d/cert.key;

  client_max_body_size 50M;

  error_log   /var/log/nginx/elasticsearch-errors.log;