ceteri

$ ./bin/spark-shell
14/04/18 15:23:49 INFO spark.HttpServer: Starting HTTP Server
14/04/18 15:23:49 INFO server.Server: jetty-7.x.y-SNAPSHOT
14/04/18 15:23:49 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:49861
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 0.9.1
      /_/

miglen

Amazon Web Services (AWS) & Google Cloud Platform (GCP) explained in simple English

This guide is only representative from my point of view and it may not be accurate and you should go on the official AWS & GCP websites for accurate and detailed information. It's initially inspired by AWS in simple English and GCP for AWS professionals. The idea is to compare both services, give simple one-line explanation and examples with other software that might have similiar capabilities. Comment below for suggestions.

Category	Service	AWS	GCP	Description	It's like
Compute	IaaS	Amazon Elastic Compute Cloud (EC2)	Google Compute Engine	Type-1 virtual servers	VMware ESXi, Citrix XenServer
	PaaS	AWS Elastic Beanstalk	Google App Engine	Running your app on a platform

miglen

AWS Certification notes

Those are my personal notes on AWS Solution Architect certification preparation. Hope you find them usefull.

To pass AWS certification, you should have:

• Sound knowledge about most of the AWS services ( EC2, VPC, RDS, Cloudfront, S3, Route53 etc,)
• Hands on experience with AWS services.

meisheep

from __future__ import print_function
import json
import boto3
import os
import time

client = boto3.client('emr')
s3client = boto3.client('s3')
job_id = os.urandom(16).encode('hex')
city_dict = {'TPE': 'A','NTP': 'F','TY': 'H','HCC': 'J','HC': 'O','ML': 'K','TC': 'B','NT': 'M','CH': 'N','YUL': 'P','CY': 'I','CYC': 'Q','TN': 'D','KH': 'E','PT': 'T','KL': 'C','YIL': 'G','HL': 'U','TT': 'V'}

vaquarkhan

AWS Certification notes

Those are my personal notes on AWS Solution Architect certification preparation. Hope you find them usefull.

To pass AWS certification, you should have:

• Sound knowledge about most of the AWS services ( EC2, VPC, RDS, Cloudfront, S3, Route53 etc,)
• Hands on experience with AWS services.
• Good knowledge of disaster recovery, security and High availability architectures.

If you do not have prior hands-on experience and knowledge or you have little knowledge about AWS services, it is better to take an online course. If you already have experience in architecting solutions on AWS it is not necessary to take an online course.

SimoneStefani

/**
 * Code written by P. Gajland
 * https://github.com/GaPhil
 *
 * IMPORTANT:
 * This code is for educational and demonstrative purpose only.
 * If you need to do serious encryption for "production" it is
 * recommended to investigate more traditional libraries and
 * gain some specific knowledge on cryptography and security.
 */

pgolding

Handling of StreamingBody Response from Invoking a Lambda Function

When calling a Lambda Function via Boto3, the returned object is something like:

{u'Payload': <botocore.response.StreamingBody object at 0x7f8d5b62ac50>, 
'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '5bdbb3ca-3b35-11e7-9816-397b140c3bac', 'HTTPHeaders': {'x-amzn-requestid': '5bdbb3ca-3b35-11e7-9816-397b140c3bac', 'content-length': '1636', 'x-amzn-trace-id': 'root=1-591ca199-00d34d5474d16275ec2c8d10;sampled=0', 'x-amzn-remapped-content-length': '0', 'connection': 'keep-alive', 'date': 'Wed, 17 May 2017 19:16:41 GMT', 'content-type': 'application/json'}}, u'StatusCode': 200}

The Payload parameter is <botocore.response.StreamingBody> which is a data streaming object.

jamesrajendran

1.mapPartition() instead of map()    - when some expensive initializations like DBconnection need to be done

2.RDD Parallelism: for No parent RDDs, example, sc.parallelize(',,,',4),Unless specified YARN will try to use as many CPU cores as available 
	This could be tuned using spark.default.parallelism property.
	
	- to find default parallelism use sc.defaultParallelism
	  rdd.getNumPartitions()
	  rdd = sc.parallelize(<value>, numSlices=4)
	  rdd.getNumPartitions() will return 4
	  

tsaastam

// illustration of https://issues.apache.org/jira/browse/SPARK-21109                                          
// see also https://lobotomys.blogspot.co.uk/2017/07/spark-union-column-order-issue.html                      

import org.apache.spark.rdd.RDD
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

// if using spark-shell, skip the next 4 lines                                                                

pratos

bleach==1.5.0
certifi==2016.2.28
cycler==0.10.0
decorator==4.1.2
entrypoints==0.2.3
html5lib==0.9999999
ipykernel==4.6.1
ipython==6.2.1
ipython-genutils==0.2.0
ipywidgets==7.0.3