Raidus

Apache Spark installation + ipython/jupyter notebook integration guide for macOS

Tested with Apache Spark 2.1.0, Python 2.7.13 and Java 1.8.0_112

For older versions of Spark and ipython, please, see also previous version of text.

Install Java Development Kit

Raidus

// NOTE: only escapes a " if it's not already escaped
function escapeDoubleQuotes(str) {
	return str.replace(/\\([\s\S])|(")/g,"\\$1$2"); // thanks @slevithan!
}

escapeDoubleQuotes("ab");   // ab  =>  ab
escapeDoubleQuotes("a\"b");   // a"b  =>  a\"b
escapeDoubleQuotes("a\\\"b");   // a\"b  =>  a\"b
escapeDoubleQuotes("a\\\\\"b");   // a\\"b  =>  a\\\"b
escapeDoubleQuotes("a\\\\\\\"b");   // a\\\"b  =>  a\\\"b

Raidus

I’m looking for any tips or tricks for making chrome headless mode less detectable. Here is what I’ve done so far:

Set my args as follows:

const run = (async () => {

    const args = [
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-infobars',

Raidus

(function() {
    'use strict';

    // keep track of all the opened tab
    let tabs = {};

    // Get all existing tabs
    chrome.tabs.query({}, function(results) {
        results.forEach(function(tab) {
            tabs[tab.id] = tab;

Raidus

# See list of docker virtual machines on the local box
$ docker-machine ls
NAME      ACTIVE   URL          STATE     URL                         SWARM   DOCKER   ERRORS
default   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.9.1 

# Note the host URL 192.168.99.100 - it will be used later!

# Build an image from current folder under given image name
$ docker build -t gleb/demo-app .

Raidus

user web;

# One worker process per CPU core.
worker_processes 8;

# Also set
# /etc/security/limits.conf
#   web soft nofile 65535
#   web hard nofile 65535
# /etc/default/nginx

Raidus

Moved to git repository: https://github.com/denji/nginx-tuning

NGINX Tuning For Best Performance

For this configuration you can use web server you like, i decided, because i work mostly with it to use nginx.

Generally, properly configured nginx can handle up to 400K to 500K requests per second (clustered), most what i saw is 50K to 80K (non-clustered) requests per second and 30% CPU load, course, this was 2 x Intel Xeon with HyperThreading enabled, but it can work without problem on slower machines.

You must understand that this config is used in testing environment and not in production so you will need to find a way to implement most of those features best possible for your servers.

Raidus

# Redis Cheatsheet
# All the commands you need to know


redis-server /path/redis.conf  # start redis with the related configuration file
redis-cli                      # opens a redis prompt


# Strings.

Raidus

require('dotenv').config()
const cors = require('cors')
const bodyParser = require('body-parser')
const express = require('express')
const expressJwt = require('express-jwt')
const cookieSession = require('cookie-session')
const jwt = require('jsonwebtoken')
const passport = require('passport')
const GoogleStrategy = require('passport-google-oauth20').Strategy
const jwtSecret = Buffer.from('Zn8Q5tyZ/G1MHltc4F/gTkVJMlrbKiZt', 'base64')

Raidus

/**
 * Install and enable the rabbitmq_delayed_message_exchange plugin as described by Alvaro Videla in this blogpost:
 * https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
 */
const amqp = require('amqplib');
const exchange = 'yourExchangeName';
const queue = 'yourQueueName';
const queueBinding = 'yourQueueBindingName';

// Message consumer