crossi202

Git-Flow Cheatsheet

Prerequisites

• To have a local working copy of a remote git repository.

Initialize the Repository for git-flow:

$ git flow init -d

crossi202

Git-Flow SVN Cheatsheet

In this cheatsheet, the Git Flow process is followed using the tools:

• git svn
• svn
• git

Prerequisites

crossi202

Configure Proxy Settings to View Hadoop Services Web GUIs Hosted on a EC2 Hadoop MapReduce Cluster

Because of Hadoop usage of internal names to configure the cluster, you will need to perform the following actions:

• Setup of a SSH tunnel with the Master node,
• Configure a browser proxy tool (for example FoxyProxy)

After that you will be able to view the Hadoop Services Web GUIs.

Part 1. Setup of a SSH tunnel using dynamic port forwarding

crossi202

VPN (L2TP/IPsec) configuration at Interoute

Server side

To enable the VPN server at Interoute, the steps are described at:

http://cloudstore.interoute.com/main/knowledge-centre/library/how-enable-software-vpn-connection-vm

Note. The authentication is performed through a Pre-Shared Key (PSK).

crossi202

Setting up the number of mappers per job

Rationale

The number of mappers per job is a function of the number of blocks across all the files used as input for the mapreduce job. It could be necessary to setup explicitily the number of mappers per job when, for instance, the inputs are just references to files (and the input file containing the references occupies just one HDFS block).

Procedure

• Add the following property to the mapred-site.xml configuration file on all the tasktrackers nodes of the Hadoop cluster:

crossi202

Application packaging and Github usage

Introduction

This procedure will guide you on the usage of automation tools, in order to build an Application suitable for the Terradue Cloud Platform (see these examples). Such applications have the main advantage to be self-contained. In other words, a single package (both in RPM and Docker image formats) contains the application resources and the dependencies specification. For this step, we will also add a new repository under your Github organization.

Prerequisites

• A Github account. If you don't have it, you can request a new one here https://github.com/join,
• An application name for the Github repository under your Github organization. We will add the dcs- prefix in order to be homogeneous with the existing repositories.

crossi202

Installing CentOS 6.7 on Hetzner with partitions larger than 2TB

Install the CentOS 6.7 64 bit minimal

• Start the server in Rescue mode [1],
• Type

$ installimage

crossi202

Generate and install the SSH key

SSH keys are a way to identify trusted computers without involving passwords. You can generate an SSH key and add the public key to your Terradue Cloud Platform account by following the procedures below.

Generate a new SSH key

• Go to https://www.terradue.com,
• Type your credentials,
• Click on the top-right botton showing your username,
• Click on Profile,

crossi202

MELODIES - Exploiting Open Data

Open Data Tools - Session 1: Working with Cloud resources

Brussels, 13rd October 2016

Prerequisites

• You have successfully performed the SSH key pair and VPN Setup (Training Preparatory Step 1/2),
• You have successfully performed the Cloud Dashboard access and Sandbox deployment (Training Preparatory Step 2/2).

Get the Application

crossi202

MELODIES - Exploiting Open Data

Open Data Tools - Session 2: Working with Data resources

Brussels, 13rd October 2016

Prerequisites

• You successfully performed the Open Data Tools - Session 1: Working with Cloud resources.

Get the API key