Skip to content

Instantly share code, notes, and snippets.

@crossi202

crossi202/EC2 Proxy Settings to View Hadoop Websites.md

Last active January 24, 2016 10:14
Show Gist options
  • Save crossi202/3f0b8c81d09ff72b42a2 to your computer and use it in GitHub Desktop.
Save crossi202/3f0b8c81d09ff72b42a2 to your computer and use it in GitHub Desktop.
Download ZIP
Configure Proxy Settings to View Hadoop Websites Hosted on a EC2 Hadoop MapReduce Cluster
Raw
EC2 Proxy Settings to View Hadoop Websites.md

Configure Proxy Settings to View Hadoop Services Web GUIs Hosted on a EC2 Hadoop MapReduce Cluster

Because of Hadoop usage of internal names to configure the cluster, you will need to perform the following actions:

  • Setup of a SSH tunnel with the Master node,
  • Configure a browser proxy tool (for example FoxyProxy)

After that you will be able to view the Hadoop Services Web GUIs.

Part 1. Setup of a SSH tunnel using dynamic port forwarding

To connect to the local web server on the master node, you create a SSH tunnel between your computer and the master node. This is also known as port forwarding. If you create your SSH tunnel using dynamic port forwarding, all traffic routed to a specified unused local port is forwarded to the local web server on the master node. This creates a SOCKS proxy. You can then configure your Internet browser to use an add-on such as FoxyProxy to manage your SOCKS proxy settings. Using a proxy management add-on allows you to automatically filter URLs based on text patterns and to limit the proxy settings to domains that match the form of the master node's public DNS name. The browser add-on automatically handles turning the proxy on and off when you switch between viewing websites hosted on the master node, and those on the Internet.

Before you begin, you need the public IP address of the master node. For information about how to locate the master public IP address, see the documentation available at http://docs.terradue.com/developer-sandbox or raise a question at https://support.terradue.com

Set Up an SSH Tunnel to the Master Node Using Dynamic Port Forwarding on Linux, Unix, and Mac OS X

  • Open a terminal window. On Mac OS X, choose Applications > Utilities > Terminal. On other Linux distributions, terminal is typically found at Applications > Accessories > Terminal.

  • Type the following command to open an SSH tunnel on your local machine. Replace <yourcertificate.pem> with the location and file name of your .pem file, replace 8157 with an unused, local port number, and replace @ with the master public IP of your cluster.

ssh -i <yourcertificate.pem> -N -D 8157 <username>@<master ip>

After you issue this command, the terminal remains open and does not return a response.

NOTE -D signifies the use of dynamic port forwarding which allows you to specify a local port used to forward data to all remote ports on the master node's local web server. Dynamic port forwarding creates a local SOCKS proxy listening on the port specified in the command.

  • After the tunnel is active, configure a SOCKS proxy for your browser. For more information, see Part 2. Configure FoxyProxy

When you are done working with the web interfaces on the master node, close the terminal window.

Set Up an SSH Tunnel to the Master Node Using Dynamic Port Forwarding on Windows

Windows users can use an SSH client such as PuTTY to create an SSH tunnel to the master node. Before connecting to the Amazon Hadoop master node, you should download and install PuTTY and PuTTYgen. You can download these tools from the PuTTY download page http://www.putty.org/.

PuTTY does not natively support the key pair private key file format (.pem) generated by the Terradue's Certification Authority. You use PuTTYgen to convert your key file to the required PuTTY format (.ppk). You must convert your key into this format (.ppk) before attempting to connect to the master node using PuTTY.

For more information about converting your key, see http://docs.terradue.com/developer-sandbox/start/sandbox.html section connecting-from-windows.

  • Double-click putty.exe to start PuTTY. You can also launch PuTTY from the Windows programs list.

Note If you already have an active SSH session with the master node, you can add a tunnel by right-clicking the PuTTY title bar and choosing Change Settings. If necessary, in the Category list, click Session.

  • In the Host Name field, type
<username>@<master node ip>

  • In the Category list, expand Connection > SSH, and then click Auth.

  • For Private key file for authentication, click Browse and select the .ppk file that you generated.

  • In the Category list, expand Connection > SSH, and then click Tunnels.

  • In the Source port field, type 8157 (an unused local port).

  • Leave the Destination field blank.

  • Select the Dynamic and Auto options.

  • Click Add and then click Open.

  • Click Yes to dismiss the PuTTY security alert.

After the tunnel is active, configure a SOCKS proxy for your browser. For more information, see Part 2. Configure FoxyProxy.

When you are done working with the web interfaces on the master node, close the PuTTY window.

Part 2. Configure FoxyProxy

FoxyProxy provides a set of proxy management tools that allow you to use a proxy server for URLs that match patterns corresponding to the domains used by the Amazon EC2 instances in your Hadoop MapReduce cluster.

To install and configure FoxyProxy in Firefox

  • Download and install the Standard version of FoxyProxy from http://getfoxyproxy.org/downloads.html.

  • Using a text editor, create a file named foxyproxy-settings.xml containing the following:

<?xml version="1.0" encoding="UTF-8"?>
<foxyproxy>
   <proxies>
      <proxy name="ec2-socks-proxy" id="2322596116" notes="" fromSubscription="false" enabled="true" mode="manual" selectedTabIndex="2" lastresort="false" animatedIcons="true" includeInCycle="true" color="#0055E5" proxyDNS="true" noInternalIPs="false" autoconfMode="pac" clearCacheBeforeUse="false" disableCache="false" clearCookiesBeforeUse="false" rejectCookies="false">
         <matches>
            <match enabled="true" name="*ec2*.amazonaws.com*" pattern="*ec2*.amazonaws.com*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
            <match enabled="true" name="*ec2*.compute*" pattern="*ec2*.compute*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
            <match enabled="true" name="*compute.internal*" pattern="*compute.internal*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
	  </matches>
         <manualconf host="localhost" port="8157" socksversion="5" isSocks="true" username="" password="" domain="" />
      </proxy>
   </proxies>
</foxyproxy>

  • Click Firefox > Add-ons.

  • On the Add-ons tab, to the right of FoxyProxy Standard, click Options.

  • In the FoxyProxy Standard dialog, click File > Import Settings.

  • Browse to the location of foxyproxy-settings.xml, select the File, and click Open.

  • Click Yes when prompted to overwrite the existing settings and then click Yes to restart Firefox.

  • When Firefox restarts, on the Add-ons tab, to the right of FoxyProxy Standard, click Options.

  • In the FoxyProxy Standard dialog, for Select Mode, choose Use proxies based on their pre-defined patterns and priorities.

  • Click Close.

To install and configure FoxyProxy in Google Chrome

  • Download and install the Standard version of FoxyProxy from http://getfoxyproxy.org/downloads.html.

  • When prompted, click FREE to install the FoxyProxy extension and then click Add.

  • Using a text editor, create a file named foxyproxy-settings.xml containing the following:

<?xml version="1.0" encoding="UTF-8"?>
<foxyproxy>
   <proxies>
      <proxy name="ec2-socks-proxy" id="2322596116" notes="" fromSubscription="false" enabled="true" mode="manual" selectedTabIndex="2" lastresort="false" animatedIcons="true" includeInCycle="true" color="#0055E5" proxyDNS="true" noInternalIPs="false" autoconfMode="pac" clearCacheBeforeUse="false" disableCache="false" clearCookiesBeforeUse="false" rejectCookies="false">
         <matches>
            <match enabled="true" name="*ec2*.amazonaws.com*" pattern="*ec2*.amazonaws.com*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
            <match enabled="true" name="*ec2*.compute*" pattern="*ec2*.compute*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
            <match enabled="true" name="*compute.internal*" pattern="*compute.internal*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
	  </matches>
         <manualconf host="localhost" port="8157" socksversion="5" isSocks="true" username="" password="" domain="" />
      </proxy>
   </proxies>
</foxyproxy>

  • Click Customize and Control Google Chrome > Tools > Extensions.

  • On the Extensions tab, below FoxyProxy Standard, click Options.

  • On the FoxyProxy Standard page, click Import/Export.

  • On the Import/Export page, click Choose File, browse to the location of foxyproxy-settings.xml, select the file, and click Open.

  • Click Replace when prompted to overwrite the existing settings.

  • At the top of the page, for Proxy mode, choose Use proxies based on their pre-defined patterns and priorities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment