prework.md

Creating the Proxy Server

The pre-work incorporates a few steps:

1. Setup your Node.js environment
2. Complete the introductory NodeSchool.io Workshoppers
3. Build a proxy server and setup logging
4. Develop a command-line interface (CLI) to configure the server
5. Submit the project for review via Github
6. Extend your proxy server with additional functions

Questions? If you have any questions or concerns, please feel free to email us at foo@bar.com.

1. Setup Node Environment

Before you get started, setup your Node.js environment.

Start reading about JavaScript and Node.js. You can find a good quick JavaScript overview here and here and a quick node.js overview here. A more complete JavaScript primer from Mozilla is here and for node.js see the API documentation. If you prefer to learn by doing, the self-paced Nodeschool.io workshoppers are the gold standard. Checkout javascripting, scope-chains-closures, learnyounode, count-to-6 and functional-javascript-workshop.

2. NodeSchool.io Workshoppers (Optional but Recommended)

Please complete the javascripting and learnyounode NodeSchool.io Workshoppers and include screenshots in your Pre-Work README showing all exercises completed:

3. Build the Proxy Server

Watch this Proxy Server video walkthrough to get the basic Proxy Server functionality up and running. Steps are also documented below within this section.

Basic Setup

1. Create a new directory for your project and cd into it

2. Run npm init in your project root and follow the prompts

3. Create index.js and add the following to create a server using the core http module:

   let http = require('http')
   
   http.createServer((req, res) => {
   	console.log(`Request received at: ${req.url}`)
   	res.end('hello world\n')
   }).listen(8000)

   Note: We'll be using ESNext throughout the course. The above snippet utilizes the arrow-function syntax. See the Basic JavaScript Guide for a language syntax introduction.

   Note: With the exception of package.json and the dependencies in node_modules, all the code for the Proxy Server Prework will go in index.js.

4. Run your server using node installed in the Setup Guide:

   $ node index.js

   and verify it's running:

   $ curl http://127.0.0.1:8000
   hello world

5. Use npm start instead to run your server:

   The convention is to use npm start to start your server. To use npm start, we must add a "start" entry to the package.json "scripts" object. See npm's package.json documentation for details, specifically the scripts section.

   Here is a sample package.json with a "start" script:

   {
     "name": "proxyecho-demo",
     "version": "1.0.0",
     "description": "",
     "main": "index.js",
     "scripts": {
       "start": "node index.js"
     },
     "author": "Me",
     "license": "MIT",
     "dependencies": {}
   }
   

Bonus: Have your server auto-restart on code changes using nodemon: nodemon index.js

Note: Follow the instructions in the Setup Guide to install nodemon and the nodemon alias.

Bonus: Have npm start start your server with nodemon. See the npm package.json "scripts" documentation for how to implement this.

Echo Server

Our goal here is to create a server that echos our requests back to us.

We'll be using node.js streams throughout this assignment. Think of node.js streams as arrays of data over time instead of memory. For the Proxy server, you need only know the readableStream.pipe(writableStream) and writableStream.write(string) APIs. See our Stream Guide or the Stream Handbook for more information.

Common readable streams: req, process.stdin

Common writable streams: res, process.stdout

1. Instead of using res.end to send a static response, let's send the request back to ourselves by piping our readable request stream to our writable reponse stream:

   req.pipe(res)

   and verify it's working:

   $ curl -X POST http://127.0.0.1:8000 -d "hello self"
   hello self

2. Since most HTTP request headers can also be used as response headers, let's echo back the headers too by adding the following line:

   for (let header in req.headers) {
   	res.setHeader(header, req.headers[header])
   }

   Excellent. Now our request headers will be echoed back as well:

   $ curl -v -X POST http://127.0.0.1:8000 -d "hello self" -H "x-asdf: yodawg"
   * Rebuilt URL to: http://127.0.0.1:8000/
   * Hostname was NOT found in DNS cache
   *   Trying 127.0.0.1...
   * Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
   > POST / HTTP/1.1
   > User-Agent: curl/7.37.1
   > Host: 127.0.0.1:8000
   > Accept: */*
   > x-asdf: yodawg
   > Content-Length: 10
   > Content-Type: application/x-www-form-urlencoded
   > 
   * upload completely sent off: 10 out of 10 bytes
   < HTTP/1.1 200 OK
   < user-agent: curl/7.37.1
   < host: 127.0.0.1:8000
   < accept: */*
   < x-asdf: yodawg
   < content-length: 10
   < content-type: application/x-www-form-urlencoded
   < Date: Mon, 13 Apr 2015 00:45:50 GMT
   < Connection: keep-alive
   < 
   * Connection #0 to host 127.0.0.1 left intact
   hello self

Proxy Server

Now, let's build a proxy server. A proxy server is just a server that forwards a request on to a destination server, URL or IP, and responds with the destination server's response.

1. We'll make our echo server our destination server (We'll make this configurable later):

   let destinationUrl = '127.0.0.1:8000'

2. And create a separate proxy server:

   http.createServer((req, res) => {
     console.log(`Proxying request to: ${destinationUrl + req.url}`)
     // Proxy code here
   }).listen(8001)

3. Next we want to make a request to the destination server. For convenience, we'll use the request package instead of the core http.request functionality

   1. Install request:

      project_root$ npm install --save request

   2. Require request at the top of index.js:

      // All require calls go at the top of index.js
      let http = require('http')
      let request = require('request')

   3. Make a request to the destination server at the same path using request:

      http.createServer((req, res) => {
      	// Proxy code
      	let options = {
      		headers: req.headers,
      		url: `http://${destinationUrl}${req.url}`
      	}
      	request(options)
      }).listen(8001)

      Note: The above code uses ESNext string interpolation, a feature provided by Babel.js.

   4. Forward the destination server's reponse:

      request(options).pipe(res)

   5. Lastly, for non-GET requests, we'll want to forward the request body coming into the server on req and set option.method to req.method. To accomplish the former, pipe the incoming req to the outgoing request(options):

      options.method = req.method
      // Notice streams are chainable:
      // inpuStream -> input/outputStream -> outputStream
      req.pipe(request(options)).pipe(res)

   6. Verify the proxy server at http://127.0.0.1:8001 operates exactly the same as the echo server at http://127.0.0.1:8000:

      $ curl -v http://127.0.0.1:8001/asdf -d "hello proxy"
      * Hostname was NOT found in DNS cache
      *   Trying 127.0.0.1...
      * Connected to 127.0.0.1 (127.0.0.1) port 8001 (#0)
      > POST /asdf HTTP/1.1
      > User-Agent: curl/7.37.1
      > Host: 127.0.0.1:8001
      > Accept: */*
      > Content-Length: 11
      > Content-Type: application/x-www-form-urlencoded
      > 
      * upload completely sent off: 11 out of 11 bytes
      < HTTP/1.1 200 OK
      < user-agent: curl/7.37.1
      < host: 127.0.0.1:8001
      < accept: */*
      < content-length: 11
      < content-type: application/x-www-form-urlencoded
      < connection: close
      < date: Mon, 13 Apr 2015 02:03:29 GMT
      < 
      * Closing connection 0
      hello proxy

Congratulations, you've successfully built a fully functional Proxy Server!

Logging

1. Basic Logging

   Let's make sure all our requests and responses are logged out to stdout. .pipe will allow us to do this quite nicely:

   // Log the req headers and content in the **server callback**
   process.stdout.write('\n\n\n' + JSON.stringify(req.headers))
   req.pipe(process.stdout)

   and for the destination server's response:

   // Log the proxy request headers and content in the **server callback**
   let downstreamResponse = req.pipe(request(options))
   process.stdout.write(JSON.stringify(downstreamResponse.headers))
   downstreamResponse.pipe(process.stdout)
   downstreamResponse.pipe(res)

   Note: We must serialize the headers object with JSON.stringify since the default JavaScript object serialization uselessly outputs [object Object].

4. Adding a CLI

Currently, our proxy server isn't very useful because it only proxies to a single hardcoded url. We want to make that URL in addition to a few other things configurable. So let's add a CLI.

Questions? If you have any questions or concerns, please feel free to email us at ****.

Destination Url

1. We'll be using the yargs package (specifically, version 3), so we'll need to install the packge locally in our project:

   project_root$ npm install --save yargs@3

   Note: Be sure to review the yargs Documentation for supported features.

2. Now, let's pass the destination url on the --host argument:

   // Place near the top of your file, just below your other requires 
   // Set a the default value for --host to 127.0.0.1
   let argv = require('yargs')
   	.default('host', '127.0.0.1:8000')
   	.argv
   let scheme = 'http://'
   // Build the destinationUrl using the --host value
   let destinationUrl = scheme + argv.host

3. Next, we'll add --port support:

   Note: In JavaScript, || can be used as a null coalescing operator. In short, if the first value is empty, use the second value.

   // Get the --port value
   // If none, default to the echo server port, or 80 if --host exists
   let port = argv.port || (argv.host === '127.0.0.1' ? 8000 : 80)
   
   // Update our destinationUrl line from above to include the port
   let destinationUrl = scheme  + argv.host + ':' + port

4. And --url for convenience:

   let destinationUrl = argv.url || scheme + argv.host + ':' + port

5. Finally, if present, allow the x-destination-url header to override the destinationUrl value.

   Headers can be obtained like so req.headers['x-destination-url'].

Better Logging

1. Write to log file

   When the --log argument is specified, send all logging to the specified file instead of process.stdout. The simplest way to implement this is to create an outputStream variable, and use it instead of process.stdout. Since .pipe will cause the destination stream (the log file stream in this case) to close when the source stream closes, we'll need to create a new destination stream every call to .pipe

   Note: process.stdout is special and never closes.

   let path = require('path')
   let fs = require('fs')
   let logPath = argv.log && path.join(__dirname, argv.log)
   let getLogStream = ()=> logPath ? fs.createWriteStream(logPath) : process.stdout
   
   //...
   // Replace .pipe(process.stdout) with
   req.pipe(getLogStream())

2. Use pipeOptions instead:

   While the above works, we can do better and not close the destination stream at all. To do this, use readable.pipe's end option to keep our source streams from closing the destination stream:

   let logPath = argv.log && path.join(__dirname, argv.log)
   let logStream = logPath ? fs.createWriteStream(logPath) : process.stdout
   //...
   // Replace req.pipe(getLogStream()) with
   req.pipe(logStream, {end: false})

3. You can also write directly to the logStream:

   logStream.write('Request headers: ' + JSON.stringify(req.headers)

4. Be sure to verify that your logfile is created and contains all the expected contents!

6. (Optionals) Extending your Server

After initial submission, you should iterate by adding several additional features to your Proxy server. Engineers that submit Proxy servers with extended functionality and improved APIs are significantly more likely to be accepted into the bootcamp. Be sure to refer to our new node.js guides. Try your hand at implementing the following user stories and any other extensions of your own choosing:

1. Process Forwarding

   HTTP proxying is nice, but let's make this a multi-purpose proxy process. Allow a process name to be specified in the --exec argument. You'll want to use child_process.spawn to pipe stdin/stdout/stderr between the process and the child process.

   Note: When run in process proxy mode, don't start the server.

   Why on earth are we doing this?!

   Because we can! Actually, it's because node.js is becoming an extremely popular scripting option, especially for devops. This feature will familiarize you with CLI scripting in node.js, especially stdin/stdout and argument handling.

   Example 1:

   $ cat index.js | grep require
   $ # Is equivalent to...
   $ cat index.js | node index.js --exec grep require
   
   let http = require('http')
   let request = require('request')
   let argv = require('yargs')	

   Example 2:

   $ cp index.js index.js.bkp
   $ # Is equivalent to...
   $ node index.js --exec cp index.js index.js.bkp

   Example 3 (from this StackOverflow):

   $ # Kill all node processes
   $ ps aux | grep node | grep -v grep | awk '{print $2}' | xargs kill
   $ # Is equivalent to...
   $ alias run='node index.js --exec' # For brevity
   $ ps aux | run grep node | run grep -v grep | run awk '{print $2}' | xargs kill

2. Better Logging

   When the --loglevel argument is specified, only output logs greater than or equal to the specified level. See the Syslog Severity Levels for a list of recommended log levels.

   This is a little more difficult, but your best bet is a functional programming approach. Consider implementing and utilizing the following utility functions:

   function log(level, msg) {
     // Compare level to loglevel
     // Is msg a string? => Output it
     // Is msg a string? => Stream it
   }

   Hint: Detect streams with instanceof and the stream.Stream constructor or Strings with typeof foo === 'string'

3. Documentation, -h

   Yargs has built-in support for documentation using the .usage and .describe methods. Document all the functionality you've implemented, and expose them with -h.

   $ node index.js -h
   Usage: node ./index.js [options]
   
   Options:
     -p, --port  Specify a forwarding port
     -x, --host  Specify a forwarding host
     -e, --exec  Specify a process to proxy instead
     -l, --log   Specify a output log file
     -h, --help  Show help
   
   Examples:
     node index.js -p 8001 -h google.com
   
   copyright 2015

4. Support HTTPS

   Send an https request to the destination server when the client sends an https request. There is a core https module you can use as a drop-in replacement for the http module and an https tutorial here.

   Bonus: Add --host-ssl and --port-ssl CLI arguments for separate https destination ports.

5. Tweak the log styling, play with colors, spacing and additional data (For colors checkout chalk).

Please reply back to the github issue after pushing new features. Be sure to include in the README an updated GIF walkthrough using LiceCap of the app demoing how it works with required user stories completed.

Troubleshooting Notes

During setup, there are a number of ways that things can go wrong and this can be very frustrating. This is an attempt to help you avoid common issues as much as possible.

See the Node.js Setup guide for help setting up your environment and the Troubleshooting guide for help with issues like debugging.