Instructions for the creation of a code evaluation service

evaluator-service.md

Code Evaluator Service

The goal of this project is to create and publish an API that allows users to evaluate code in different programming languages using isolated Docker containers.

Use cases for this project include an online IDE for pair programming, a remote interview tool and evaluation of challenges (like Make it Real), among others.

Before using this service users have to sign up so that they can grab a generated API Key that they have to use to authenticate the requests.

Assuming that the service is published in the domain http://eval.io/, it should be posible to make a POST to the path /v1/eval/<platform> where <platform> is one of the languages or platforms supported.

We need to support at least Ruby and NodeJS for this exercise.

Simple evaluation

POST https://eval.io/v1/eval/node

{
  "code": "console.log('hello world');"
}

Response:

{
  "status": "finished",
  "exit_code": 0,
  "output": "hello world"
}

If the code doesn't finish executing in 15 seconds it should timeout:

{
  "status": "timeout"
}

Async evaluation

{
  "code": "puts 'hello world'",
  "async": true
}

This will return the id of the evaluation that can be used to check the result. Also, the user can open a WebSocket channel and watch the output in realtime.

Evaluating code with stdin

{
  "code": "console.log('hello world')",
  "stdin": ["1", "2", "3"]
}

Evaluating code with dependencies

{
  "code": "console.log('hello world')",
  "dependencies": {
    "faker": "latest"
  }
}

Evaluating from git repo

{
  "git": {
    "repo": "http://github.com/germanescobar/test",
    "command": "node main.js"
  }
}

Storing evaluation requests

Every time a request is received we need to store the following in the database:

• user_id
• status (created, evaluating, timeout, finished)
• platform: string
• request: text
• exit_code: integer
• output: text