Skip to content

Instantly share code, notes, and snippets.

@woloski

woloski/runscope vs node.md

Last active October 3, 2015 22:48
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save woloski/39eb2418d9f6f1595f52 to your computer and use it in GitHub Desktop.
Save woloski/39eb2418d9f6f1595f52 to your computer and use it in GitHub Desktop.
Download ZIP
Runscope vs Node+Mocha
Raw
runscope vs node.md

This is a comparison of two testing approaches: Runscope service and pure Node.js coding. Written by Hernan Meydac Jean (@VerlicRedclaw)

Runscope

This service provides a simple to use editor to create API tests with no effort.

Pros

  • Nice postman-like editor to create API tests.
  • Lots of configurable options, initial variables, shared settings, conditionals, etc.
  • Uses Chai.js as the Assertion library for responses
  • Allows custom scripts to assert responses
  • Test scheduling allows you to run tests every X minutes
  • Integration with Slack, Jenkins, GitHub, etc.
  • Runscope API and trigger URLs allows test customizations
  • Can execute tests using multiple regions
  • Traffic Inspector shows performance, usage, and error metrics.

Cons

  • Custom scripts are not versioned. If your API changes you must change your scripts manually on Runscope.
  • Complex asserts must be done in the Custom scripts section.
  • Limited NPM packages available.
  • No payload or JSON schema validation. Asserts must be specified individually for each property.

Node.js + Mocha Tests

Create all tests using Mocha or other test framework to perform integration tests against the API.

Pros

  • Code flexibility. Everything can be done.
  • Easy payload and JSON schema validation using libraries.
  • Tests can be versioned in GitHub.
  • Common triggers exist in Jenkins to execute tests.
  • Any integration (e.g. Slack) can be built in code or by including external packages.
  • We can reuse these tests for stress testing.

Cons

  • Creating tests is considerable more expensive (time).
  • Metrics, monitoring, reporting, and integrations must be built manually.
  • Scheduling must be configured in Jenkins.
  • No multi-region testing.
  • Depends on the CI server (not agnostic/external)

Conclusions

Both approaches have their advantages. Runscope is great for monitoring, metrics, and integrations. Tests are easily created in Runscope. It’s a great service to detect errors as soon as possible when monitoring the APIs (e.g. downtime, performance issues, expected HTTP codes, basic asserts).

For business logic, complex asserts, payload validations, custom code is the winner. We have the flexibility to create any type of test and detect errors in the logic of the API. The downside is that we need to code every integration we would like to have, like monitoring, response times, send messages to Slack, etc.

I believe that Runscope is great to start with. We’ll have an error detection mechanism that will alert of any possible issue with the APIs providing some basic payload validations. In the meantime, integration tests can be created in parallel to provide a broader coverage.

In the long term, code tests might replace Runscope capabilities and we could keep our automated tests with a proper monitoring infrastructure.

It would be nice if Runscope supported extensibility the way Auth0 supports Rules i.e. in the same way you can write nodejs that executes on Auth0 servers in the authenitcaiton pipeline, it would be great if Runscope could allow devs to execute full fledge Node.js code for complex asserts, using webtask.io.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment