notes-apicraft-testing.md

API testing and validation

• Difference?

  • Validation: API receives a message and decides if it's valid or not
  • Testing: different use cases
    • TDD as you build the API
    • Test if the API is running (lifecycle/ci)
    • Debugging
  • API performance testing

• Language-agnostic tooling

• Schemas are complicated/take some effort

  • Only a few people in the audience are using it
  • Someone using annotation to define a schema that is then used to test
  • It seems like duplicated effort
    • Although there's the possibility of extracting a schema or using it to generate code?

• Validate input with schema (declaratively) or custom code?

  • Seems like majority has code doing it
  • Schemas are harder to use
  • How to implement validation with Schema when you're doing something like TDD?
  • Some validation doesn't fit a schema, so you might end up with both!
    • Two sources of validation
  • Using a declarative approach allows you to use different languages, move validation into client, etc.
  • Schema is not the only way, people considering a DSL for more powerful (and centralized) validations
  • SOAP providers are taking JSON input and converting to XML to validate and forward on

• URI templates: /user/{id} vs the Ruby convention /user/:id

• Very few validation around headers

• What's the workflow between writing an API spec and implementing it?

  • Having the spec be an artifact like a PDF makes it harder for collaborative/iteractive work

• Integration testing

  • How to setup?
    • Clean up the database and leave it in the desired state before/after each test run
    • Mock database
  • What to test?
    • Some go all the way back to http, testing it locally (eg boot server, make requests locally)
    • Some tools help testing within the language/framework (eg rack+rack-test), and this seems to be sufficient most of the time
  • One idea for Blueprint is to integrate with Cucumber so you can effectively integrate api docs and testing
  • Gavel is a tool to diff API calls

• Performance testing

  • JMeter + monitoring tools to look at metrics (median, perc95 and perc99)
    • Test in production
    • Gather a lot of continuous metrics
    • Setup alerts on specific thresholds or patterns (eg response time spike)
  • LoadRunner
  • Outsource/services: BrowserMob, WebMetrics and others

• Unit testing and integration testing

  • Integration can take a long time, be selective
    • Some are separating integration tests into categories (eg run some often, run slow/less important ones only on CI or after, etc)
  • CI
    • Different uses with Jenkins
  • API testing is easier than UI testing

• Tooling

  • Rest-assured
  • Fitnesse
  • Cookies!? Eg for a sudo session