SEIP

A Case Study of Automated Example Driven REST API Documentation at Cisco using SpyREST

Abtstract
  what is the problem? -> REST API documentation
  why case study? -> to see how it can be used in practice
  what was done -> used in a commercial setting
  how does it help -> how it could be used in other place, and implications for the design of automated documentation tool
  
Introduction
  Why we need automated REST API Documentation
  The Problem at Cisco -> Background information
  SpyREST
  Adopted as a case study
  Contributions
  Layout of the rest of the paper
  
Related work
  API documentation
  
  Augmenting API Documentation with Insights from Stack Overflow: Robillard: Auto relate API documentation with Stack Overflow Q & A. " Some of the most
severe obstacles faced by developers learning a new API are related to its documentation [32], in particular because of scarce information about the API’s design, rationale [31], usage scenarios, and code examples [32]."
  
  
  
  REST API documentation
  Example driven documentation
  Case studys on API documentation in practice
  
  Flow:
    Studies to understand API usability -> documentation
    Research to combine formal API documentation -> Usage examples
    Research to find usage examples from StackOverflow, open source repositories
    Research to extend learnings into Web APIs
      MDA, hRESTS, Swagger, RAML, WADL -> Synthetic concept retrofit vs. work as is.
    Microservices
    
    Extracting Models from Web API Documentation: Aims to define a structure to represent non-SOAP Web APIs. The structure contains opertations, parameters, input and output formats. No mention of grouping of operations under resources, and the use of HTTP headers, Verbs etc.
    REST API Management and Evolution Using MDA: Versioning + structure.
    A Maturity Model for Semantic RESTful Web APIs: JSON-LD(HATEOS), HYDRA
    
    
    Shortfall

SpyREST
  Core concept
  System architecture overview
  
Case study
  The API
  The team
  The process
  SpyREST adoption
    Example code
    Screenshots
      Overview/explanation
      Auto generated documentation
    Feedback from API developers
      Process
        Informal
        Limitation that the API developers are co-workers
        Semi-Structured questionnaire / Focused groups -> sharing and collaboration
          How long they used
          Years of experience with REST APIs
          Things worked well
            Efficiency
            Effectiveness
            Ease of use
          Things that didn't work well
            Troubles
    Usage Summary

Discussion
  Lessons learned:
    1. Auto generated REST API documentation from functional test code work well in practice.
  - Developers write test code anyway.
  - With real examples, meaningful data is captured for documentation.
  - The structure of REST APIs payloads do not always follow strict schema.
  
2. Custom content is used to provide overview.
  - Such as API access information, rate limits, error handling and error codes etc.
  - Changelog
  
3. Continuous update helps API evolution, before and after release.
  - Documentation helps dev and QA teams work in parallel.
  - Late changes before release automatically reflect on the documentation.
  - Documenting a new version only needs to run the same tests against the new version for API elements that aren't affected.
  
4. API documentation needs same release pipeline as the API.
  - Dev, Staging environment(s), production environment(s)

5. Consistently repeatable API examples are needed.
  - Need seed data for API objects that aren't accessible through the API.
  - Need a setup/teardown for API examples that change state.

6. Version controlling custom content is challenging.
  - Custom content is edited on the wiki, outside of the code.
  
7. Confidential information need to be obfuscated from documentation.
  - API credentials
  - API example data such as internal URLs, emails etc.
8. Manual changelog can be time consuming.
  - Done based on the test code.

  Generalizability:
  The concept of generating REST API documentation with API structure and usage examples by intercepting API calls:

1. Can be used to document other REST APIs.
2. Can be applied to non REST APIs.

  
Conclusion
  
Future work

Parts to decribe documentation

A Case Study of Automated Example Driven REST API Documentation at Cisco using SpyREST

Key reviewing criteria are listed below. Note that not all criteria are appropriate for every submission — e.g., improvement on the state-of-the-practice may be irrelevant for an experience report — and that we will adjust criteria to fit the given type of submission.

Relevance to ICSE SEIP audience
The core concepts of the work either originate in research, either at ICSE or a related conference, or are novel ICSE-appropriate topics.

The core concepts originated in research as described in the ASE papers. Also, API and API documentation are topics of interest to the ICSE audience.

Improvement on the state-of-the-practice
The amount of improvement that the work achieves above and beyond the state-of-the-practice.

1. Automated REST API documentation.
2. A novel approach using HTTP proxy servers to intercept example API calls.
3. Qualitative feedback from developers.

Impact of tech transfer activity
The scale of the impact (e.g., individual vs team vs several teams) of the tech transfer work (only relevant for tech transfer activities).


Backstory
1. Commercial API for accessing and controlling the features of a cloud based cyber security application.
2. API is developed and maintained by a team of engineers
3. API is used by 100+ customers
4. 70K+ API calls/day
5. SpyREST has been used for 1.5 years.
6. Documents two APIs, and 2 versions for each API.
7. In total 58 API endpoints, 100+ examples.
8. A team of engineers involved.
9. Cloud and on-premise options.

Generality of results
The probability that the work, approach, or lessons learned are applicable to developers outside of the studied group.


Clarity of lessons learned
The clarity in which the lessons learned are presented and how well they are supported with data and discussion.

1. Auto generated REST API documentation from functional test code work well in practice.
  - Developers write test code anyway.
  - With real examples, meaningful data is captured for documentation.
  - The structure of REST APIs payloads do not always follow strict schema.
  
2. Custom content is used to provide overview.
  - Such as API access information, rate limits, error handling and error codes etc.
  - Changelog
  
3. Continuous update helps API evolution, before and after release.
  - Documentation helps dev and QA teams work in parallel.
  - Late changes before release automatically reflect on the documentation.
  - Documenting a new version only needs to run the same tests against the new version for API elements that aren't affected.
  
4. API documentation needs same release pipeline as the API.
  - Dev, Staging environment(s), production environment(s)

5. Consistently repeatable API examples are needed.
  - Need seed data for API objects that aren't accessible through the API.
  - Need a setup/teardown for API examples that change state.

6. Version controlling custom content is challenging.
  - Custom content is edited on the wiki, outside of the code.
  
7. Confidential information need to be obfuscated from documentation.
  - API credentials
  - API example data such as internal URLs, emails etc.
8. Manual changelog can be time consuming.
  - Done based on the test code.

Overall quality of the manuscript
While we intentionally do not limit submissions to certain categories, we do encourage potential authors to view ICSE SEIP 2016 list of papers for examples of appropriate submissions.